T1 Summary and User Testing

[lilymaechling]

T1 Summary and User Testing

Summary

Summarize your user testing and any feedback you got during demos.

Users started on the landing page and were prompted to create a new file. Afterwards they would say one of the commands avaliable or a command with one of the key terms. After code was produced, we asked users to download the file.

Observations

Record your observations and any insights/conclusions you may have about them. Identify top patterns and write them up. Record the number of email signups you got.

Microphone and Run UX:

• Unaware that to start voice to text processing that had to press microphone button
  • Most users did not read the message in the side bar with instructions
• Unaware of need to press to start and stop
• Many thought that the play button was what they were supposed to press to start recording despite the "Run" message we added General Use Cases:
• Less expereinced coders needed prompts for code -A CS 001 student might not know where to begin without prompting -People with no coding experience have no way to use or try the program to see what coding is like
• Experienced coders tried to push the boundaries of OpenAI
• Seemed more effective and helpful for inexpirienced coder who needed to be more vague in their input
• Experienced thought that they could say very specific code as opposed to a more abstract idea -Ex. "for i in range 0 to 10" rather than "create a for loop that iterates ten times"
• Unaware if their voice commands were processing -Not having a progress wheel is especially problematic given the varying latency from OpenAI
• No way for the user to verify that the voice to text was correct
• If the speech to text didn't work, the application crashed: must add an error handle if the speech to text becomes confused and can't formulate a response -People generally frustrated with how long it took but impressed with the code results returned We had a total of 16 responses to our user survey during Technigala.

Analysis

Is there any analysis you can do of the testing - what percentage of users claimed X or Y. If you had numerical answers was there any statistical significance to the answers?

• User responses show that people would mostly use ConvoCode for educational purposes, assignment help, and interview prep.
• Users are interested in seeing the full capacity of openAI and wanted to expand beyond our basic, limited examples to see how it can handle algorithms, Linked Lists, DFS, Binary Trees, etc
• Users also wanted to allow for stdin to be supplied to produced code
• Users need to know more about what is actually happening behind the scenes - if they say something without a keyword, it would be helpful to tell them that this is why they didn't get the desired result instead of just saying "Sorry, no code found"

Our survey revealed that users wanted to use our product to[]:

1. To learn python
2. Assignment help
3. Interview prep

Notes To Future Us

Leave notes to yourself for next term - what should your next steps be? Filled out on zenhub. Big picture, we want to work on chat bot, UI fixes, and OpenAi plugins like runtime and code explanation. We would also really like to speed up the process of voice to code as depending on complexity of imput, our processes can take a very long time.

TeamWork

Reflect on how your team is working together Our team is working really well together. I think we have a nice pace of work and understanding of other people's schedules. We frequently meet up or communicate what work we are doing. Our group coding sessions on FFB are highly effective as well as our team social events.

What has been working well?

give kudos and shoutouts, comment on communication techniques that have worked, pr review practices, etc

• Communication
• Team work
• PR reviews are prompt (and imposing branch rules has been hepful!)
• We have flexible schedules that allow us to meet frequently

What has not?

try to discuss any problems openly and respectfully. communication breakdowns? are some teammembers taking others tasks for themselves? is anybody ghosting? etc

Over the past term the team has gotten really close. As this friendship has developed sometimes we don't hold each other 100% accountable and allow each other to slack off or not put our full effort in.

Steps

if there are problems, think of some ways to resolve them, go back to your team contract from last term to see what you said you would do for resolution

• I don't think there are any team problems. Every member of the team feels a large ownership and responsibility over our work.

[timofei7]

nice demo and some decent feedback at least on the interface.

tbh, all ya'all still need a big picture purpose for the product. i know i keep harping on it and we've discussed it several times. i had hoped something would become clearer during demos. it looks like folks suggested it as an educational platform. i'm curious what would need to change to wrap it for that purpose. would you structure lessons? perhaps the convodex can help but without prompts / structure folks will struggle for what they can do.

we had also talked about an accessibility focus. going all in on coding purely with voice - navigating the code window with voice etc. to-date coding with just voice has been hard because it ends up being narrating syntax and takes a long time - with openai that frees up the voice only coder to be much more effective.

but then without a graphics library, multiple code files, and stdin support users are limited in what the code can do and as such it seems are restricted to toy examples (which speaks more to the educational model where they can have short contained problems) or would you need to expand support to be a more full-fledged IDE?

i think you are all in a good place where you have a tech demo - you can do voice to txt to code - now what will you do with that power? overall glad you all are having a good time with this and i'm excited to see where you take it!