[Research] VS Code - First time experience for /commands

@toolmantim This is the research issue I've created in regards to user testing for the new first time UX in VS Code (slack: https://sourcegraph.slack.com/archives/C05AGQYD528/p1691674809056849). Some asks for you:

Are the research questions good? Would you alter or add any? I try to keep a limit of 3 - 4 questions since the test itself will be about 10 - 15 minutes max.
What were you thinking in terms of a prototype? Figma prototype? Or live in VS Code (this will require testers to install the Cody extension first)? I just need to know to better write the test plan.

For these kinds of quick tests, I'll build the test plan right in usertesting.com and link to it right here in the ticket. Once we've agreed on questions to answers, I'll go ahead and build that test plan and then follow up to get you (and the team's -- > @abeatrix @chenkc805 etc.) feedback about the plan itself.

chenkc805 commented 1 year ago

Thanks for kicking this off! Excited to learn more. CC: @csells and @taylorsperry for visibility into commands

I'm curious what people's expectations are of commands. So when they see the command palette, asking them "What do you expect each of these things to do?"

meglynst commented 1 year ago

@toolmantim @abeatrix Currently, I'm having users start from the VS Code marketplace page (https://marketplace.visualstudio.com/items?itemName=sourcegraph.cody-ai) and clicking install for this test. If we ran this test right now, what authentication options do users have after they've installed the extension? Do we need to have users create Sourcegraph accounts for this test? Since we're not testing those first few steps (installing, signing in, etc.), just commands, I want to make it as frictionless as possible so we can get them to using commands as quickly as possible.

One option is to have them sign in with an S2 account we use for testing purposes. However, will this affect other people (e.g. they won't have a true "first time" experience) if each tester is logging into the extension with the same credentials? Does this put S2 at risk in any way?

chenkc805 commented 1 year ago

I think it's fine to use the S2 account for testing purposes, since our onboarding is evolving all the time at the moment and the goal of this is to get feedback regarding commands and the command interface.

The current onboarding flow is too cumbersome to go through for UXR purposes (they'll have to create an account, get confused because they'll think they have to download a cody app, etc.)

meglynst commented 1 year ago

This is the test plan I have for this test at the moment in usertesting.com:

The two "Image Tasks" are how to log in and sign out of Cody AI (it's a little tough to describe so I wanted to use images).

@toolmantim (and team) let me know what you think!

toolmantim commented 1 year ago

I agree it's a good idea just to give them some credentials to use.

This is the test plan I have for this test at the moment in usertesting.com

That test plan looks good to me! Are 3+4 meant to repeat?

Tiny nit: we've standardised on Cody instead of Cody AI.

Does this put S2 at risk in any way?

No idea! That's a question for tech ops / engineering.

meglynst commented 1 year ago

There was repetition! I've removed Task 4.

In regards to Cody AI, I wasn't sure if we should keep referring to it as "Cody AI" or "Cody" for VS Code tests going forward because the extension itself is called "Cody AI."