[User testing] Combined Profile and Account: Create conversation guide

[Samara-Strauss]

Epic #2092

• [x] Create conversation guide

[Samara-Strauss]

General

• Please always link a document — whether it's a convo guide, research plan, research writeup, etc — to the top comment in the ticket. This is really helpful for tracking/documentation purposes. Also, please always solicit feedback by linking to a ticket that links to a doc and not the doc itself. When feedback ends up in Slack, it's lost to time and it's hard to follow the conversation on whatever deliverable (or anything) we're talking about.

Intro

• You might want to ask if they are comfortable sharing their screen and confirming that before you get started.
• For sessions that involve a prototype and not the real site, I like to include a blurb explaining that we're looking at a prototype and that not everything will be clickable like it is on a real website. I find that this reduces questions and confusion later on. Totally up to you, but you might find it equally as helpful. Here's an example from my address validation convo guide.

Warm up questions

• "How often do you visit the VA website?" The VA has a lot of websites. You'll need to specify that you're talking about VA.gov. They also may get confused and think you're talking about MyHealtheVet and say they go their all the time not realizing you're talking about a different website. Just watch/listen out for this.
  • Also, I would probably ask "Have you ever been to VA.gov?" and then ask how often they use the website on average.
• This would be a good opportunity to ask them what they do on the website when they come to it, assuming they've used the website at all.
• "Have you used the VA profile or account pages on the current site?" I don't think that the participants will have any idea what you're talking about re: Profile and Account. May be better to assess this by asking if they've ever used the site to change their address or direct deposit information (or asking how they've done these things, and then seeing if they say VA.gov).
• "Do you have any applications currently connected with your VA account?" I doubt anyone will have any idea what you're talking about. Instead, you might want to ask if they used have the Apple Health app. This is the only app that can be connected to VA.gov. If they have used the app, then you could ask if they've connected it to VA.gov. Also, I actually think this is the wrong place entirely for this question. I would ask about this later on when you're looking at the feature.

Screenshare

• I'm going to send you a link to a clickable prototype. Once you have it pulled up, will you share you screen with me? They'll likely need step by step help from you on how to do this. This may take a few minutes of time depending on what tool you're using for the calls.
• Also, this is a good time to talk about how they'll be seeing a fictional person in the prototype. Explain that this person is not real and all their information is made up. The participant should pretend to be the person represented in the prototype for the session.

First Task: Wayfinding and profile navigation

• I really like the scenario! Realize that if you are starting people from the homepage (and you should), they'll click the Change your address link from there, so you'll need that content page in your prototype.
• Walk me through what you think these sections mean. I have no idea what you mean by this, and I think it will also confuse a participant.
• I love that you have a few things to watch for, but you should have an outline of questions you want to ask as well. Whether you ask all of them or ask them verbatim will obviously depend on what happens in the session. This isn't about having a rigid script, but making sure you don't miss anything you want to get answers to. It also makes it hard to determine whether we're evaluating everything we need to, or to determine how long we think the session will take without a list of questions. Again, this conversation guide is a decent example.

Second Task: FAQ

• Same as above — without a proper list of questions, I'm not really sure what you're trying to evaluate here. Can you flesh this out more clearly?

Third Task: Connect an account

• You have been using the AppleHealth Kit application (which is an outside app) and you'd like to disconnect it from your VA account. Show me how you would you do that? People will have no idea what this app is, so you should give them more background on what this app is and what it means to have connected it to their VA.gov account.
• Same as above — we need actual questions here.

Post-Task Interview

• What sort of information do expect to see on the VA profile? What tasks would you expect to do on the profile page? — I might ask this in a way, or supplement this question with the following:
  • "Tell me your overall impressions of the profile we looked at today."
  • "Did this profile meet your expectations? How so?"
• Is there anything that you didn't see that you wish was on the VA profile? — I would reword this slightly. Let's stay away from asking them what they "wish," because that could go off the rails. Instead, "Is there anything you expected to see in the VA.gov profile that you did not see?"
• What are you favorite parts of the profile? Least favorite parts? — "Favorite" is too subjective and doesn't really get at what we want to know, which is whether they found the profile easy to use and navigate and whether it met expectations. Instead, any of the following might be more useful in evaluating their needs and impressions:
  • "What did you think was clear/what worked well?"
  • "What could be improved?"
  • "Was there anything you found confusing? If so, what? Why?"

Thank-You and Closing

• Thanks! Lastly, do you know any other Veterans, caregivers, or service members who might be willing to participate in a future user research session? If Yes: Thank you! I'll have our team send you an email with a little blurb that you can pass along. Will you confirm we still do this? We might, but I'm not sure.
• Also, you should mention their compensation. Here is an example of how to do that.

My overall feedback

TBH, this needs a lot of work before I'd feel comfortable kicking off recruiting. There was a lot outlined in your research plan that I did not see reflected in what you plan to evaluate here. These are a few of those things that come to mind:

• The navigation on both desktop and mobile
• The nametag
• Whether they can find more than just the stuff on the front page (address/how to change their email address)
• Scannability
• Overall impressions
• Points for improvement
• (maybe) How they think this compares to the old profile

Additionally, I can't tell by looking at this script whether you plan to test on desktop or mobile or both, or what the planned flow of information is. While users are going to go where they go (and we should let them), the script should give a clearer sense of intended structure.

From here, I would suggest revisiting this script and fleshing it out more. If it needs to wait until the visual design work (connected accounts/edit in place) is done, that's fine. But this really needs some dedicated time and to be fleshed out more so we can be sure that we're going to evaluate all of the things that need to be evaluated in the session. If it's helpful, you may want to work with @sshein to get this into shape. I'm happy to review it again once it's been given an overhaul.

[tressaellen]

@Samara-Strauss this is super helpful!! Thank you! I'll keep working on it!

[careyotto]

Adding to @Samara-Strauss and @sshein's already great feedback:

• If we're changing the section name from "connected accounts" to "connected apps," as previously discussed, this and convo guide will need to be updated to reflect.

• Agree with @sshein about starting flow from VA homepage and starting with connected account task

• For Warm-up Questions, would be interested in asking:

  • What do you expect to see in a section called “connected apps?” 
  • Have you connected any apps to your VA health records? (replacing: "Do you have any applications currently connected with your VA account?")

• For Third Task: Connect an account:

  • Suggested rewrite of the app question: "Imagine that last month, you used your iPhone's Health app to share your VA medical records with an outside doctor. You want to remove the Health app's access to your VA records. How you would you do that?" (*If y'all don't think that does the trick, please push back.)
  • I haven't seen Peggy's re-write, so if button language changed from "disconnect" to "remove access," use the opposite word in the test script.

• For Things to watch for:

  • This question confuses me: "If user doesn't have an app - can they figure out where to add one?" Users can't add apps from VA.gov. With the new designs they'll be able to find a list of available apps, but they'll still have to go to the app stores or another website to download them.
  • What does the empty state for connected apps look like? Are you testing this as well? Just curious.

• For: Post-Task Interview:

  • Add: Was anything confusing? if so, why?

FYSA, regarding early comment that Apple Health app is the only app that can be connected to VA.gov: We now have 3 apps that can connect in production, with more on the way. Apple is currently the most popular with 25k Veterans connected! Stats from this week.

[tressaellen]

Hi @Samara-Strauss, I've made some updates to the conversation guide and would love your thoughts!

[Samara-Strauss]

I'll take a look at it tomorrow!

[Samara-Strauss]

My feedback based on the latest iteration of the script:

General

• Can you include the link to the prototype at the top of the script so we can check it out more easily?
• Can you add time estimates for each section so we can get a sense of how long you think this will take?

Warm up questions

• I would also ask them what, if any, benefits they are currently receiving from the VA. This will give us a fuller picture of who we are talking to and what their experiences with the VA might be like.
• They may need clarification on “personal information.” I think it’s OK to specifically ask here about their address/phone number/email address since we’re asking about a past experience and not leading them to one answer over another.
• Also they may need some clarification on what you mean by “connecting apps to your health record”, so just be prepared to clarify for them.

First task

• I’m confused about the part where you mention sending the link. I thought you already did that. Are you sending them something different than you’ve already outlined in the “screen share” portion of the script? If you are trying to simulate log in, you should have login be part of the prototype. The easy way to hack this is to take screenshots and drop those screenshots into the prototype. You don’t have to include the 2FA screens to make things go a little quicker. If you need an example, I can probably dig up one.
• I still think this section needs work. You have a bunch of great bullet points in the “Things to watch for” section, yet no actual questions on these points in this section. If you want to see how someone interacts with the name tag or navigates to military information, how are you going to evaluate that without the questions outlined?

Second task

• In the follow up questions, I would ask something like “how could we improve the process we just looked at?”
• What are your impressions of the mobile navigation? What does this mean? And what are you trying to evaluate here? I don’t think a user will know what you’re asking. Are you trying to ask evaluate whether they thought this section was easy or hard to find?

Third task

• I’m confused — in the first section, you had a bunch of “Things to watch for” notes on watching for how they navigated to different sections, yet those tasks are in the third section. Can you clarify what you are looking for in each section and adjust as necessary?
• Why is there another link to another prototype? Is this to simulate login again? I would just add those screens to the prototype for a more seamless experience.
• Again, in your “Things to watch for” section, you have a bunch of things to watch for that don’t have questions to match. We need to make sure everything is reflected in the questions so nothing gets missed. It’s hard to evaluate these scripts without that.

Post-task interview

• No on is going to know what you mean by “name tag.” That’s our internal name for it but will mean nothing to veterans. Also, this should be asked prior to the post-task interview questions. This section is for evaluating things as a whole, not specific elements.

[sshein]

Samara, in terms of the "2 links" - the general idea is that we want users to do their "wayfinding" on VA.gov in production, not the prototype, so that they can actually click wherever they want to look for the section. Once they click "login" or whatever is relevent, Tressa will direct them to the prototype

[sshein]

Also, I think it's fine to follow up at the end with questions about specific design elements that the user may not have mentioned or that was not discussed earlier. Agree that "name tag" is not going to work as a question though. Rather, I'd just show the user whatever we want to follow up on and ask specific questions about it. Personally, I think folks will just skip over the "nametag" area and there won't be much to talk about, but we'll see!

[sshein]

Tressa - for examples of putting links to prototypes and time estimates by tasks, see the Caregiver conversation guide. https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/caregivers/1010cg-mvp/usability-testing-jan2020/conversation-guide.md

[Samara-Strauss]

Samara, in terms of the "2 links" - the general idea is that we want users to do their "wayfinding" on VA.gov in production, not the prototype, so that they can actually click wherever they want to look for the section. Once they click "login" or whatever is relevant, Tressa will direct them to the prototype

Got it. Thanks for clarifying! This makes more sense now.

Also, I think it's fine to follow up at the end with questions about specific design elements that the user may not have mentioned or that was not discussed earlier. Agree that "name tag" is not going to work as a question though. Rather, I'd just show the user whatever we want to follow up on and ask specific questions about it. Personally, I think folks will just skip over the "nametag" area and there won't be much to talk about, but we'll see!

Yeah, to be clear — I don't think it's a problem to follow up on specific elements based on what came up in the actual call. It just doesn't make any sense to me to call the name tag out in the script in the follow-up section. This should be discussed earlier if we want feedback on it.

---

Question — I know we had talked about doing some sort of comparison of the new profile to the existing profile. Did this fall through the cracks, or did you ultimately feel this wouldn't be helpful? I think the original thought was getting some sort of comparison would help us validate whether they thought the paginated navigation or single-page navigation was clearer, but I'm OK if this was dropped because you determined it wouldn't be helpful.

[sshein]

If we have time I think it could be cool to show them the old profile at the end and ask their thoughts - for example on the caregiver guide I linked above, at the end, we ask

"I'm going to quickly show you the paper form (moderator shares screen with paper form). (if they have benefits) Do you remember if you filled out this form? If so, how did you do it? Together with the caregiver/veteran?

Based on going through this flow, would you prefer to fill out this form online or on this paper form? Why? How would this be different than filling out the paper form?"

So we could do something similar

[Samara-Strauss]

+1 to ☝️

[careyotto]

@tressaellen Speaking with @meganhkelley today, VSP would like to include questions/tasks about the FAQ and App Directory in usability testing, but are okay with not actually including those screens in the prototype. Examples of how we might do this are:

• After connected app task, ask was anything confusing? if yes, where would you look for answers? if no, imagine you did have concerns, where would you go for more information?
• Follow up with "How would you connect a different outside app?" to suss out if they find the link to the app directory, next to FAQs.
• Users may be confused with the term "outside app." We might want to nix the term "outside" and just say "app." If users start talking about MyHealtheVet or VA MobileApps, we could clarify with a follow up question re: non-VA apps. "Have you used any non-VA apps to access your VA health records?" "If you wanted to find new apps on your phone to use, where would you look?"

[tressaellen]

Okay, sounds good. I'll update the conversation guide with these.

[tressaellen]

Hey @Samara-Strauss and @sshein - i've updated the conversation guide if you'd like to take another look at it.

[Samara-Strauss]

Overall, this is looking so much better. Great job! I have just a few final notes.

Second task

• What was your overall impression of the mobile navigation? “Mobile navigation” still feels a little inside-baseball to me. Assuming you’re trying to assess whether they thought the profile was easy or hard to navigate from the mobile view, maybe ask something like “Did you find it easy or hard to switch between the different sections of the profile?” Or “Did you find it easy or hard to use the menu at the top of the page to find what you were looking for in this profile?” Or something word-smithed better than that.

Third task

• What would you expect to be in the 'Account Security' section? I think we should ask this, but I would also expect zero people to guess correctly what we have here since things like verifying your identity and 2FA aren’t typical things you’d find on every website. So, that’s all to say I don’t think the section is a failure if they don’t guess totally correctly. I think this question can be best used to assess if we should name the section something different.

Post-interview

• What are your overall impressions of the va.gov profile? Just be sure to clarify here that we are asking about the new profile and not the old one since they’ll have just seen both.

[sshein]

Cool. I'm not seeing time estimates, but assuming you're going to add them after the practice session :)

[Samara-Strauss]

Here are my notes from the practice session. Sorry they're a little bit messy:

General

• We may have issues with sound if they are using computer mic/speakers (because of sound breaking up or they might not have them). We should be prepared to help them call in.
• Let’s book an hour for these calls just in case. People likely will not be as fast as Shawna, and we want time for ask follow-up questions/probe.

Feedback

• Intro section — Might want to probe on what the form was? When? Don’t get lost in the questions but see if you can confirm what they are talking about.
• If they receive disability — did they get this by applying through the website? When? This also means they might have health care. Are they managing this online?
• Login — show the login flow for mobile
• We should add some language to explain that they are seeing mock data for Kim Washington. She’s made up and not real, and nothing about her is real. They should pretend she’s her.
• Home address: Don’t answer the question! Ask them what they think. If they are still stuck on it, you can say you’ll answer at the end of the session.
• Can we use a more real address for the mailing address?
• +1 to having them navigate to other sections in mobile to see how they navigate.
• “Personal information bar” – Let's just describe/point to the section instead of using that term since it doesn't really mean anything.
• For the connected accounts question, agreed with Shawna’s feedback — if they are totally stuck, you’ll need to figure out how to probe to get some information.
• When Shawna said she didn’t know what to expect, there could be follow-up here. How could we make it clearer what’s in this section? How could we improve this experience?
• Add login for desktop
• Ask about "service history"/"period of service" instead of "military information"
• Account security — make it clear they shouldn’t click on it until you say so. Also, totally missed opportunity to ask if this met expectations, if so, how/not?
• If the mishap with the staging profile happened in testing, how would you handle it? Just make sure to go with the flow, relax, and resolve the issue without drawing too much attention to it. If you don't make a big deal of a mistake, the veteran won't either.
• “What are you impressions of the desktop navigation?” They likely won’t know what you mean by "desktop navigation", We need to be more specific here and not use tech language.
• “What could we improve?” This needs more probing unless you’ve already talked about specifics ad nauseam earlier in the session.
• Weird questions at the end. People WILL ask these questions. If you can, help them! Direct them to the section of the profile/account where we link to that information. Just make sure they have an answer before hanging up, even if we need to send them to the helpline or another person.

[Samara-Strauss]

Hey! Some final feedback. Mostly, this is looking awesome. Great work! Just a few points for clarity.

First task

• Do you feel like you saw all of your personal information represented? Are you trying to assess whether information is missing? If so, I would stay away from "do you feel" language and be more direct — "Take a look at the 'personal information' section: what information would you expect to see here? Is anything missing?"
• What are your thoughts on having mailing address and home address? Maybe "Why do you think we show a mailing address and home address?"
• "Great, from here, could you show me how you would see what bank account you have connected to your VA profile?" I don't think veterans will think of their bank account as something they connect to a profile — they'll likely think about it as how they receive payments from the VA. I'd use language like that since it reflects the actual use case more closely.

Third task

• "Without clicking on it, what would you expect to be in the 'Account Security' section?" — You have almost the same question in the first task (question 5). I would cover the account security question in one section or the other. Right now, this feels too duplicative. FWIW, I would go with the wording you have in the third task.