[Discovery] Logged-in homepage: User research — Create research plan

[Samara-Strauss]

Background

Epic #7268

As part of the logged-in homepage discovery phase, we have decided to conduct user interviews/testing to help us better gauge user needs and expectations.

Tasks

• [x] Review overall research goals and homepage redesign questions
• [x] Create research plan
• [x] Share with team
• [x] Revise as necessary

Acceptance criteria

• [ ] Research plan is ready to be sent to Lauren to kick off recruiting

[andaleliz]

@Samara-Strauss, I worked on the plan with @sshein, and here's what we came up with.

Some FYI points + questions:

• I added some more questions to what you posted in Slack based on the product brief and original research needs document. I think answers to these will be helpful as we move forward, and the questions are manageable to include in this study.
• We're not sure on how to work in the co-design aspect of this. It would be helpful if we could discuss as a group in our meeting tomorrow. The Methods and Participants and Recruitment sections will need to be refined based on the outcome of our convo.
• I included the team listed in the product outline as observers. Anyone else you want to be sure we include?

[Samara-Strauss]

Hey! Thanks for sharing this. It's looking really good! My answers to your questions + some other thoughts are below:

I added some more questions to what you posted in Slack based on the product brief and original research needs document. I think answers to these will be helpful as we move forward, and the questions are manageable to include in this study.

This list looks good to me. One thought — I think evaluating What do people think the difference should be between the profile and dashboard/logged-in homepage? Should there be a meaningful difference? is going to be challenging. Let's talk about this tomorrow.

We're not sure on how to work in the co-design aspect of this. It would be helpful if we could discuss as a group in our meeting tomorrow. The Methods and Participants and Recruitment sections will need to be refined based on the outcome of our convo.

Yeah, let's definitely talk about this. One idea I had was using Mural to "build" a homepage by moving around cards with different tasks/benefits on them. But idk if that's a good idea or not, so we should definitely talk about what we want to get out of this section of the research to see if the co-design exercise is a good idea or if we should do something different.

I included the team listed in the product outline as observers. Anyone else you want to be sure we include?

You don't need to include Tze or Jen. Can you add Clarice? I don't think she'll be able to make it but I want to at least include her as optional. Otherwise, the list looks good.

Other questions or thoughts

• How many people are you looking to talk to? I didn't see this in the participants section and wasn't sure if that was left out intentionally or not. Let me know if you want to talk about how many people we need.
• When you say you are looking to talk to "enlisted or officer", are you talking about existing service members? They're extraordinarily hard to recruit. If you mean people who are veterans but were enlisted or officers when they served, I would clarify.
• For participants, I would add age requirements so that we get a good range of folks. If you want to talk to 1 – 2 folks who are under 35 or 40, I would say so. Otherwise, it's possible it's going to be an entirely older crowd. We love our older veterans, but I also don't want to miss out on people who haven't been with the VA as long.

[andaleliz]

@Samara-Strauss Thanks for the feedback! I intentionally left out the details on participants mostly because I felt like our decision on co-design approach would influence how many we needed - went ahead and filled out the rest of the criteria though.

I had something similar in mind for a co-design exercise too, so gtk we're on the same page if we do move forward with that approach. I feel like that approach would give us similar insights to a card sort though. We can talk through it today.

For enlisted vs officer, I did mean Veterans. That came from a recommendation from Jennifer Strickland, which I understood to be about rankings. I asked her for more info, and she clarified that it's really about the education level.

If I could, I would ask for 70% of the participants to have an enlisted service background, because they're the ones who struggle the most to understand all this stuff, for both reading comprehension reasons, as well as insecurity that prevents asking for help to understand.

I don't think reading comprehension is as big of a hurdle w/ a dashboard (compared to a content page explaining benefit rules, as an example), but I still think that is an important area for diversity. After looking at the form Perigean actually uses for registering participants, I see that they don't even ask Veterans about their rank, and so updated our criteria to be about education level.

[Samara-Strauss]

Great, thank you for all of this! No additional thoughts now — looking forward to our call in a bit.

[andaleliz]

I've updated the research plan to reflect what we discussed yesterday. Most notable changes (also did some general clean up):

• Updated method section to reflect our plan of co-design exercise in Mural with card sort as a back-up
• Updated participant criteria with number of participants and specific info about how many "at least X number" for the various categories
• Updated availability for pilot test w/ Veteran

@Samara-Strauss, please let me know if you have any further edits before we close this out. Thanks!

[Samara-Strauss]

This will be followed by a moderated co-design exercise. We've chosen this collaborative approach to gain insight into what Veterans expect to see on their logged-in homepage (versus their profile), and where they would expect to see these items. Engaging with participants in this way will also help us see any difference between what participants say, and what they actually do.

Since we haven't done a co-design exercise on this team before, I think a little clarification here could be helpful so Lauren feels comfortable approving this plan:

• This will be followed by a moderated co-design exercise. We will use Mural to help the veteran "create" their own personalized homepage by sharing our screen and conducting an interview in which we build a personalized homepage with their input. The interviewer (Liz) will manage all of the interactions in Mural since we think veterans would have trouble navigating this tool themselves. We think this is the most helpful method to gain insights into what people need on their homepage because it doesn't overly prime/influence them by showing them existing designs or a canned list of tools we think they might want on the homepage.
• Given that we have not run a co-design study before, we are going to run two pilot tests before we run the full research study to make sure this method works. If the co-design method doesn't seem like it will be as successful as we hope, we are going to have a card sort as a backup method. This card sort will present the veteran with a list of about 25 tools and it will ask them to organize those tools into two buckets: the dashboard or profile. While we think the card sort is a perfectly valid method regardless of the co-design exercise, we have chosen it as the backup method since it presents veterans with a canned list of options and doesn't allow them to start from a truly blank slate.

Experience with VA benefits and services (at least 3 people who have 3 or more benefits)

Just curious as to why you went with 3 benefits for this requirement. Totally fine leaving things this way but looking to understand this better.

To ensure inclusivity, we request some participants that have identified cognitive impairments and/or functional disabilities. Diagnoses that may align with this would be Traumatic Brain Injury (TBI), Post-Traumatic Stress Disorder (PTSD), Attention-Deficit/Hyperactivity Disorder (ADHD), Autism, and Vertigo. Other conditions may be autism, aphasia, dyslexia, dyscalculia, distractibility, memory loss, reading difficulties, non-native English speakers, low tolerance for cognitive overload, and intellectual/adaptive functioning challenges such as learning and problem-solving.

I would be specific and ask for at least 1 participant that falls into this category.

Pilot: Please indicate a date before your sessions begin for piloting your research. Which member of the design team will you pilot your research with?

I would clarify here that you plan to run two pilots: One with a member of our team, and one with an actual veteran, and that we are doing this to verify the co-design method works and will get us the insights we want.

---

Overall, this looks great, and I am SO excited to see this research happen. This is legit the most excited I've been about research in my two years on the team. It's going to be great!

[andaleliz]

Yay, I'm glad you're as excited as I am! This is definitely high up on my list of studies I've been Most Excited about.

I'll add the clarification about pilot participants, good point.

Just curious as to why you went with 3 benefits for this requirement. Totally fine leaving things this way but looking to understand this better.

I chose 3 because I think it's a better representation (than 2, or simply saying "multiple") of how things might be for Veterans who would have a lot of items on their dashboard and I'm interested to see how they start to prioritize when there are a bunch of items to address.

Since we haven't done a co-design exercise on this team before, I think a little clarification here could be helpful so Lauren feels comfortable approving this plan

Agree with you that the extra clarification is helpful.

• I'm going to add "The card sort can also serve as an option if for some reason Mural fails in a particular session." to your second bullet point.

• A comment on that bullet point:

This card sort will present the veteran with a list of about 25 tools and it will ask them to organize those tools into two buckets: the dashboard or profile.

As I've been working on the conversation guide, I feel we should intentionally not mention the word profile to avoid leading/biasing the participant and see if it comes up on its own. My hunch is that it will come up once the participant sees the pre-populated cards with things like "contact info, military history, etc).

OptimalSort's documentation says they have a hybrid card sort option where you can have predefined and "fill in the blank" categories, so I was thinking we'd have one called "homepage" and one blank (instead of labeling it as "Profile") in case they feel like a card belongs somewhere else. Same with the Mural board - we'd start w/ two "pages" set up.

Thoughts?

[Samara-Strauss]

I chose 3 because I think it's a better representation (than 2, or simply saying "multiple") of how things might be for Veterans who would have a lot of items on their dashboard and I'm interested to see how they start to prioritize when there are a bunch of items to address.

This makes a ton of sense, and good thinking! I think it will be super helpful to see what people do when they are forced to prioritize different benefits/tools.

As I've been working on the conversation guide, I feel we should intentionally not mention the word profile to avoid leading/biasing the participant and see if it comes up on its own. My hunch is that it will come up once the participant sees the pre-populated cards with things like "contact info, military history, etc).

OptimalSort's documentation says they have a hybrid card sort option where you can have predefined and "fill in the blank" categories, so I was thinking we'd have one called "homepage" and one blank (instead of labeling it as "Profile") in case they feel like a card belongs somewhere else. Same with the Mural board - we'd start w/ two "pages" set up.

Agree 100% that we should avoid mentioning the word "profile" if we can to avoid biasing them. Great idea, and I'm glad that Optimal Workshop can accommodate this!

[andaleliz]

I just published the latest version with all of the updates incorporated. We're getting close to being able to submit everything! Any other changes you'd like to see?

FYI - I'm going to add links to the a mural board and the card sort to the doc before we submit, so I'd like to leave this issue open until those are added. I'm sure it'll be helpful for Lauren to see those.

[Samara-Strauss]

Looks great! Ship it!! :D