[Discovery] Logged-in homepage: Stakeholder interviews — Synthesis: Round 1 of interviews

[Samara-Strauss]

Background

Epic #7085

We will be conducting stakeholder interviews for the logged-in homepage discovery per our research plan.

Tasks

• [ ] Synthesize research findings from round 1 stakeholder interviews

Acceptance criteria

• [ ] Round 1 research findings have been reviewed and synthesized

[andaleliz]

Linking to folder with detailed notes, and mural board to start organizing themes

cc @sshein to start watching this

[Samara-Strauss]

This mural board is amazing and I am obsessed with it 😍

[andaleliz]

I love it too! Full credit goes to @sshein for coming up with this as a template for research session notes!

[sshein]

I'll pass most of the credit on to Mural, since I just adapted one of their boards haha ;)

[Samara-Strauss]

Creating two tickets for synthesis since synthesis will likely cover different sprints. Leaving this ticket for round 1 of interviews and using #7679 for round 2 interviews

[Samara-Strauss]

Here's my feedback on the Trends/Insights section so far. First, everything is looking great for the most part. I think you got the main conclusions we talked through the other day. I know this is a lot of data to go through, so nice job tackling it all!

Some notes:

• We're not providing Veterans with an experience that reflects what we know about them once they log in — About to get nit-picky here, but I think this matters. We do show what we know about people on My VA, but (1) not effectively and (2) I think people were more talking about the homepage itself. To add insult to injury, the navigation barely changes, so it's hard to differentiate the logged out from logged in state. I would update the conclusion here to be more specific and reflect this. Something like "The logged in homepage and navigation largely stay the same when people log in, which gives little sense of personalization, doesn't effectively present what we know about veterans, and likely doesn't meet user expectations."
• We're not effectively guiding Veterans — Also being nit-picky about specificity here. I would maybe change to something like "We're not effectively guiding Veterans to tasks and tools from the logged-in homepage or via the navigation."
• A Veteran's logged-in homepage should be a data driven experience that reflects what we know about them — Chef's kiss Love this.
• The logged-in homepage should guide Veterans in a meaningful way that builds trust and helps them complete tasks — I agree with everything here, but I think we should separate out the "building trust" part. That really gets down to more of an issue with data integrity across the VA and is less about wayfinding to specific tasks. If you want to include trust/data integrity, I would actually do this with the first point in this section ("A Veteran's logged-in homepage should be a data driven experience that reflects what we know about them") since that has to do more with data and what we know about people.
• An abundance of stakeholders will prevent us from making incremental improvements in a timely way. For this risk, I would present it as a "might/could" statement just because this wording makes it sound like we definitely think this will happen. So, something like "Having many stakeholders could prevent us from delivering incremental improvements in a timely way."
• The logged-in homepage will become a dumping ground. Same here, "The logged-in homepage could become a dumping ground."
• Metrics — I agree with your conclusions here, but for the report, I think the most important info here is a list of what metrics we might use to measure these things.
• Comparative experiences — Same point as metrics. For the report, we maybe need a couple sentences to explain why these experiences are a good analogy for VA.gov, but then just feel free to list what those experiences are.

Great work!!

[andaleliz]

@Samara-Strauss thank you! I really appreciate the feedback. Just so you know - I don't perceive it as nit-picky at all. Details matter, and the suggestions you have make some important differentiations that I agree with.

I'll be sure the write-up reflects these comments. Will share here soon. Thanks!

[andaleliz]

@Samara-Strauss here's the draft in progress for the synthesis. I still need to complete the details for all but two findings, but have a few comments/questions:

1. Regarding your suggestion for the first problem statement - (this is also possibly nit-picky) I removed the "likely doesn't meet user expectations". I do agree that it doesn't, but that didn't exactly come up in these interviews, and I think that we'll be able say that more definitively in our larger report once we factor in past research + conduct user interviews.

2. Recommendation section - I'm thinking it makes more sense to provide recommendations on the overall, final study based on everything we learn instead of making them on each individual study. What do you think?

3. Can you take a look at the metrics section and see if that lines up with what you had in mind? I definitely have room to grow in the world of analytics so not totally sure if this is the right approach. I think it'd be helpful to have an actionable list we could give to someone who is, who could then help us measure these things as we continue forward.

And of course, welcome any other feedback you have if you feel like reading through the draft state :) Thank you!

[sshein]

Hi hi :) Instead of "Problems with the current UX" can we call it "Stakeholder perceived problems with the current UX" or something? Since this isn't based on user testing. And can "Goals" be written out so it's more clear? e.g. whose goals? "Project Risks" is also "Perceived project risks" or something, since none of them are actually verified risks. Agree that recommendations might not be necessary at this point. Looking forward to seeing more :)

[Samara-Strauss]

Hey @andaleliz! This is looking great. My feedback/answers to your questions are below:

My feedback

• Research goals: Can we add something about understanding stakeholders' perceived pain points with the current logged in homepage/My VA?
• Key Findings: For each statement, can we consistently end it with a period OR not end it with a period? There are some statements with a period and some without. This is the most minor of details but please indulge my persnickety brain. :D
• Key findings and details: Can we combine this into one section? IMO, it's not necessary to separate out the high-level findings into their own section since they are repeated in the "details" section and are bolded/easy to scan.
• Metrics: Formatting-wise, I would bold the key headings ("Measuring interactions can help.." and "Measuring authenticated sessions...").
• Metrics: For the second question, I think "user satisfaction" is clearer than "user sentiment." Also, I would change "number of profiles" to "number of people who verify their identity (LOA3)" and "number of authenticated sessions" to "number of users who log in."
• Details:

Since the logged-in homepage and the majority of navigation don't change when you log in, we're falling short in giving Veteran's a personalized landing pad that makes them feel welcome, and missing an opportunity to provide them with a more targeted experience that reflects who they are. Also, the content and actions presented to them are redundant across the logged-in and logged-out experiences, so the value of logging in is easily missed. A Veteran can find personalized content in MyVA, but the Personalization 2.0 research conducted in the summer of 2019 taught us that users either don't know about it, or don't have much of a need for it.

• Suggested edits below:

Since the logged-in homepage and the majority of navigation don't change from the logged-out experience, we are falling short in providing a more targeted experience that reflects Veterans' circumstances and needs, guides them to tasks and tools, and makes them feel welcome. Without a more tailored authenticated experience, Veterans don't have enough incentive to log in.

-Comparative experiences

Conversations with stakeholders validated that banking and finance experiences closely match the use case for VA.gov - users can transact on the site, submit applications and receive a level of personalization that creates a strong differentiation in the logged-out to logged-in experience. These experiences, when done well, also allows a user to easily understand their status/standing, and receive notifications.

• Suggested edits below:

Conversations with stakeholders validated that banking and financial sites serve as valuable models for VA.gov. When done well, financial and banking sites provide a level of personalization that create a strong differentiation from the logged-out experience. They allow users to complete a number of transactions, including submitting applications, viewing account statuses and updates, and receiving notifications. They also keep marketing content separate from the logged-in experience.

Your questions

Regarding your suggestion for the first problem statement - (this is also possibly nit-picky) I removed the "likely doesn't meet user expectations". I do agree that it doesn't, but that didn't exactly come up in these interviews, and I think that we'll be able say that more definitively in our larger report once we factor in past research + conduct user interviews.

You're totally right. All of this sounds good to me!

Recommendation section - I'm thinking it makes more sense to provide recommendations on the overall, final study based on everything we learn instead of making them on each individual study. What do you think?

I don't think we necessarily need recommendations the way we think of them traditionally, but maybe some actionable takeaways (is that just another way of saying "recommendation" 😂?). What I mean by this - to me, some action items coming out of this are (in no particular order)

• I should regroup with DEPO stakeholders on how to involve VA stakeholders in this project.
• You should include BoA, USAA, Delta, Google, Apple, etc. in the comparative analysis.
• We should think more broadly about how to guide veterans in the logged-in experience — how does a logged-in navigation factor into this project?
• We'll want to work closely with VSP to build-up/scale the visual design system to accommodate authenticated tools. This redesign is a good playground for that.

...and there are probably other things that are not quite recommendations but more considerations that will affect our approach either immediately or in the future. Happy to discuss this more if it's helpful to put our heads together.

Can you take a look at the metrics section and see if that lines up with what you had in mind? I definitely have room to grow in the world of analytics so not totally sure if this is the right approach. I think it'd be helpful to have an actionable list we could give to someone who is, who could then help us measure these things as we continue forward.

I think the overall structure here is good. No additional feedback besides what I already said above.

[sshein]

FYI regarding this suggestion, @Samara-Strauss "Key findings and details: Can we combine this into one section? IMO, it's not necessary to separate out the high-level findings into their own section since they are repeated in the "details" section and are bolded/easy to scan." - I usually tell my folks to do the 2 sections for the stakeholders that get easily overwhelmed and literally only want to scroll for a tiny bit before bouncing. But not crazy tied to it if you'd rather not do that here.

[sshein]

Agree with the comment about Action items / Takeaways now that i see examples :)

[Samara-Strauss]

I usually tell my folks to do the 2 sections for the stakeholders that get easily overwhelmed and literally only want to scroll for a tiny bit before bouncing. But not crazy tied to it if you'd rather not do that here.

Ah, ok, noted! I realized this was a standardized template right after I posted that comment when I reviewed Tressa's profile research write-up. Totally fine to leave it the way it is.

[andaleliz]

Thank you both for the feedback! Almost all makes sense. A couple of questions on metrics for @Samara-Strauss :

Metrics: I would change ... "number of authenticated sessions" to "number of users who log in."

I see these as two different metrics - we might need both. When this came up in interviews, I was thinking stakeholders would want to understand how many times someone is viewing content/interacting with VA.gov while logged in (would include the same person logging in repeatedly) vs a count of individual people who log in. Personally I think they'd both be interesting to track. What do you think?

Second question, just for my own understanding about:

Number of people who verify their identity (LOA3)

I don't know anything about LOA3 (googled it and discovered it is also an acronym for a video game!). Does this page describe that you're referring to? Is this the check that takes place when someone tries to set up an account on va.gov (and not the same thing as entering a password/2fA when logging in each time_?

[Samara-Strauss]

I see these as two different metrics - we might need both. When this came up in interviews, I was thinking stakeholders would want to understand how many times someone is viewing content/interacting with VA.gov while logged in (would include the same person logging in repeatedly) vs a count of individual people who log in. Personally I think they'd both be interesting to track. What do you think?

Great point! Yeah, you're right — that's a small but important nuance that I hadn't considered. Let's definitely count them as two separate metics.

I don't know anything about LOA3 (googled it and discovered it is also an acronym for a video game!). Does this page describe that you're referring to? Is this the check that takes place when someone tries to set up an account on va.gov (and not the same thing as entering a password/2fA when logging in each time_?

Yep! That page is a good find and does describe what I am talking about. Sorry for the inside baseball language. There are two types of LOA we account for on VA.gov — LOA1, which means someone has signed up/signed in, but they have not verified their identity, and LOA3, which means someone has signed up/signed in AND gone through a process to prove they are who they say they are. For LOA1 users, we can't show them much beyond applications they've started while signed in because we don't know if they are who they are claiming to be. We require people to be LOA3 on VA.gov in order to view most things — their profile, health information and tools, disability information and tools, etc. Basically anything that involves us showing you PII/health info/personal data requires that you prove you are who you say you are (aka being LOA3) before we show it to you.

[andaleliz]

@Samara-Strauss and @sshein this is ready for your eyes again! All the details have been added, and there are many - not expecting an immediate reply. Thanks in advance for your feedback!

[sshein]

This is great. Great job.

A few small wording change requests:

1. Can you update the titles in "Key findings" to match the titles in "Details of findings"? e.g. "Perceived problems with the current UX"
2. "The logged in homepage and navigation largely stay the same when people log in" seems a little redundant. Maybe "The homepage and navigation largely stay the same when people log in"
3. Instead of "While myVA does guide Veterans to tasks and tools, it's not used much" let's say "While myVA does guide Veterans to tasks and tools, it's not used very frequently"
4. "...whether that's 2 or 20 tasks, and if they are new Veteran, or if they've have been using the VA for decades." should be "...whether that's 2 or 20 tasks, and if they are new Veteran or if they've been using the VA for decades."

Additionally, in "Stakeholder Goals" should we say anything about the differentiation Steve K. mentioned? Like We want to show veterans status on their current items but we also want to lead them to particular applications that might be relevant to them based on what we know about them?

I really like the perceived project risks section

Happy weekend!

[andaleliz]

Thanks for the feedback, @sshein ! Will definitely make those wording changes.

Yeah, I think I should update the goals to more specifically include what Steve said (I'm fairly certain a few others mentioned something like that too). My intention was that the first sentence under the first stakeholder goal would cover that, about surfacing most relevant information related to...

By leveraging the data we have for each Veteran profile, we'll be able to surface the most relevant information related to benefits, claims, and preferred VA facilities to each individual.

What do you think of the suggestion below? Also, unrelated: when writing, should I refer to the site as va.gov or VA.gov?

A Veteran's logged-in homepage should be a data driven experience that reflects what we know about them.

By leveraging the data we have for each Veteran profile, we'll be able to surface the most relevant information related to current benefits (including status updates), applications for additional benefits they're eligible, and preferred VA facilities. Providing information and updates that reflect their specific situation, highlighting how they can further use their VA benefits, and empowering them to complete tasks quickly and efficiently, will likely also help us build Veteran trust in the VA.

[Samara-Strauss]

What do you think of the suggestion below? Also, unrelated: when writing, should I refer to the site as va.gov or VA.gov?

I usually go with VA.gov since VA is an acronym.

By leveraging the data we have for each Veteran profile, we'll be able to surface the most relevant information related to current benefits (including status updates), applications for additional benefits they're eligible, and preferred VA facilities. Providing information and updates that reflect their specific situation, highlighting how they can further use their VA benefits, and empowering them to complete tasks quickly and efficiently, will likely also help us build Veteran trust in the VA.

I want us to be careful about how we talk about suggesting benefits to veterans, because while that's the dream, it is extraordinarily complicated to do reliably and to do well. It's true that if we had a robust set of data about each veteran, we'd be able to suggest benefits they might be eligible for, but this is not something we're going to do in a near term future. TL; DR — I don't want to imply that benefit recommendations is something we feel we're ready for, especially in a V1 of this project.

Also can we flip the structure of the sentence starting "Providing information..." as to not bury the lede? So, "We can build Veteran trust by..."

[Samara-Strauss]

Overall

1. Overall, I think the content is generally there in the report. I've left some feedback below that addresses content-related things specifically.
2. However, the overall grammar/wording/flow could use some work. A proofread by someone on the Ad Hoc or GCIO side of things would be helpful (though you might want to drop it into Google Docs to do that). Additionally, read through the report and think about what words really need to be there and what words don't. I know I'm super wordy, BUT for reports, I like to go with the opposite of how we all wrote in college — the shorter, the better. In general, our point is this — we need task-based navigation, we need notifications, we need personalization. That's mostly it. We don't need to get wordy with explanations. If a high-level point needs a one-sentence blurb below it or even no blurb, great, let's go with that.

As usual, happy to answer questions!

Detailed feedback

While myVA does guide Veterans to tasks and tools, it's not used much (we learned this in our Personalization 2.0 research study), and the visual experience hasn't scaled in such a way that it helps Veterans find what they're looking for quickly.

• Can you change it to "My VA" throughout the report?
• This is picky, but that page doesn't really "guide" (which I can totally say since I own it 😆). I would say it elevates personal information (yay!) but does it poorly. So maybe something like "While My VA does elevate personal information..."

We can do better than presenting a stacked list that requires significant mental processing and scrolling to understand all that is available.

NICE! This gets at the point well.

Stakeholders view the logged-in homepage as place to elevate the tools their teams build, and as a centralized place to elevate notifications and updates to users.

I want to reframe this. The goal is not just stakeholders elevating their tools — it's about meaningfully guiding veterans to tools they need. So, yes, stakeholders would have a place to elevate tools, but only if those tools actually need to be on the homepage. "Stakeholders can use the homepage as a place to elevate valuable tools, status updates, and alerts that Veterans need."

The logged-in homepage is valuable real estate for reaching Veterans, and stakeholders commented throughout interviews that every team/business line will want their tools featured there. Concerns were raised about conflicting information, and the struggle to find balance between Veteran and business priorities. Using data and personalization to inform what Veterans see on their logged-in homepage will help us keep this space decluttered and useful to Veterans.

You mention "conflicting information" here, which feels like a separate point about data integrity. I would take this out or clarify.

Volume of calls to call center related tasks available through the logged-in homepage

This is confusing to me. I know what you're trying to say, but I think it needs clarification. "Whether the volume of calls to call centers changes as a result of the new logged-in homepage."

Volume of manual and in-person submissions related tasks available through the logged-in homepage

I don't know what this is trying to say.

Number of authenticated sessions Number of users who log in

We need to clarify the difference here because even I don't remember 😆

[andaleliz]

@Samara-Strauss, thank you! No questions, all the feedback is clear. I'll incorporate your feedback before requesting a proofread later today.

[andaleliz]

@Samara-Strauss the final version is ready for your review. I will hold off on closing this out until you confirm it meets your expectations. Happy reading :)

[Samara-Strauss]

@Samara-Strauss the final version is ready for your review.

Is it weird I got giddy reading seeing that the final version was ready? Probably 😂but I did.

Honestly, this is exceptional. A couple nitpicky things but besides that, this is good to go. I do not need to review again:

Stakeholders feel Veterans rely on the Top Tasks on the VA.gov homepage to get to the tasks and tools they're looking for rather than going to My VA or using the main navigation.

"Stakeholders feel" — this has been seen in user research, so maybe we want to clarify that this is an observed insight and not a feeling/hypothesis. "In user testing, stakeholders have observed that Veterans rely on the Top Tasks...."

The logged-in homepage needs to be a place users can easily and efficiently find what they're looking for, whether that's 2 or 20 tasks, and if they are a new Veteran or if they've been using the VA for decades.

Cleaning this up slightly — "The logged-in homepage needs to be a place users can easily and efficiently find what they're looking for, whether they are a new Veteran or if they've been using the VA for decades, or if they are presented with 2 or 20 tasks."