[Research] My VA - Payments Information & On-site Notification: Create Conversation Guide

hdjustice commented 2 years ago

Background

While we are requesting a Midpoint review, we should start the conversation guide for adding a new Payment Information section and an On-site Notification to My VA dashboard.

Tasks

[x] Create a conversation guide for the new Payment Information section on My VA.
[x] Create a conversation guide for the new On-site Notification on My VA.
[x] Send to Samara for review
[x] Confirm Research plan is completed - Ticket 33295
[ ] Send a Slack message to Lauren Alexanderson (Lalexanderson) with both the Research Plan and Conversation Guide

Acceptance Criteria

[ ] A draft conversation guide is available for review and use in Veteran User Research sessions for this initiative.

hdjustice commented 2 years ago

updated ticket to add On-site Notification scope, as the research and conversation guide efforts will be combined to include both projects.

andaleliz commented 2 years ago

@hdjustice I'm adding the research plan ticket as a blocker to this- I want to be sure we're aligned on research goals before I write the conversation guide.

andaleliz commented 2 years ago

@tressaellen here's a draft of the conversation guide - thanks in advance for your review and edits. Since I'll be out, it would be wonderful if you could tag Samara for her review once you think this is ready to rock. I'd love to have this ready to submit the first week of January, but I'm anticipating I will be busy with profile research that week. So I extra-appreciate your help :)

A few notes:

I went ahead and made a quick desktop/mobile prototype, as I'm going to guess the LOE will be too high for Taylor to code this out. Bummer because it'd be so great to be able to include AT users! next time. I think the only interactivity we really care about is the alert. LMK if I'm missing something there or if we need to update anything there.
We have a research question "When does a user expect to see an alert about a new debt?" and I'm not totally clear on the purpose of that question.
- Are you wondering overall, in their experience with the VA, or are you asking at what point after they log in do they expect to see an alert about new debt? It'll be helpful for me to fully understand that for facilitation purposes.
- Or, you can remove it from the convo guide if we don't need to ask that anymore.
I didn't include a post-task interview section because I think the tasks get at everything we need to know.
I added "goals of this task" to each task to aid the review process. I know sometimes it's not totally clear why I'm asking the questions I'm asking people in these guides.

Thanks so much for your feedback and help!

tressaellen commented 2 years ago

Hello @andaleliz - this looks really great! A couple thoughts.

We have a research question "When does a user expect to see an alert about a new debt?" and I'm not totally clear on the purpose of that question.

I am wondering at what point after they log in do they expect to see an alert about new debt. That's more on the onsite notification side of things, but I think it'll help us shape the MVP. I updated the wording around that question to better reflect this.

As far as the prototype goes - I think you're right. We're mostly focused on the alert as far as interactivity goes. For everything else, I think we're okay to just ask where they expect those links to go to make sure we have the wording right.
Usually for these logged-in tests we have the user pretend to be whatever user (Kim Smith) shows in the prototype. This hopefully helps people imagine that they're seeing their information and not get caught up in what might work for someone else. I added a few sentences around that into the conversation guide.
In most of the sections I added 'What do you find confusing?'/'What works about this?'/What doesn't work about this?', just so we can account for anything that isn't working since we're not doing a post-task interview.

First Task

I added 'I'd like you to imagine that new debt has been added to your account- after you log-in, when would you expect to see that information?' just to see if people have thoughts around how they expect that onsite notification to work. (We touched on this in the previous action items research, but it doesn't hurt to include it here). It might be confusing to ask people at this point in the conversation, so only bring it up if it makes sense.

Second Task

I added the following questions to make sure we have the right links/wording.

Say you came to this page looking to see if your most recent Compensation and Pension deposit had been made. How would you do that?
Say you wanted to see all of the payments that the VA had made to you, how would you do that?
Imagine you wanted to update your bank information that the VA has on file. How would you do that?

Third Task

I added the following questions.

'What would you change about these call-outs?'
'What are your thoughts on the information shown in those call-outs?' (and removed 'Is that information important enough to you that it should be highlighted like that?')
If you were to click on 'Manage your VA debt', what would you expect to happen?
Say you clicked on 'Manage your VA debt', viewed your debt, and then came back to this page, what would happen to this notification?
Say you paid your VA debt, what would you expect to happen to this notification?
Can you scroll down to the 'Benefit payments and debts' section. What would happen if you were to click on 'Manage your VA debt' here?

I think we're about there! Thanks for this draft!

tressaellen commented 2 years ago

Hello @Samara-Strauss ! I think we're ready to have you review the conversation guide for the payment information research. Liz started the draft and I just finished making some edits. Thank you!

Samara-Strauss commented 2 years ago

Warm-up questions

As part of the scheduling for this session, we asked if you had either received payments from VA or incurred education, compensation or pension debt. Can you tell me how you got info from VA about updates on those payments or debts?

I saw Randi post something in the VA Notify channel that made it sound like the debt tool might only handle education debt right now. Can we confirm whether that was an accurate read of her comment or not? If they don't handle disability or pension-related debt right now, we shouldn't mention this (and if they do, then no changes here).

What kind of information about your payments and debts did you get?

What does this mean? The wording is confusing.

Tell me what you know about My VA.

Users are likely going to have no idea what the dashboard is by name, so let's either remove this question entirely, OR assess later in testing whether they have seen the page before.

First Task: Wayfinding for payment information - 3 minutes

Great! No changes.

Second Task: Review My VA and financial info - 8 minutes

You will already need to be logged into staging. Please DO NOT GIVE OUT STAGING USER INFO IN SESSIONS. That is a huge security risk. If needed, log in on your computer and give the participant keyboard and mouse controls.

Take a look at this page, and before you click on anything, tell me about what you see here.

If needed:

What do you think this page for? What stands out to you on this page? What is confusing about this page? Imagining that this was your personalized page, talk to me about how this lines up with information that is important to you.

If needed: How would you rank the sections you see here, in order of importance to you?

Can we put all of these at the end of this section? I think we should evaluate payment info before general thoughts on the page.

Third Task: Review onsite notifications - 5 minutes

How are you going to get the debt notice to show/trigger during the session given you are using a FE prototype? Totally fine if it is there from the beginning, just know you'll have to get it "reset" for each session if you are having them dismiss the alert.

In general, if you have a new notification on VA.gov, how would you know about it?

I am confused as to why this is being asked after you've already had a lengthy conversation on notifications. Can you clarify as to why this is at the end and not the beginning of the conversation?

tressaellen commented 2 years ago

Thanks so much for the feedback @Samara-Strauss ! I'm going to go through and update the questions/wording feedback, and I'll hand off the execution feedback to @andaleliz since she'll be the one running the sessions.

Warm up Questions

I saw Randi post something in the VA Notify channel that made it sound like the debt tool might only handle education debt right now. Can we confirm whether that was an accurate read of her comment or not? If they don't handle disability or pension-related debt right now, we shouldn't mention this (and if they do, then no changes here).

I just double-checked with the debt team - and the debt portal DOES INCLUDE disability compensation + pension debts as well as education debts.

What kind of information about your payments and debts did you get?- What does this mean? The wording is confusing.

I updated this to: You mentioned that you receive updates on your payments or debts via insert what the participant said. What information about your your payments and debts is included? Was that helpful? Was anything missing?

Tell me what you know about My VA.

I added a post-task interview and included this question.

Second task: Review My VA and financial info

You will already need to be logged into staging. Please DO NOT GIVE OUT STAGING USER INFO IN SESSIONS. That is a huge security risk. If needed, log in on your computer and give the participant keyboard and mouse controls.

^ this is a great point and we definitely don't want to have any security issues. I'll let @andaleliz figure out the best way to do this, since she'll be running the sessions.

Can we put all of these at the end of this section? I think we should evaluate payment info before general thoughts on the page.

I included all of those questions in the post-task interview. This is what it currently says:

Now that we've walked through this page a bit, let's talk about your thoughts on the overall page.

Have you ever used this dashboard before on My VA?
What do you think this page for?
What stands out to you on this page?
What is confusing about this page?
What information is missing on this page?

How are you going to get the debt notice to show/trigger during the session given you are using a FE prototype? Totally fine if it is there from the beginning, just know you'll have to get it "reset" for each session if you are having them dismiss the alert.

I'll also hand this one off to @andaleliz as well

In general, if you have a new notification on VA.gov, how would you know about it? - I am confused as to why this is being asked after you've already had a lengthy conversation on notifications. Can you clarify as to why this is at the end and not the beginning of the conversation?

I removed this question.

__

I think that covers the feedback you had, let me know if there's anything else I need to update!

Samara-Strauss commented 2 years ago

Sounds good -- thank you!

andaleliz commented 2 years ago

Hi @Samara-Strauss Thanks for all your feedback! And thank you @tressaellen for taking care of all but 2 updates.

You will already need to be logged into staging. Please DO NOT GIVE OUT STAGING USER INFO IN SESSIONS. That is a huge security risk. If needed, log in on your computer and give the participant keyboard and mouse controls.

I can log in on my computer and pass control, but I think that somewhat defeats the purpose of being able to inclusively recruit since my computer may not have the needed accommodations for a participant using AT. Let me chat with our a11y specialists about how to best navigate this. I didn't expect that the prototype would be behind auth, which in retrospect, I realize I didn't specify or ask about. I figured it could be an HTML page somewhere on our site that we could share a link to without any problem.

If it turns out it is an a11y problem we can still use what Taylor built and I'll pass control on zoom.

How are you going to get the debt notice to show/trigger during the session given you are using a FE prototype? Totally fine if it is there from the beginning, just know you'll have to get it "reset" for each session if you are having them dismiss the alert.

Taylor is working to make the alert dismissible but reappear if the page is reloaded, which is how it would behave in a UX Pin prototype too. A reload will essentially be our "reset".

andaleliz commented 2 years ago

Oh, as an aside, I'm curious to know why it's a huge security risk to share a staging login user's information with a participant if it's a fake user. I naively thought it wouldn't matter since it's not real data and we could (theoretically) change the PW after the sessions are complete. Any insights for a security noob? Thanks!

Samara-Strauss commented 2 years ago

@andaleliz tbh I am not sure, though we could ask in the #vsp-identity channel. However, I know this is a no-no -- this is why staging user info/passwords have to be kept in the private repo and why passwords have to be changed if staging user info is shared in our public Slack. I'm not 100% sure why it's a problem, but I am 100% sure that it is a problem.

andaleliz commented 2 years ago

Joe Niquette shared some excellent reasons for the no-no in this slack thread:

The access to existing test accounts is the concerns as it can be difficult to maintain expectations of what those accounts are built for, what features are enabled, etc. Security wise, its due to availability. If a person were to accidentally save the creds for a test account and start entering in their real data through our claims system then that would be a PII nightmare.

He goes on to say:

all the existing test accounts are used by dev teams and wouldnt be able to be deleted or arent expected to be ephemeral

Just want to give an update that I'm doing some final digging into how testing would work on staging with logging in before calling this final and being ready to submit. I think we are still on track to submit everything to Lauren tomorrow to kick off recruiting.

Samara-Strauss commented 2 years ago

Thank you for this! I really appreciate you digging into the details here.

andaleliz commented 2 years ago

Samara and I spoke today and she gave the green light to move forward with testing on production per Cody's suggestion in slack (where we also confirmed there are no security concerns with this). I have also talked with @tmitchellgcio (slack thread) about the importance of ensuring this stays behind the flipper, and our timeline. Lastly, I confirmed with the collab cycle team (slack thread) that no additional reviews are required if we are testing on production vs a prototype.

🎉 we can move forward with inclusive recruiting! Bonus that we can get general feedback on actual My VA from users, instead of a static prototype.

Taylor feels he can submit his PR Thursday or Friday, so we should still be able to run our pilot test before we start sessions on 1/12. We have the prototype as a back up plan if for some reason that doesn't work out.

I've updated the conversation guide to include:

A reminder to stop recording once the user logs into production, to minimize PII issues.
Language to confirm with participants that we will be able to see their actual personal information once they log in, and that we will not be asking them to make any changes that impact their information or benefits.
"If using prototype" scenarios as a reminder for the facilitator, in case we need to default to that
A shift to the prototype during task 3 for participants who do not have debt so we can still get onsite notification feedback from each participant.

I've also updated the research plan in the following ways:

It now reflects that we're testing on production, won't record PII, and have the prototype as a back-up plan.
Recruitment criteria has been updated to require login email prior to session so we can prepare to flip the feature.
Criteria has been updated so all participants must have payments, and 50% of participants must have debt.
Increased target from 5 to 8 sessions to increase opportunity for diversity and inclusion, and ensure we get at least 4 participants who have debt (though we can show the prototype if that can't be met).

I can't think of anything else that we need to do to submit this and kick off recruiting but it's nearing 5pm and my brain is likely tiring 😆 .

@Samara-Strauss please let me know if you think this is good to submit for Shane's review :) (TIL Lauren no longer reviews research plans)

Samara-Strauss commented 2 years ago

@andaleliz this is incredible work. It's good to go IMO!

andaleliz commented 2 years ago

@Samara-Strauss thank you, and HOORAY! I will send a message for review first thing in the AM.

department-of-veterans-affairs / va.gov-team