Research: Prep tasks: VBA

[jilladams]

Prototype ticket: #13960

Tasks

Alexis S90 - 8 points total for S90

• [x] Draft high level timeline including research sessions - which days, and identifying timeslots that are compatible for PT / CT
• [x] Schedule as PDF attached to ticket
• [x] Review partial prototype and recommend revisions to the conversation guide

Other coverage S90

Prototype review & research plan prep

• [x] Review prototype and update conversation guide as needed - Cindy
• [x] Update research plan and conversation guide with current timeline and protocol updates, based on actual prototype - Cindy
• [x] Share research plan and conversation guide with team and Michelle for feedback and approval
• [x] Share research plan and conversation guide with research-ops for approval and recruiting

S91 - 13pts

Week of 8/14

• [x] by 8/18? Review Perigean's recruitment survey before attending Kickoff meeting - Dave & Alexis
  • (Note: please check for the full text and all options for every screening question listed in the research plan)
• [x] Make note-taking spreadsheet - Cindy
• [x] Create a Mural board for research debrief sessions - Cindy
• [x] Schedule meeting for 8/21 - Cindy talk through notetaking spreadsheet, Mural board, and process for debriefing and synthesis
• [x] Recruit pilot participants from Ad Hoc - Cindy
• [x] Schedule pilots when Alexis/Cindy/Dave can make them + Jane, Michelle (8/22 only) if possible - Cindy/Dave

Week of 8/21

Research transitioning - Cindy

• [x] Share links to sample research reports from Income Limits: topline report, full findings report, and readout presentation - Cindy
• [x] Share links to sample notes in github and Platform's instructions for sharing with Perigean for notetaking - Cindy
• [x] Give overview of note-taking spreadsheet and mural board - Cindy - During 8/22 call

Pilots - Cindy/Alexis/Dave

• [x] Schedule debrief sessions after each pilot - Cindy/Dave/Alexis - During 8/21 call
• [x] Schedule prep time before and debrief time after each pilot - Cindy/Dave/Alexis - During 8/21 call
• [x] Test Zoom login from Perigean - Cindy & Alexis
• [x] Confirm the day before with all pilot participants and ask to use smartphone - Cindy
• [x] Change display name of Perigean Zoom account to "Cindy (Moderator)" and then "Alexis (Moderator)"
• [x] Run pilots - 1 Cindy, 2 & 3 Alexis
• [x] Debrief pilots - 1 Cindy, 2 & 3 Alexis
• [x] Send thank you slack msg to pilot participants - Cindy

Recruitment / scheduling sessions - Dave/Alexis

• [x] Review recruited participant list for requirements met
• [x] Follow-up with Perigean re participant list, if needed
• [x] Make sure all research sessions are in calendar
• [x] Make signup sheet for observers, and share with Facilities team, stakeholders (incl. Michelle Middaugh, Amanda Klausmeier) to sign up - Dave
• [x] ~Share signup sheet with Perigean so they can~ invite signed up observers to each session - Dave
• [x] Add participant numbers, dates, and times into recruitment intake ticket Participant Tracker (to check off later after each session) - Dave/Alexis
• [x] Make notetaking doc in github for each session to share with Perigean right before each session starts - Dave
• [x] Schedule team debrief sessions once a day - Dave/Alexis
• [ ] Other tasks as needed

Acceptance Criteria

• [x] Completed prep tasks in support of research actions

[jilladams]

@cindymerrill @ALogsdon3 @Dottisea @xiongjaneg for clarifying for 90 commitment

[Dottisea]

Cindy, Alexis and Dave P met and managed this in a meeting on Aug 2, 2023 and we will discuss briefly again in refinement on Aug 3, 2023.

[jilladams]

From UX refinement, noting for reference:

• Alexis OOO - 8/9-8/16
• Cindy OOO - 8/16-8/18
• Dave is here all week.

First step of recruitment is recruitment request and research repo ticket, and a kickoff with Perigean, which Cindy has done and Dave / Alexis have not. We will do our best to keep this process moving around PTOs.

To understand working backward schedule required to kickoff with Perigean by EOD 8/15, Alexis's draft schedule from the first AC will help us understand the tasks required.

[ALogsdon3]

Here's a rough timeline for VBA research. This schedule is tentative based on how what we know now, but may change based on factors outside our control. It's a snapshot of where we are now but isn't a living document we will maintain. VBA research timeline.pdf

[cindymerrill]

Reviewed design/prototype with content dropped in for the open home loans accordion, and made minor revisions to what participants should click on in tasks 6 and 8 in the convo guide to match. The content looks great! Can't wait to see the content for the other two accordions that open...

[cindymerrill]

Alexis shared her availability with me, and I updated the Timeline section of the research plan for our availability for research sessions and a kickoff meeting with Perigean.

[ALogsdon3]

I finally had time to sit down with the Figma design, spreadsheet with open accordion content, Helpful docs Word doc, and conversation guide all open and go through the user tasks!

I have a bunch of suggested updates. Most of them are tweaks to the conversation guide for @cindymerrill, but some might be of interest to @davidmpickett and @thejordanwood as possible updates to the design. Since I will be OOO starting directly after sharing this with you all, I will leave it to you to make a judgement call on these.

Task 5	Accordion content	Comment	Suggest
“not currently receiving monthly disability payments...you’d like some help applying for them”	“Get help applying for or managing disability benefits”	Language of our task is too close to the language in the accordion.	“you'd like to some assistance with your application” instead of "you’d like some help applying for them”
“Do you need an appointment? [no but it’s preferred]”	"Visit our office, no appointment necessary"	Accordion doesn't specify that appointments are preferred.	Update guide to say "[no preference noted]" or skip this question.
“When you go, what should you have with you? [photo ID, DD214]	Not mentioned in accordion	What to bring accordion lists medical records in addition to ID, DD214.	Add medical records to the list of correct responses
Task 6	Accordion content	Comment	Suggest
"How could you talk to someone? [in-person / virtual / phone]"	Schedule an in-person or virtual appointment. You can also come in anytime during our office hours. You don't need an appointment.	Accordion doesn't list phone as an option.	Delete phone from list of correct responses
"Do you need an appointment? [No but it's preferred]"	Visit our office, with or without an appointment. Virtual appointments available	Accordion doesn’t specify that appointments are preferred.	Update correct response to just say "no" or omitting this question
Task 7	Accordion content	Comment	Suggest
"What do you need to have with you? [DD214]"	Not specified in accordion	In the What to bring section, several documents are personal info types are are listed, not just DD214.	Add the sentence "One of the items listed is a DD214” before “If you don't have a DD214, how could you get it?"
Task 10	Accordion content	Comment	Suggest
"How could you talk to someone? [in-person / virtual / phone]/Do you need an appointment? [no but it’s preferred]"	Call us to schedule an in-person appointment at our office. You will not be able to get through building security without an appointment.		Updating convo guide to reflect that this is something you must make an appt for
"When you go, what should you have with you? [photo ID, DD214]"	Not specified in accordion		Possibly omit or update this task because it's now very similar to Task 7's subtask
Task 11	Accordion content	Comment	Suggest
task that has “(Other services > Veteran direct deposit)” as the correct response	Not specified in accordion	The task is very similar to Task 9 because it now refers to a task that is not specified on this screen.	Update the task to just ask them where they'd go to update direct deposit information to get an idea where Veterans would expect to find something like that on this page?

CC: @jilladams @mmiddaugh @xiongjaneg for visibility

[cindymerrill]

Thanks much for your feedback, @ALogsdon3 ! I'll respond to each below and make revisions in the convo guide as needed. FYI @xiongjaneg @jilladams

Task 5:

• DONE--revised task wording in convo guide
• Appointments ARE preferred, according to top-level accordion Prepare for your visit > Appointments. "No appointment necessary" in the content doesn't convey that they're preferred. Might we want to change this in the Disability accordion, @davidmpickett @mmiddaugh ? I could go either way.
• DONE--added medical records to list of possible responses in convo guide

Task 6:

• Are people able to call into COE appointments? What exactly are "virtual appointments", @davidmpickett @mmiddaugh ? Do they have to include video, or can they be just phone/audio? The icon is a phone, so this is a bit confusing because it implies phone. Maybe an icon of a computer screen would be more helpful? Maybe there's a standard icon for telehealth.

Task 7:

• DONE--adding wording to prompt in convo guide

Task 10:

• DONE--updated convo guide to say that appointment is required
• DONE--deleted question about what to have with you

Task 11:

• Updated the correct response to be (Other services > Direct deposit) because that IS the name of an accordion on the page, even though it's not openable

[cindymerrill]

I went through the prototype as it exists today with our conversation guide, and I have some feedback:

1. For the Disability task to get help with your application, do you have to visit the office? The accordion doesn't mention meeting virtually or over the phone. (Just want to confirm this before marking in the convo guide that this can only be done in person at the office.)
2. Visible hotspots will need to be turned off before piloting, but that can be done later.

Everything else looks great! THANK YOU, @davidmpickett and @thejordanwood !

I made a few more minor updates to the conversation guide, and now it looks to be in sync with the prototype :). Just waiting for input on my two questions for @davidmpickett and @mmiddaugh in my previous comment.

FYI @jilladams @xiongjaneg

[cindymerrill]

@davidmpickett @xiongjaneg @jilladams @mmiddaugh In preparation for tomorrow’s Facilities UX Refinement, I’ve updated the VBA conversation guide per the prototype and research plan per today’s Option 3 timeline. Here are the details left to finalize for these two research deliverables before we can launch recruiting:

• [x] Any changes to dates for kickoff meeting and research sessions
• [x] Any changes to Alexis’ availability for the new research dates of Aug 29-31
• [x] Any changes to the prototype, including in response to my 2 questions above
• [x] Any feedback from anyone on this thread, including Michelle's approval
• [x] Updated link to the prototype (when @thejordanwood has one that doesn’t show hotspots)
• [x] Link to card in research repo (need to create this; doesn't take long)
• [x] Link to recruitment intake ticket (need to create this; doesn't take long)

[mmiddaugh]

• Task 5: Appointment guidance in the Prepare for your visit > Appointments area applies to all of the services available and therefore may not be completely aligned
• Task 6: Rather than getting involved in the details about virtual appointments and associated design choices, perhaps we should ask the participants how they interpret what they see.

[cindymerrill]

I've updated the VBA conversation guide and research plan per @thejordanwood's updated prototype and yesterday's Option 4 timeline.

PLEASE REVIEW these two documents, especially @davidmpickett @xiongjaneg and @mmiddaugh by end-of-day Monday if possible. If you have any feedback on the documents, please share them in this ticket. If @mmiddaugh approves, I'll create a recruitment intake ticket and research repo card and launch recruiting at the end of the day Monday or early Tuesday. FYI @jilladams

[cindymerrill]

Updated VBA research plan and conversation guide per Michelle's feedback, created research repo card and recruitment request intake, pasted both links into the research plan, and submitted both for VA research-ops review and approval. FYI @jilladams @xiongjaneg @davidmpickett

[cindymerrill]

@davidmpickett If you have any feedback on task/question ordering/priority, let me know and I can tweak the conversation guide.

[davidmpickett]

@xiongjaneg @jilladams For sprint reporting purposes, please note that all items listed under Sprint 90 are complete. Barring any unforeseen delays, the rest of this ticket will happen in Sprint 91

[cindymerrill]

Added a bunch of research transitioning tasks on my mind to discuss with @davidmpickett in our VBA research prep handoff meeting tomorrow. We can talk about who's doing what and when, which will be my input to Sprint Planning when I'll be on PTO.

FYI @xiongjaneg @jilladams

[cindymerrill]

Some VBA research updates for @ALogsdon3 cc @davidmpickett ...

• I’ve made minor edits throughout the VBA research plan and conversation guide per the prototype, feedback from Michelle and Dave, and other things I noticed.
• Note that when you run research sessions, you may need to help disambiguate a task for a participant if they seem confused. During the warmup questions at the beginning of the session, you’ll ask which what VA benefits (if any) they have used before, and any tasks that pertain to benefits that they’re not familiar with could cause confusion, in which case you might need to clarify on the fly during the session for the participant. You probably already realize this, but in case not, I’m mentioning it here while it’s on my mind.

[cindymerrill]

Recruited 3 Veterans to be pilot participants next week and scheduled them for Aug 23 and 24. Sent dates, times, and email addresses to Perigean via study slack channel here: https://dsva.slack.com/archives/C05NM1JGWKA/p1692134833970129

[cindymerrill]

Created a notetaking spreadsheet for this study, which I'll discuss with @ALogsdon3 and @davidmpickett during our meeting on Aug 21: https://docs.google.com/spreadsheets/d/1adPKduNfvJTDuNDRzxyyQDw2r24-5_L-ouYOMrVqdeM/edit?usp=sharing

[cindymerrill]

Created a Mural for debriefing research sessions, which I'll discuss with @ALogsdon3 and @davidmpickett during our meeting on Aug 21: https://app.mural.co/t/departmentofveteransaffairs9999/m/departmentofveteransaffairs9999/1692138851053/f1df345d8bafd86292047c476004c855e025e34e?sender=uff8fa85716c1796ccf022668

[jilladams]

Noting: I've counted 8 points for S90, removed the estimate, and we need to re-point effort for Sprint 91 ACs. (@davidmpickett @ALogsdon3 )

[davidmpickett]

Screenshots from Perigean

Notes to discuss in kick-off meeting

• There is no question on here about time zone. Do you collect that information from participants separately?
• Question 4: Indicating that responding NO will disqualify participants could skew behavior by incentivizing people to answer YES falsely. How is this risk balanced against other considerations? Are there other ways to ensure participants who RSVP will be prepared for sharing their mobile screen?
• Question 5: Possible wording tweak: "Have you ever visited a VA Regional Benefit Office, Veteran Readiness and Employment office, ~and~ or VA Regional Benefit Satellite Office?"

[cindymerrill]

@ALogsdon3 @davidmpickett As promised, here are links to sample research reports from Income Limits:

• topline summary
• full findings report
• readout presentation

[cindymerrill]

@ALogsdon3 @davidmpickett As promised, here are the instructions for enabling Perigean to take session notes in github, and a link to my notes folder on github from income Limits: https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/products/income-limits-app/research/2023-01-income-limits-usability/session-notes

Let me know if you have any questions about this or about the reports I shared above.

[cindymerrill]

@ALogsdon3 Also I just added a link to the MVS checker spreadsheet into the tasks for the VBA research synthesis & reporting ticket.

[cindymerrill]

@ALogsdon3 and I met to discuss how to moderate and debrief research sessions, as well as how to log into and use Perigean's zoom account. I have a few small tweaks to make to the conversation guide, and then we'll be all set for pilot 1 tomorrow :)

[cindymerrill]

@ALogsdon3 As promised, here's my After Research Session Tasks list and a printable version of the conversation guide. As we agreed, I'll debrief pilot 1 in the notes spreadsheet, and you'll debrief pilots 2 and 3.

[davidmpickett]

I updated the Recruitment ticket with the participant date and times https://github.com/department-of-veterans-affairs/va.gov-research-repository/issues/362

[davidmpickett]

I updated the Recruitment checker for equality with the participant details. After sessions are complete, no-shows will be updated from 1s to Ns

[cindymerrill]

Wow, you're amazing at tackling these administrative details early, @davidmpickett !

[cindymerrill]

Ran first pilot test and debriefed observations into notes spreadsheet. Debriefed with team and decided to reorder the conversation guide a bit, which @ALogsdon3 revised. Met with Alexis to discuss more, before she runs the second and third pilot tests tomorrow.

[cindymerrill]

@ALogsdon3 ran the second and third pilot tests today. She's still working on adding observations into the notes spreadsheets. Debriefed with team and didn't make any changes to the convo guide. I'll review some of her notes in the spreadsheet today and maybe more after she finishes.

I sent thank you notes to all pilot participants.

[davidmpickett]

Note templates created: https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/products/facilities/regional-offices/research/2023-06-veteran-facing/transcripts

[davidmpickett]

Closing this out, I will still be communicating with observers while sessions are happening, but all major tasks are complete