Research Feedback - Midpoint Review - Debt Resolution Team, Combined Debt Portal

[claytonzook]

VFS acceptance criteria

• [x] Let Platform know when the Must feedback has been incorporated
• [ ] Leave any comments for feedback you decide not to take
• [ ] VFS team closes the ticket

Thoughts/questions

• This research looks well thought through and put together. However, consider this a first review, as this research is still required to go through the Research Review touch point to begin recruitment. I'm happy to answer any questions on this feedback, or brainstorm things like keeping the integrity of diverse participant demographics when splitting into smaller task-related cohorts.

Feedback

• Must = the feedback must be applied
• Should = the feedback is best practice
• Consider = suggestions/enhancements

Must:

Should:

• Split this into 2 or even 3 studies. It's a complex study that also seems to lend itself into easily splitting up. Splitting would ease recruitment for each group/cohort and make it more likely you get the desired split. Additionally, there would be a minor benefit of reducing possible confusion as to which participant gets which set of questions during each session.

• A reminder that in seeking participants who use screen readers, your team should be prepared with a prototype that is accessible for these participants - that usually means a coded prototype. It looks like this is your plan, but just want to reiterate that point.

Consider:

Platform directions

• Update "Issue Title"
• Link to collab cycle touchpoint meeting epic
• Add to the Feedback pipeline on the Platform Governance team board
• Add your feedback
• Add assignees based on collab cycle touchpoint
  • Design Intent: VFS designer, VFS PM (optional), yourself (optional)
  • Midpoint Review: VFS PM, yourself (optional)
  • Staging Review QA Only: VFS PM, yourself (optional)

[claytonzook]

@rileyorr besides splitting up into completely different studies, here is a possible alternative to better ensure successful recruitment - example of the multiple cohort approach

[rileyorr]

Thank you for the feedback @claytonzook! It's interesting you mention splitting recruitment into cohorts because I originally explored that and even referenced the research plan you linked! However, after receiving feedback on that approach and thinking about how the screener questions are essentially the same for each cohort, I thought approaching recruitment with a tree/branching structure of questions may be helpful for Perigean. I referenced this pilot conversation guide.

Recruiting for VA debt and copay studies has historically been challenging because we're only exposing specific debt types in the portal at this time. Additionally, in the past Veterans have considered copay bills to be debts - which is one of the reasons why we're now adding copay bills to the debt portal! As a result, I've continued to iterate on the criteria and screener questions for each study.

I'm excited to hear your thoughts and chat more about the recruitment approach!

[claytonzook]

@rileyorr Rereading through your research material again, I see how it makes sense to keep together. I missed that no matter which group a participant is from, they are essentially going to the same place just interacting with a different specific item that will show up for them.

1. Even in a single study, you can have Perigean recruit as if they were different cohorts. Hypothetical: group 1a is easier to recruit so gets eight Ps, 1b gets one or even none and group 2 gets none. This is a potential outcome the way the recruitment is written. a. If it's important to get more of a mix from each group, a solution might be stating both a max and min for each group. Do you need at least 2 to adequately test with Group 2? Is just 1 enough for 1b? or is it truly ok if they recruit none? b. Also to consider, I don’t think there’s a good way to submit the Group Diagram image to Perigean, otherwise that would absolutely help make this clearer (says me, a visual learner/experiencer). Again after rereading, I do think it's stated pretty clearly in text.
2. If you’ve noticed in the past that it seems to be difficult to recruit for this type of product testing, I would then add that you might also consider a. As much as I advocate for diversity in research, you could consider having less demographic criteria as primary and moving some to secondary criteria. b. Allowing for a larger recruitment window - longer than the 1-week minimum lead time between recruiting kickoff and first session can help for more specific or harder to find cohorts. If the timeline works for your team, you could try giving Perigean 2 weeks and maybe we even flag this recruitment request as "historically harder to recruit." When we make the recruitment request, we essentially copy/paste from the research plan, so you could put those words in with your criteria section.

I appreciate the extra context - I'm a big fan of using your past experience to iterate on the process in order to find what works best with your product and team!

[rileyorr]

@claytonzook those additional points make a lot of sense - thank you! I made some adjustments to the research plan:

• Added a minimum of 3 participants for groups 1b and 2 since it's important to make sure those participants are represented in our study
• I moved the must-have demographic participant requirements to a separate section below the breakdown of the groups and added a note that while these must-haves are our goal, we understand that historically it's been challenging to recruit participants with VA debt.

I have one additional question, but I'm happy to return to it during the research plan review if the answer will be covered during that touch point:

Also to consider, I don’t think there’s a good way to submit the Group Diagram image to Perigean, otherwise that would absolutely help make this clearer (says me, a visual learner/experiencer). Again after rereading, I do think it's stated pretty clearly in text.

Could you tell me more about this? Admittedly, I haven't run research since the research repo spun up, but in the past we've provided Perigean a GitHub link to the research plan and conversation guide.

Thanks again for the detailed feedback, I appreciate it!

[claytonzook]

@rileyorr You are absolutely right that we submit links for these documents, so they will be able to see this image in the Research Plan when they look it over. And I'm not sure what I was even thinking at all, because we can even send the GH link for that specific image with the recruiting criteria text we send, so that part is all good!

Instructions for the research repo board just in case you need it