Improve a11y usability testing experience for designers and participants

[humancompanion-usds]

Description

(This is borrowed from the work of the Forms team, specifically @jeana-adhoc and @rachelshearerux, on 2 recent studies)

The Veteran Facing Forms team participated in two separate studies in June and July 2024. The form status and form upload usability studies included users of assistive technology.

Only 2 participants out of 6 were able to start the study on their device.

Testing with users of assistive technology has been a north star of the research practice on VA.gov. When those users can't access the study from their device, it frustrates researchers and participants.

A retro was held with the designer-researchers and product manager on the Veteran Forms Team, a designer-researcher from My VA, and 3 CAIA accessibility support staff who supported these research sessions.

In total, CAIA attended 13 of the scheduled 17 assistive tech studies. Over half of those studies (7) were either no shows (3) or the participant did not use the technology they identified on the screener (4). In the remaining 6 studies CAIA worked patiently with the participants to try to enable screen sharing, and at times even coaching/training the participants in how to use their assistive tech. In the end, only 2 participants were able to start the study.

Key findings

1. Many processes already exist that help with successful studies. Successful processes focus on simplicity. CAIAʼs support is immensely valuable and much appreciated.
2. Pivoting to a Q&A session yielded valuable information for the researchers. Even when participants could not start the study using their device, we still learned valuable information from Veterans.
3. Veterans want to be self-sufficient and independent in their interactions with VA, and participating in research sessions helps them do that.
4. Some participants joined the studies and did not need to use the assistive tech they listed in the recruitment surveys. Perigean struggled to find additional participants to fill these assistive tech slots.
5. Participants needed help switching apps, screen sharing, and loading URLs in browsers when using their assistive technology. Veterans who struggled with screen sharing or using their assistive tech for the study felt they had failed. They thought they were not helpful.
6. Some participants had not used VA.gov before to apply for benefits. They lacked the self-trust that they could successfully complete an application. Researchers assumed that participants had used VA.gov before.

Recommendations

1. Consider the ʻmagic wandʼ ideas (see below)
2. Continue to work on ways to make accessible prototypes in staging, codepen, or another system.
3. Work with VA.govʼs Accessibility Lead, Martha Wilkes, to define what a completed assistive technology usability session looks like.
4. Work with full-time researchers to improve the process of conducting usability studies with assistive tech users.

Magic wand ideas

If you had a magic wand to create your ideal research session what would it look like?

1. Before the session begins, Perigean should ensure participants are ready to start the study on their device using the assistive technology that they identified in recruiting.
2. More studies would be in-person.
3. Logging in would never be required, even to test an authenticated experience.
4. There would be at least two “levels” of testing. Are we testing, does this thing technically & functionally work? Or are we testing what the user experience like?
5. Continue to use CAIA for support. Also, train embedded Accessibility specialists to provide the same support.
6. Coded prototypes are the norm. They're easy for engineers to build without duplicating effort.

Tasks

• [ ] Have Jeana and Rachel present to Shane and Martha the findings from the Forms team so they can ask questions.
• [ ] Matt, Martha, and Shane to brainstorm additions to this initiative based on the findings and recommendations. Then refine this initiative (or create sub-initiatives)
• [ ] Talk to Perigean about how they can help us
• [ ] Determine who can help move us forward on using either Preview environments or Codespaces to setup better usability testing environments in staging
• [ ] Improve platform documentation as we go
• [ ] Incorporate findings from upcoming AT retros to inform any changes to the research processes

### Tasks

[humancompanion-usds]

Full PDF report from the Veteran-facing Forms team: Form status and Form Upload Assistive Tech Usability Study Retro.pdf

[artsymartha68]

Four assistive tech retros are now complete, with 2 additional retros from product teams (AskVA and Forms). Next step is to synthesize, waiting for Robyn Singleton who wants to be involved.