search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
282 stars 202 forks source link

[CAIA A11y Research]: Accredited Representative Management (ARM), Find a Representative (End of March/Early April 2024) #74488

Closed jfinnerty13 closed 6 months ago

jfinnerty13 commented 8 months ago

Project Overview

[!NOTE]

  • Timeframe:
    • ~Old Dates: March 25 - April 5, 2024~
    • New Dates: 4/1-4/12/2024; No Tuesdays
    • Time Slots (Range): 10 a.m. to 4 p.m. EST.
  • Sync: Thursday, February 29⋅10:00 – 11:00am
  • Pilot(s): Ticket 77592
  • Assistive Tech Users: 4 to 6 AT Users
  • Assistive Tech Being Used: Screen Readers & Magnifiers
    • Also, see the table in this ticket: VoiceOver Magnification Voice to Text
  • Artifacts Available: February 2024 Live as of 3/5/2024
  • Testing Environment: Live website 3/5

CAIA Assignee(s)

Team Info

Toggle to reveal additional team information ### Do you have an accessibility champion? Yes, we have a champion ### If you have an accessibility champion, what's their name? Janelle Finnerty ### Is anyone on your team interested in becoming an accessibility champion? _No response_ ### Calendar checking - [ ] Based on this calendar, confirm we have specialists who can support your study in your desired timeframe - [ ] Confirm you are filing this support ticket at least 1 week in advance of said timeframe, so we have enough heads up to provide support ### Have you reviewed the CAIA capacity calendar? - [ ] Yes, we have reviewed the CAIA capacity calendar and can confirm specialists are available to support in our research timeframe ### When is the timeframe for your testing? 3/25 - 4/5 (Flexible on timing)

Research Sessions

Toggle to reveal information related to the research sessions ### Timeframe - **Pilot Session:** See table 3/25 and 3/26 `COMPLETED` - **Timeframe of Study:** ~3/25 - 4/5/2024~ 4/1-4/12/2024 - **AT Technology Being Tested**: `VoiceOver` `Magnification` `Voice to Text` ### Length of Sessions - **Session length:** `TBD in minutes` - **Buffer time between session:** `recommendation 2x non AT sessions` `TBD in minutes` - **Maximum number of sessions per day:** `pending info` ### Testing Schedule CAIA A11Ys will be looking for device and AT information. We realize that some of this information will not be available until the day of the session. Please use the following table to enter the following information for each participant: Date and Time, Assistive tech (AT), Device and Operating System, Browser, and Skill level: P# | Date and Time (ET) | AT | Device and OS | Browser | Skill | CAIA A11ys Available ------------------|------------------|--------------|--------------|--------------|--------------|-------------- **PILOT** | 3/25/2024 10 a.m. ET | VoiceOver | Desktop | `Needed` | Pilot 1 | @coforma-jamie **PILOT** | 3/26/2024 12 p.m. ET | iOS VoiceOver | Mobile (iPhone) | `Needed` | Pilot 2 | @eli-oat P1 | 4/1/2024, 10 a.m.ET | Magnification | iPhone | Safari | Intermediate | @coforma-jamie P2 | 4/1/2024, 1:00 p.m. ET | ~~Magnification~~ none | ~~Tablet~~ iPhone | Safari, Chrome | Novice 3-6 months | @coforma-jamie P3 | 4/2/2024, 10 a.m. ET | ~~Magnification~~ none | Desktop| Edge | Novice, Intermediate, Advanced `TBD` | @eli-oat P4 | 4/2/24, 1 p.m. ET | ~~Magnification~~ none | ~~Mobile~~ iPad | Safari | Novice, Intermediate, Advanced `TBD` | @coforma-jamie P5 | 4/4/24, 10 a.m. ET | ~~Magnification~~ AT not related to navigation (hearing aid) | Desktop, Windows | Chrome | unsure | @coforma-jamie P6 | 4/4/24, 1 p.m. ET | Computer | C2 Voice to Text `didn't need for session tasks` | Chrome | Novice 3-6 months | @coforma-jamie 🛑 P7 `no show` | 4/5/24, 10 a.m. ET | ~~Tablet~~ | ~~C2 Prefer not to answer~~ | ~~Safari, Chrome, etc.~~ | ~~Novice, Intermediate, Advanced~~ | @coforma-jamie ❌ P8 `no coverage` | 4/5/24, 1 p.m. ET | Mobile (iPhone) | Magnification | Safari, Chrome, etc. `TBD` | Novice 3-6 months | @SarahKay8 🛑 P9 `no show` | 4/8/24, 10 a.m. ET | Desktop | VoiceOver | Safari, Chrome, etc. `TBD` | Intermediate 3-6 months | @sara-amanda 🛑 P10 `no show` | 4/8/24, 1 p.m. ET | Desktop | VoiceOver | Safari, Chrome, etc. `TBD` | Intermediate | @sara-amanda P13 | 4/12/24, 2 p.m. ET | TalkBack | Android, Samsung | Chrome | Advanced | @coforma-jamie - **Consideration**: You can also consider asking participants more specific AT and device combinations. - **For instance:** TalkBack on Samsung Galaxy 8, or VoiceOver on iPad with magnification. CAIA A11ys will review the coverage needed, and self-assign, based on a11y availability.

Supporting artifacts

All research documentation will be coming in Feb. 2024. We'll be testing on the live VA.gov website or staging.

Toggle to reveal remaining supporting artifacts that are pending - [User flows](Insert URL here between the parenthesis) - [Epics/issue that may be helpful](Insert URL here between the parenthesis)

Next steps

### Accredited Representation Management (ARM) Team - Next Steps
- [x] Post a link to this ticket in the accessibility-help Slack channel and tag Terry Nichols.
- [x] Your team's researchers and observers read through [platform guidance on research with assistive technology users](https://depo-platform-documentation.scrollhelp.site/research-design/research-with-assistive-technology-users)
- [x] **KICKOFF MEETING:** To be coordinated with @coforma-terry `ORIGINAL KICKOFF 9/26/2023` - meeting 2/29/2024 10-11 a.m. ET
- [x] Process Update Provided via Slack to Sara and Jamie [2/13/2024](https://dsva.slack.com/archives/C06JCHUAM7D/p1707752000861689)
- [x] **PILOT(S)**: `Recommended` CAIA will need to know the AT being used prior to the pilot for pilot prep. [See the pilot ticket 77592 for the comment relating to running two pilots](https://github.com/department-of-veterans-affairs/va.gov-team/issues/77592#issuecomment-1979020042).
- [x] **LINK TO PROTOTYPE**: Prototype will be reviewed in the following environment:`TBD` with CAIA A11y(s) during the pilot session. _FYI - CAIA grabbed two URLS from the conversation guide comments and put them in the supporting artifacts section above._  3/12/2024
- [x] **INVITE CAIA A11YS**: CAIA Specialists will need to be assigned to your research, be sure to add the respective a11y specialist(s) to your session invites (include the Zoom or Teams link). Be sure to invite the CAIA A11y Capacity Calendar - if possible.  `Scheduling started as of 3/22/2024 - invites c/o Perigean`
- [x] **GOOGLE SHEET:** Shared Session Scheduling to Reference on 3/22/2024 [Google Sheet](https://docs.google.com/spreadsheets/d/1MJB_o5qDzjHY9NYnGriGV_SQ1AVmfi7_itcT5Iiq8ho/edit?usp=sharing)
- [x] **AT PARTICIPANT COUNT AND DETAILS:** See table in the body of this ticket (_above in the Device & AT Combinations section_). Provide specialists your participant details like skill level, operating system, assistive tech used, browser `Prior to Sessions - If Possible` `Partial Info Prior to Sessions Helps`
### CAIA A11Y - Next Steps
- [x] Ticket formatting and updates with task lists 1/29/2024.
- [x] **CALENDAR HOLDS - RESEARCH WINDOW** Put initial calendar holds on [CAIA A11y Capacity Calendar](https://calendar.google.com/calendar/embed?src=c_f982350fc73a8162a1088ab14d00dd37335b2b559a60d6b67eb8021416f81c2a%40group.calendar.google.com&ctz=America%2FNew_York) `HOLDS AND INVITES TO DATE - AS OF 3/22/2024`
- [x] Calendar holds updated to reflect new dates for AT research 4/1-4/12/2024
- [x] Kickoff Meeting with @coforma-terry 9/26/2023 - `Original Kickoff - COMPLETED`
- [x] Sync scheduled for 2/29/2024 at 10 a.m. ET
- [x] Ticket moved from `blocked` to `ready` on 2/13/2024
- [x] Ticket moved from `ready` to `in progress` on 3/22/2024
- [x] Next Steps Update Provided via [ticket](https://github.com/department-of-veterans-affairs/va.gov-team/issues/74488#issuecomment-1941703425) and [Slack](https://dsva.slack.com/archives/C06JCHUAM7D/p1707835434082679) 2/13/2024
- [x] **REVIEW PROTOTYPE:** This is LIVE, there's no prototype. CAIA grabbed two URLS from the conversation guide comments and put them in the supporting artifacts section above._ 3/12/2024
- [x] **REVIEW ARTIFACTS:** CAIA Specialists review your conversation guide and research plan. `JAMIE - REVIEWING`
- [x] **PILOT INVITE(S)** Pilot session(s) with team: **Pilot 1:** 3/26 at 12 p.m. see pilot ticket 77592 below.
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/77592
- [x] **PILOT SESSION** Participate in a pilot session with team on 3/26 (ELI)
- [x] **ARM SCHEDULING**: Shared Session Scheduling to Reference on 3/22/2024 [Google Sheet](https://docs.google.com/spreadsheets/d/1MJB_o5qDzjHY9NYnGriGV_SQ1AVmfi7_itcT5Iiq8ho/edit?usp=sharing) also [updating table in this ticket with a11y assignments](https://github.com/orgs/department-of-veterans-affairs/projects/929/views/27?pane=issue&itemId=51172238#:~:text=Research%20Sessions,the%20research%20sessions).
- [x] **CALENDAR HOLDS - OFFICIAL INVITES** Put official calendar holds on [CAIA A11y Capacity Calendar](https://calendar.google.com/calendar/embed?src=c_f982350fc73a8162a1088ab14d00dd37335b2b559a60d6b67eb8021416f81c2a%40group.calendar.google.com&ctz=America%2FNew_York) `INVITES HAVE BEGUN AS OF 3/22/2024 C/O TEAM/PERIGEAN`
- [x] **PARTICIPATION:** Participate in User Research Sessions: `3/25-4/5/2024`
- [x] **A11Y FEEDBACK:** CAIA Specialists created a [research notes template](https://docs.google.com/document/d/17O4Ct-_FStNIwytuCir6eGb3h0QyMbOlVQgxqzejW5M/edit?usp=sharing) for these sessions.
- [ ] **POST-RESEARCH:** CAIA Post research, specialists will add accessibility findings to your research folder
sara-amanda commented 8 months ago

Intake Received

Hi, @jfinnerty13 ! Just a quick note here to let you know that we received your intake ticket and will be processing it later this week. It normally isn't like this, we have had a sudden influx of studies, taking place prior to yours.

Next Steps

All the best,

Sara

cc: @coforma-terry

jfinnerty13 commented 8 months ago

Hi @sara-amanda absolutely zero rush :) we have plenty of time.

Thanks again!

sara-amanda commented 8 months ago

Intake Processed

Hi, @jfinnerty13 , just a quick update that I have updated your ticket and moved it to Ready in our CAIA ticket pipeline! You will notice that I created two sets of task lists:

a11y support

February Kickoff

Pilot Session

We also recommend all teams participate in a pilot session, to review for accessibility issues that could impact the study ahead. We would like to schedule a time to conduct that pilot session with your team, if you would like to do that as well. Just let us know your availability.

All the best!

Sara

sara-amanda commented 8 months ago

Research Timeline Update

Moved from Blocked to Ready on 2/13/2024

Dates and Process

c/o @jfinnerty13 via Slack 2/13/2024

My normal process usually includes (all in business days)

  • 2 days (March 14 - 15)/ submit to collab cycle review
  • 5 days (March 18 - 22)/ recruitment with Perigean
    • In those 5 days I'll conduct the pilot session
  • 7 days (March 25 - April 4)/ conduct studies (minus Tuesdays those are our sprint days) - sessions will be 1 hour long to help with technology needs, and 2 sessions a day since I'm the only researcher on the team

Next Steps for Research

For reference, these are taken from the task lists in this ticket.

Schedule pilot between March 18-22

New Kick-Off Meeting

User Research Sessions

@coforma-jamie @coforma-terry

jfinnerty13 commented 7 months ago

@sara-amanda @coforma-terry @coforma-jamie I had three questions please

  1. Here it mentions a kickoff was last August and checked off, should that be updated to reflect the need for a kick-off? I would like one to review the materials once they're together (I actually started in August so not sure where that date came from?)
  2. I reviewed the Platform site and it mentions to split out AT efforts, so I'm actually going to split this effort from my mobile testing and am recommending doing mobile testing 3/25 - 3/29 and AT testing then 4/1 - 4/12 - please let me know if this is possible or I can keep 3/25 - 4/5 as originally intended, but I was thinking push it out a week gives ya'll more time to prepare + I remembered some OOO's - just let me know which is best for you
  3. I've received some feedback that drop off for AT users is high, and others recommended a 2-3x recruit, so if I want 4 participants recruit 12. What are ya'lls thoughts on this? I want to make sure we hit 4 - 6 participants on AT for this study to ensure our live product is usable by our users so I'm open to the extra participants but wasn't sure if ya'll were
sara-amanda commented 7 months ago

CAIA Follow-Up

Happy to answer and clarify @jfinnerty13! Including @coforma-jamie and @coforma-terry on this reply, should you have additional questions, so we can jump in and assist you.

Project Sync

Kickoff

So, here are the notes from the project Kickoff with CAIA that I referenced.

Toggle to view question number one >> Here it mentions a kickoff was last August and checked off, should that be updated to reflect the need for a kick-off? I would like one to review the materials once they're together (I actually started in August so not sure where that date came from?)

Sync Scheduling with CAIA

We are available to do a sync between CAIA A11ys and your team.

Research Sessions

Separate AT Study

Yes, you are correct. It is recommended to separate out AT participants, so that the study can focus on users of AT.

Toggle to view question number two >> I reviewed the Platform site and it mentions to split out AT efforts, so I'm actually going to split this effort from my mobile testing and am recommending doing mobile testing 3/25 - 3/29 and AT testing then 4/1 - 4/12 - please let me know if this is possible or I can keep 3/25 - 4/5 as originally intended, but I was thinking push it out a week gives ya'll more time to prepare + I remembered some OOO's - just let me know which is best for you

Platform Guidance

Conduct a separate study Consider conducting a separate study just for assistive technology users. This will help the moderator and team focus on the unique needs of conducting this research.

CAIA Availability

Users of AT and Attendance

Toggle to view question number three >> I've received some feedback that drop off for AT users is high, and others recommended a 2-3x recruit, so if I want 4 participants recruit 12. What are ya'lls thoughts on this? I want to make sure we hit 4 - 6 participants on AT for this study to ensure our live product is usable by our users so I'm open to the extra participants but wasn't sure if ya'll were

Recruitment Rule of Thumb

When recruiting participants, Perigean recommends that you should ask for ~25% more people than you need.

Platform Guidance

Provide Perigean with times that are good for you and your team, they will schedule the sessions. A general rule of thumb is to provide 2x available time slots for the number of research sessions you desire.

To ensure you get the number of participants you’re looking for, state your desired number and ask for ~25% more people than you need. This buffer will help account for no-shows. If the no-show rate is higher than expected, Perigean will continue to recruit and schedule until your desired number is reached. In some cases, your study may need to extend by a day or two to accommodate.

Maximum Variation Sampling

I located the following MVS resource that might help further answer your question and assist you in the recruitment process of AT users.

In most cases, you will be unable to achieve maximum variation in just one study let alone recruit enough participants to begin with due to too many variables to consider during recruitment. ...

Use lean MVS strategy when you’re:

  • Unable to recruit a high volume of participants.
  • Experiencing a large volume of no shows.
  • Looking to iterate on findings or need to validate new changes in a design throughout a study.

Also, Liz Lanz has written some recruitment screener questions for assistive technology (SR) that might be helpful and Naomi Marcussen has shared this document of screener questions.

jfinnerty13 commented 7 months ago

Project Sync

Kickoff

Apologies I was confused on the kick-off portion, thought this was referencing the research kick-off not the project kick-off. Makes sense, thanks for the clarification!

Sync Scheduling with CAIA

I'm also good for the a11y sync Thursday morning. Can we also invite @mtri please?

Research Sessions

CAIA Availability

Sounds like the proposed dates would work better so lets plan on changing the dates to 4/1 - 4/12 then please. @sara-amanda @coforma-jamie @coforma-terry

Users of AT and Attendance

Sounds good, let's catch up more on Thursday.

Thanks!

coforma-terry commented 7 months ago

Hey @jfinnerty13 - do you have an email address for @Mtri so we can make sure they're added to to the a11y sync on Thursday? Thank you!

CC @sara-amanda

oddball-lindsay commented 7 months ago

@coforma-terry michael.tri@adhocteam.us :)

sara-amanda commented 7 months ago

CAIA Updates

See you Thursday, @jfinnerty13!

cc: @coforma-terry @coforma-jamie

coforma-terry commented 7 months ago

CAIA Updates from Thursday 2/29 Sync

Thank you! @jfinnerty13 CC: @sara-amanda @coforma-jamie

sara-amanda commented 7 months ago

Updated ticket project overview to reflect 2/29 meeting updates.

sara-amanda commented 7 months ago

Pilot Ticket Created

[CAIA A11y Research - PILOT]: Accredited Representative Management (ARM), Find a Representative #77592 @coforma-terry

Multiple Pilots Recommended

@coforma-jamie is recommending two pilots one mobile and one desktop.. That could give you a better sense of what issues you might encounter during the study.

Pilot Testing Options

Type Session Time Slots (ET) CAIA A11ys Available
PILOT Date TBD March 18 - April 1, 2024 (ARM Team Availability) Time TBD @coforma-jamie (Pilot options include: VoiceOver + MacOS, JAWS + Windows, NVDA + Windows)
PILOT Date TBD March 18 - April 1, 2024 (ARM Team Availability) Time TBD @eli-oat (Pilot options include: VoiceOver on iOS, macOS, Android TalkBack or JAWS on Windows)
sara-amanda commented 7 months ago

Supporting Artifacts

Links added to the Pilot Session Section (copied below).

jfinnerty13 commented 7 months ago

Finalizing documentation for the research review and it asks for who will be our a11y members for the study. I have written down @sara-amanda and @coforma-jamie is that correct?

sara-amanda commented 7 months ago

Hi @jfinnerty13 ,

For now, feel free to put myself down (for the second week of the study), along with @coforma-jamie (pilot and study) and @eli-oat (helping with pilot).

Thank you!

jfinnerty13 commented 6 months ago

Sessions are starting to roll in - set up a sign-up sheet to get us started. Still waiting on specifics for devices for each session. Is there another way you'd like to sign up @sara-amanda @coforma-jamie or does this work?

sara-amanda commented 6 months ago

Session Scheduling Update

Sessions are starting to roll in - set up a sign-up sheet to get us started. Still waiting on specifics for devices for each session. Is there another way you'd like to sign up @sara-amanda @coforma-jamie or does this work?

Incoming Invites

This is great @jfinnerty13 !

I see some invites came to my inbox, so I can ...

Calendar & Ticket Updates

To date we have added to the calendar your Google Sheet and to the table in this ticket:

We can confirm assignees to the sessions next week. I am basing them currently, on our a11y calendar hold availability. I will be out of the office for those two sessions, but Jamie appears to be available.

Future Invites

Have a great weekend!

cc: @coforma-terry @coforma-jamie

jfinnerty13 commented 6 months ago

Added times to sign-up sheet @sara-amanda @coforma-jamie. Just a heads up we're in flux with recruitment but it does appear as though we'll most likely be running studies 10 - 1130 AM and 1 - 230 PM EST on every day but Wednesdays. Thank you so much for your help!

sara-amanda commented 6 months ago

Added times to sign-up sheet @sara-amanda @coforma-jamie. Just a heads up we're in flux with recruitment but it does appear as though we'll most likely be running studies 10 - 1130 AM and 1 - 230 PM EST on every day but Wednesdays. Thank you so much for your help!

Thank you @jfinnerty13 I will update our table and calendars.

Can we also capture the browser they are using?

jfinnerty13 commented 6 months ago

For all sessions or only the screen readers @sara-amanda ?

sara-amanda commented 6 months ago

We normally collect that for all users and AT types, so if we can get it for all that would be great!

jfinnerty13 commented 6 months ago

Sorry one more clarification - did you want me to have Perigean capture that before the session or are you ok with during/after the session? @sara-amanda

sara-amanda commented 6 months ago

Sorry one more clarification - did you want me to have Perigean capture that before the session or are you ok with during/after the session? @sara-amanda

Before would be great, so we can prep.

jfinnerty13 commented 6 months ago

I just messaged and tagged @sara-amanda and @coforma-jamie in Slack with the recruiter to see if we can get that info. Thanks!

jfinnerty13 commented 6 months ago

Quick update - there are NO sessions Tuesday April 9th, @sara-amanda @coforma-jamie - thank you!

jfinnerty13 commented 6 months ago

Please note Participant 8 is a reschedule and NO a11y support is needed :) thanks!

coforma-jamie commented 6 months ago

This study concluded on April 12, 2024.

Assistive tech findings summary

There were only a few accessibility-specific findings from this study. On the whole, the FIND tool worked well with all tested AT - magnifiers, screen readers, and voice-to-text.

  1. Safari: There was an issue using Autofill to fill out the text fields. This is either a VADS issue or a Safari issue, and has been reported to the DST, who will investigate.
  2. Safari: There was an issue printing from the Cernar portal on MHV. This was detailed in a Slack thread and reported to the appropriate VA business partner.
  3. A couple of UX observations:
    • Most participants were looking for phone numbers, especially those with visual disabilities who could find email or going in person cumbersome. There was frustration at VSO entries without phone number.
    • Two participants looked for a link to the tool in the page footer. This was seen as a more direct way to access it, rather than the top nav. The team should consider adding a link to the footer if possible @jfinnerty13.

All other findings are more general, and were already reported in the team's Mural.

Closing out this study now. Thanks everyone!!