search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
284 stars 206 forks source link

Discovery to determine feasibility & timing of running AT user testing for TE #96522

Closed evelynhilbrichdavis closed 2 weeks ago

evelynhilbrichdavis commented 3 weeks ago

Issue Description

OCTO has communicated that DBEX should plan to run AT user testing on the new TE questions.

We have two options and must decide on one in order to proceed with research planning:

A) Adjust our existing test plan to include assistive technology users (for example, testing non-AT users in Week 1 and AT users in Week 2, followed by analysis of both groups)

B) Conduct separate testing with AT users as a quick follow-up in 2025

Tasks

Acceptance Criteria

mayacarrolluxa6 commented 3 weeks ago

@evelynhilbrichdavis I did some slack farming to see what we could learn from other teams who have tested with Veterans using assistive technology before. These are the lessons and resources they shared.

Threads

sitewide-content-accessibility-ia

https://dsva.slack.com/archives/C01K37HRUAH/p1730919740902699

trauma-practice

https://dsva.slack.com/archives/C04F9JLSATE/p1730833877825099

research-ops

https://dsva.slack.com/archives/C0216PL32HJ/p1730833848031439

accessibility-help

https://dsva.slack.com/archives/C8E985R32/p1730833881134729

va-disability-exp-team

https://agilesix.slack.com/archives/C04MJV66ZPC/p1730840448202879

Guidance from Other Teams

•   Testing Approach: Start with the test plan and questionnaire designed for non-AT users. Modify these for AT users. Although you’ll need two distinct screeners, you can treat this as a single research plan and study.
•   Testing Sequence: Begin with AT users, then conduct tests with non-AT users.
•   AT User Cohorts: Divide AT users into two groups:
•   One group using magnification tools
•   One group using screen readers
•   Additional Questions for AT Users:
•   Include a question on any specific needs they might have with our product.
•   Ask about general challenges they face using their AT at the VA.
•   Backup Strategies:
•   Prepare a plan to guide AT users through the questions without visuals, in case their technology doesn’t work as expected.
•   Develop a contingency plan if non-AT users participate in studies intended for AT users.
•   Availability Check: Assess the number of AT user studies currently underway, as this may affect participant availability.

Open Question

•   Do we have data, or could we gather data, on the number of Veterans submitting the 526ez who use assistive technology?

Examples of Assistive Technology (AT) Testing Resources

https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/accredited-representation-management/research/2024-03-ARM-findarep-live-usertest/research-plan.md

https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/ask-va/design/User%20research/07-2024%20Assistive%20tech/Dashboard/Research%20plan.md

https://dsva.slack.com/files/U0360TY9E9H/F07QQEM7VUJ/design_research.meeting.-.research.report.-.community.care.veteran.self-scheduling.-.usability.and.concept.testing.pdf?origin_team=T03FECE8V&origin_channel=C053UDWMH7U (check slide 10)

https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/products/identity/Research/2023-07%20Terms%20of%20Use

https://github.com/department-of-veterans-affairs/va.gov-research-repository/issues/351

https://drive.google.com/drive/folders/1l1QK1_ZT7s3N3yQkXcy3OIQoyDgKD0Bz?usp=drive_link

https://dsva.slack.com/archives/C04AZ8T7XN1/p1730929657000329

Steps for engaging CAIA https://dsva.slack.com/archives/C01K37HRUAH/p1730994288211619?thread_ts=1730919740.902699&cid=C01K37HRUAH

Submit a research ticket https://github.com/department-of-veterans-affairs/va.gov-team/issues/new?assignees=coforma-terry&labels=508%2FAccessibility%2C+sitewide+CAIA%2C+sitewide+accessibility%2C+CAIA-a11y-research&projects=&template=caia-a11y-research.yaml&title=%5BCAIA+A11y+Research%5D%3A+Team+name%2C+product+name

Research Support CAIA Accessibility provides https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/teams/CAIA/accessibility#research-support

mayacarrolluxa6 commented 3 weeks ago

@bacitracin Looping you into this ticket to consult with us on planning to test with Veterans who use assistive technology.

mayacarrolluxa6 commented 3 weeks ago

@evelynhilbrichdavis I’m hoping for us to submit this as a single package to OCTO’s Research Lead on 11/18, as originally planned.

evelynhilbrichdavis commented 3 weeks ago

@mayacarrolluxa6 thanks for the resources, that sounds good. Here is what I'm thinking for the timeline on this ticket. Does this match your thinking?

  1. By EOD today (11/7) have both options outlined. Dependency: Hear back from CAIA about their availability to support during week of 12/2 or 12/9
  2. Tomorrow 11/8: Schedule synchronous alignment during typical huddle to review and confirm direction
  3. During Bi-Weekly Design Sync, we present both options with our recommendation and share progress approach to AT testing
evelynhilbrichdavis commented 3 weeks ago

Confirmed by CAIA that they have availability both weeks.

evelynhilbrichdavis commented 3 weeks ago

We propose integrating AT users into our upcoming study over running a follow-up study in Q1.

Option to include AT testing now: This option prioritizes accessibility and opens up availability in Q1 for other work. We have already confirmed that CAIA would have availability for either week of research (12/2 or 12/9). No changes to research timeline— although we recommend an early 2025 share-out due to the winter holiday sprint OOO.

Option to AT test in Q1: This option would "stay the course" and focus on either AT or non-AT users for the Q4. We learned that running separate studies to test AT/non-AT have felt redundant to other teams. If we were to keep the studies separate, we would advocate running AT testing first in Q4.

How we would proceed: Update research plan to account for an AT cohort to test one of the two weeks (preferably 12/2) and file CAIA intake. Develop conversation guide for non-AT users and then adapt for AT-users. Throughout the process, collaborate with CAIA and Tracy to ensure the study meets the needs of AT users.

mayacarrolluxa6 commented 3 weeks ago

@evelynhilbrichdavis @pacerwow Let’s stick to the approved research timelines and aim to complete testing for both AT and non-AT users, with a share-out of insights before the end of 2024.

Toxic Exposure Evaluative Research Timelines

Last updated on Nov 5, 2024

Sprint 9 October 23 - November 5 Draft research plan Draft research goals Past TE research knowledge transfer

Sprint 10 November 6 - November 19 Finalize research materials and submit to Platform. Shannon would like to have the research plan (goals, screening criteria, discussion guide) sent to Shane by the 18th at the latest.* Finalized Research Materials include: Research goals Research plan Screening criteria Discussion guide Recruit participants Kickoff meeting with Perigean

Sprint 11 November 20 - December 03 Work paused for Thanksgiving holidays Start testing on Monday 12.2 Conduct moderated usability tests (with veterans who use assistive technology and veterans who do not use assistive technology) Run semi-structured usability testing session on the 1.1 TE release in production or staging environment Collect data Synthesize data

Sprint 12 December 4 - December 17 Conduct moderated usability tests (with veterans who use assistive technology and veterans who do not use assistive technology) Run semi-structured usability testing session on the 1.1 TE release in production or staging environment Collect data Synthesize data

Sprint 13 December 18 - December 31 Write topline insights report Share report with stakeholders Analyze data for trends Share-out findings Recommend improvements

CAIA Ticket :arrow_right: Let's go ahead and open a ticket with CAIA for support testing with veterans who use assistive technology. Please review the ticket with me in a zoom meeting before submitting it.

TE Usability Testing - Synthesis and Reporting Expectations By End of Year 2024: :arrow_right: We will provide Shannon and Emily with an initial summary of results after testing each cohort. :arrow_right: Final synthesis and reporting will be completed before year-end, along with two share-out sessions: A session with the OCTO enabling team and VBA. An open-invite session for the Benefits portfolio and health team.

By Q1 2025: We will present in the OCTO Research Share-out. To secure a spot, we’ll request to be added to the waiting list before the end of 2024, as scheduling can take several months.

If you have any questions or concerns, feel free to schedule a 1:1 with me to discuss them.

mayacarrolluxa6 commented 2 weeks ago

@evelynhilbrichdavis Please set a time limit for your discovery activities on this ticket, as we need to make progress on this ticket.

evelynhilbrichdavis commented 2 weeks ago

@mayacarrolluxa6 Closing this ticket to focus on submitting research material ticket. .