Closed ajakabcin closed 3 years ago
andaleliz
commented
3 years ago @Samara-Strauss I'd love to get this submitted early in the next sprint - if you have a chance before then, would you be able to review the UAT plan and convo guide?
No worries if you can't, just figured I'd get it on your radar. Thanks!
Samara-Strauss
commented
3 years ago There are a few places in the plan where you mention evaluating usability alongside whether the feature works from a technical perspective, which is the main objective of UAT. After having also reviewed the conversation guide, I think it's ok because you mostly have notes to observe whether there are usability issues and you aren't asking a lot of explicit questions about usability. Inevitably, there might be small usability issues observed, and you should definitely note them down during UAT if they do arise. However, usability -- even just observing if there are problems without outright asking usability questions -- is not the main objective of this research. The focus of UAT is always whether the feature works or not. Just mentioning this because I don't want you to be distracted in the sessions trying to do too many things. Trust that if there are major usability issues, they will likely come up naturally without you having to ask about them.
Recruitment criteria
All the must-haves look good to me.
For the nice to haves -- how strictly does Perigean try to recruit for those requests? I know we want to be inclusive, but I also want to make sure any requests might meaningfully impact our findings. For UAT, I think it's much less necessary than other testing to include a super diverse set of demographics because we are testing the technical functionality, which should work or not no matter who is on the other end. Obviously, recruiting a diverse sample set is super important for more generative research and even for usability testing, but I think it's less important for UAT. One exception to that might be recruiting for people who use assistive technology since that is an additional technical layer and we want to make sure our functionality works with assistive technology, but I would say just about all of the rest of the secondary requests would make no difference for UAT.
Are there any notifications here that you already receive? What are your thoughts about what you see related to those notifications here?
I think you can remove both of these questions. They don't stick to the main objective of UAT. Again, if people offer thoughts around usability, please note them down, but you don't need to ask these explicitly.
Otherwise, this is great! It's ready to submit as far as I'm concerned 😃
andaleliz
commented
3 years ago Thanks Samara! I appreciate the feedback and quick response. Our team talked about additional usability testing prior to UAT given that we had some significant changes since our last study, but determined with Kevin that we could do a combo usability/UAT study to understand if the new design worked instead of running a separate study. Generally, I'm glad my intention not to specifically ask about usability issues during the test was clear after reading through the conversation guide.
In response to some specific points:
For the nice to haves -- how strictly does Perigean try to recruit for those requests?
I'm actually not sure, but based on the participants we've received in past studies, I would say they are not super strict. I agree with you it being less important for UAT, but I still think diversity is important to consider, especially if any significant usability issues come up. I don't think including this criteria will delay the recruiting process. I'll ask about this and remove it if it seems like it'll overly complicate or delay things.
I think you can remove both of these questions. They don't stick to the main objective of UAT. Again, if people offer thoughts around usability, please note them down, but you don't need to ask these explicitly.
Those two questions are related one of my biggest concerns about this feature (we discussed this a couple of weeks ago, fostering mistrust). I'm really interested to hear what people think about that. I hope they prove me wrong and it's no big deal. I expect the test will go fairly quickly, so I don't think we run the risk of going over time or anything by keeping those questions in.
This is my first time running UAT so perhaps I'm being naive, but it all feels so straightforward that I don't think there is much of a risk that asking those questions would distract me from the primary goal of completing UAT tasks. I think we could restructure the guide a bit and move the second question:
What are your thoughts about what you see related to those notifications here?
to the post-task interview section, and only follow-up if there is time, and it's relevant. We can rephrase as:
"You mentioned you already receive some of the notifications that you had to select options for. What are your thoughts about having to set up preferences for notifications you're already receiving?"
If you're 💯 against it, of course I'll take them out, but I would love to keep them in so we have the chance at gathering data around it to support VA Profile prioritizing including default opt-ins in their API.
Samara-Strauss
commented
3 years ago Ah, thanks for the clarification around the mistrust questions. That sounds good to me. Feel free to leave them. I don't think the answers here will ultimately change whether we launch the MVP, but it will be helpful information to have if people are concerned that notifications they already receive are not already reflected on the website. We'd definitely want to bring that insight back to VA Profile.
This looks good to submit! Great work!!
andaleliz
commented
3 years ago Research kicked off on 10/15
Background
We need to ensure that Notification Prefs is working properly for users. Now that we've completed the UAT research plan and conversation guide in #30165, we need our PO to review the documentation and then submit those materials to Lauren Alexanderson.
Tasks