search

cfpb / retirement

Helping Americans make choices about retirement
https://www.consumerfinance.gov/retirement/before-you-claim/
Creative Commons Zero v1.0 Universal
22 stars 28 forks source link

User research - first round #2

Closed marteki closed 9 years ago

marteki commented 9 years ago

The team came up with the following hypothesis for our first round of testing (that could potentially flow through to further rounds):

If we present a simple UI that progressively inquires for information a little at a time, Then consumers will become engaged in the process of learning more about their own retirement.

Other questions that the team brought up to answer: (go ahead and comment to add more, guys!)

Our first test script will also be added to this thread, for review/revision/refinement/whatever.

marteki commented 9 years ago

Test Script - Retirement Sketches - v1

Hypothesis:

If we present a simple UI that progressively inquires for information a little at a time, then consumers will become more engaged in the process of learning about their own retirement.

Recruiting:

4-5 adults between the ages of 50 and 62

QUESTION: Should we target a particular socioeconomic group?

Prototype:

2-4 sketches or wireframes, based on the sketches in https://github.com/cfpb/retirement/issues/1

Set-up:

Hello, we'd like to speak to you today about how you think about finances. We're from a government agency called the Consumer Financial Protection Bureau. Our agency was set up after the financial crisis of 2008-2009 to help make sure that the financial markets work for the American public and people like you, and that providers aren't able to use unfair or abusive practices.

QUESTION: Do we need this next paragraph, with an NDA? @benguhin?

Because we'll be talking about some things that aren't publicly released yet, we have to ask you to sign this legal document. Go ahead and read through it, and feel free to ask us any questions about parts of it. [get signature]

In addition, we'd like to record audio of our conversation. Is that okay? [get consent]

In our conversation today, we'll be looking and talking about some different versions of a website that we'll be rolling out to the public. We'll ask you to step through some rough prototypes as if you were using them online. We're trying to figure out which of the different versions works best for the American public; the website versions are being tested, not you.

In addition, we ask that you please speak aloud everything that you're thinking or feeling as you go through the websites. Sadly, we're not mind readers, so we can't tell exactly what you're thinking about what you see and how you interact with it. Please feel free to tell us everything you see, think, and feel about the site, even if it's negative. Again, it's the websites that are being tested! You can't give us bad or wrong feedback.

Main Questions:

QUESTION: Metrics to track for each version: Net Promoter Score, and System Usability Scale? Might be too much to add an additional 10 questions for each version of the website. But then we are missing a key usability/learnability metric. :-(

COMMENT: Once we know what order the prototypes go in, we could also ask questions specific to each one that are more probing on potential usability issues.

  1. We'd like to talk to you specifically today about the idea of retirement. Do you think about it? If you do, how do you think about it? What does retirement mean to you?
  2. Have you created a retirement plan? If so, please tell us more about what you've included in it, and what your expectations around using your plan are. If not, please tell us more about that decision.
  3. We're going to show you a few different concepts for a website that you can go to that talks about retirement. Here is the first option. Please talk through and describe how you would use this website.
  4. On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this website to a friend or family member as a place to consult about retirement?
  5. Here is the second option for the same type of website. Again, please talk through and describe how you would use this website.
  6. On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this second version of the website to a friend or family member as a place to consult about retirement?
  7. Now we'd like to show you a third option for this website. Once again, please talk through and describe how you would use this website.
  8. Again, on a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this third version of the website to a friend or family member as a place to consult about retirement?
  9. And we'd like to show you a fourth and final option for this website. Once again, please talk through and describe how you would use this website.
  10. Again, on a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this fourth version of the website to a friend or family member as a place to consult about retirement?

Follow-up Questions:

  1. Did you have a preference between the retirement website options that were provided to you? Why or why not?
  2. Was there anything missing from the websites shown to you that you expected to see, or have access to?
caheberer commented 9 years ago

Maybe we want to add some examples to 3, 5, and 7 in reference to "describe how you would use this website" Right now, pretending to be a user, I'm not sure how I would respond to that. @marteki were you thinking this is where they are talking out loud when they are using the site? Do you think it's too open-ended? I understand we can't ask them to find certain areas of the site, like with more refined designs, but maybe we could help define what we are looking for. Ideas?

marteki commented 9 years ago

@caheberer - Yes, I totally think we should do that, with some caveats. I was thinking that once we know which prototype is in which order, we could rephrase the questions to be a little more tailored to that prototype. For example, talking about what income and expenses need to be entered, or referencing that we'll be showing you a website with a timeline. I can see several questions being prompted by things that users volunteer that we can't think of right now, which might lead to additional questions in the midst of the script.

caheberer commented 9 years ago

@marteki :+1:

niqjohnson commented 9 years ago

I like it!

I'd be fine ditching both the System Usability Scale and Net Promoter Score for this round, especially if we're testing sketches. I think we'll get the most value from qualitative comments in early testing, so I'd have no problem not including any quantitative instruments right now in favor of giving participants more time for open-ended talk.

Also, one additional question to ask may be something about where participants currently look for information about claiming Social Security, which could give us insight into what information gaps our tool can fill in.

Also also, I can never pull off humor when reading testing scripts, so you'll have to give me tips on how to work that "we're not mind readers" line in without sounding like a comedy-bot.

marteki commented 9 years ago

@niqjohnson - if you play the "mind readers" line straight but with a smile, the participant usually laughs and jokes some variation on "oh, yes, haha, of course"/"wait, you're not?"/"you should be". I play the neutral moderator role as always having a smile on my face, so participants identify that as my baseline.

Maybe we add this to the very end of the follow-ups, after the questions about the options (so as not to influence any reactions to them): "Have you thought about when to claim Social Security? How did you come to that decision? Did you visit any websites to help you make that decision?"

marteki commented 9 years ago

Test Script - Retirement Sketches - v1.1

Hypothesis:

If we present a simple UI that progressively inquires for information a little at a time, then consumers will become more engaged in the process of learning about their own retirement.

Recruiting:

4-5 adults between the ages of 50 and 62

QUESTION for @SerenaEstrella - Should we target a particular socioeconomic group?

Prototype:

2-4 sketches or wireframes, based on the sketches in issue https://github.com/cfpb/retirement/issues/1

Set-up:

Hello, we'd like to speak to you today about how you think about finances. We're from a government agency called the Consumer Financial Protection Bureau. Our agency was set up after the financial crisis of 2008-2009 to help make sure that the financial markets work for the American public and people like you, and that providers aren't able to use unfair or abusive practices.

We'd like to record the audio of our conversation. Is that okay? [get consent] Thanks! [start recording]

In our conversation today, we'll be looking and talking about some different versions of a website that we'll be rolling out to the public. We'll ask you to step through some rough prototypes as if you were using them online. We're trying to figure out which of the different versions works best for the American public; the website versions are being tested, not you.

In addition, we ask that you please speak aloud everything that you're thinking or feeling as you go through the websites. Sadly, we're not mind readers, so we can't tell exactly what you're thinking about what you see and how you interact with it. Please feel free to tell us everything you see, think, and feel about the site, even if it's negative. Again, it's the websites that are being tested! You can't give us bad or wrong feedback.

Main Questions:

COMMENT: Let's keep the Net Promoter Score questions in, but hold off on the System Usability Scores until we get further down the road of prototype selection.

COMMENT: I assumed that we would show the prototypes in the order spelled out by @niqjohnson above, and tweaked the questions to be more specific to each one.

  1. We'd like to talk to you specifically today about the idea of retirement. Do you think about it? If you do, how do you think about it? What does retirement mean to you?
  2. Have you created a retirement plan? If so, please tell us more about what you've included in it, and what your expectations around using your plan are. If not, please tell us more about that decision.
  3. We're going to show you a few different concepts for a website that you can go to that talks about retirement. Here is the first option. Please talk through and describe how you would use this website. Does it match your expectations for a website about retirement, and why?
  4. Please talk to us about the data you had to enter on this site, and the results you got after entering it. Did it match your expectations for what would be needed for retirement planning, and why? How did it make you feel?
  5. What does the timeline in this website mean to you? Please explain what you see, how you understand the information that the timeline portrays, and how it makes you feel.
  6. On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this website to a friend or family member as a place to consult about retirement?
  7. Here is the second option for the same type of website. Again, please talk through and describe how you would use this website. Does it match your expectations for a website about retirement, and why?
  8. Please talk to us about the data you had to enter on this site, and the results you got after entering it. Did it match your expectations for what would be needed for retirement planning, and why? How did it make you feel? Did it feel different from the previous option that we showed you, and if so, how?
  9. What do the graphs in this option mean to you? Please explain what you see, how you understand the information that the graphs portray, and how they makes you feel.
  10. On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this second version of the website to a friend or family member as a place to consult about retirement?
  11. Now we'd like to show you a third option for this website. Once again, please talk through and describe how you would use this website. Does it match your expectations for a website about retirement, and why?
  12. Please talk to us about the data you had to enter on this site, and the results you got after entering it. Did it match your expectations for what would be needed for retirement planning, and why? How did it make you feel? Did it feel different from the previous options that we showed you, and if so, how?
  13. What do the visuals in this option mean to you? Please explain what you see, how you understand the information that the visuals portray, and how they makes you feel.
  14. Again, on a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this third version of the website to a friend or family member as a place to consult about retirement?
  15. And we'd like to show you a fourth and final option for this website. Once again, please talk through and describe how you would use this website. Does it match your expectations for a website about retirement, and why?
  16. Once again, please talk to us about the data you had to enter on this site, and the results you got after entering it. Did it match your expectations for what would be needed for retirement planning, and why? How did it make you feel? Did it feel different from the previous options that we showed you, and if so, how?
  17. How did you feel about the way that you got information on this option? How did you understand the information? How did it make you feel?
  18. On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this fourth version of the website to a friend or family member as a place to consult about retirement?

Follow-up Questions:

  1. Did you have a preference between the retirement website options that were provided to you? Why or why not?
  2. Was there anything missing from the websites shown to you that you expected to see, or have access to?
  3. Have you thought about when to claim Social Security? How did you come to that decision? Did you visit any websites to help you make that decision?
SerenaEstrella commented 9 years ago

@marteki short answer for demographics: everyone! long answer: Understanding you have 4-5 slots, try to get a mix of income because Melissa tells me that beyond health (too sick to work) and job loss (can't find a new job), those who claim at 62 range from low/moderate to higher income.

If it's easier to get a homogeneous group this time around, we can ensure we have a mix in future testing rounds.

If you're curious about the distribution across groups, see another of Melissa's papers with helpful charts: http://www.ssa.gov/policy/docs/ssb/v74n4/v74n4p21.html

SerenaEstrella commented 9 years ago

What do you think about testing your test script on Melissa and Hector? We can do it through a hangout, and would potentially kill two birds with one stone: they are eager to see your prototypes and you could practice on them. Thoughts @marteki @caheberer @niqjohnson ?

marteki commented 9 years ago

@SerenaEstrella, let's talk about it after the stand-up today. I'm definitely open to that, but want to talk through with the team, and see what others think about setup/impacts. :-)