How do we make sure most users are able to authorize applications with MyUSA?

[esgoodman]

Completing an initial sign-on is the gateway to all of MyUSA's other features. Making sure that users are able to complete the first authorization is crucial.

How do we know we have answered this question?

Ideally, we'd want 100% of users to be able to log in using MyUSA. However, There's always going to be someone who can't do it. You never get 100% success in anything, because the world is unpredictable and complex. However, @polastre raised a good point: defining success should be a team activity. That way we have a common ground from which assess our progress.

For me, right now, "most" is about 75% for an MVP. App authorization is a critical task, so 51% is too low. Run-throughs with the demo suggest that we would probably see at least 65% success with authorizing apps in the wild. So I'm not planning to measure task completion times or anything intensive like that because completing authorization doesn't seem like a big problem at the moment. (That could change when we show the demo to members of the public.)

What we're doing to answer it Preliminary testing with lightly interactive demo

• completed with federal employees (#505)
• now planning to talk to non-employees who use gov't websites such as healthcare.gov

[esgoodman]

Related question: Can people authorize a new application with MyUSA? #500

[polastre]

@esgoodman how would you define "most"? Is there a success metric?

[esgoodman]

The easiest way to measure success is just whether the user can log in to the target app at al.

Ideally, we'd want 100% of users to be able to log in using MyUSA. However, There's always going to be someone who can't do it. You never get 100% success in anything, because the world is unpredictable and complex. However, you raise a good point: defining success should be a team activity. That way we have a common ground from which assess our progress.

For me, right now, "most" is about 75% for an MVP. App authorization is a critical task, so 51% is too low. Run-throughs with the demo suggest that we would probably see at least 65% success with authorizing apps in the wild. So I'm not planning to measure task completion times or anything intensive like that because completing authorization doesn't seem like a big problem. (That could change when we show the demo to members of the public.)

I'm being deliberately loose about defining "significant difficulties," because I don't want to introduce unnecessary complexity into what should be quick turnaround testing. However, I'm informally tracking two basic measures:

• Task level satisfaction
  • how difficult do users report finding the task?
  • how do users feel while performing the task?
• Error severity: if the user makes a mistake, how much does it hamper task completion?

So completing authorization with "significant difficulties" can mean:

• Not a lot of errors but low task satisfaction
  • participants are spending a lot of time figuring out what they need to do
  • or participants don't enjoy or are uncomfortable performing the task (e.g. signing in with PayPal when they would rather use another provider)
• Lots of small errors, none of which are major but which collectively make participants feel like they are slogging through the task
• A big, severe error which doesn't stop the participant from finishing the task but which slows it down considerably or really annoys the participant

Once again, we should decide collectively how to rate difficulties so that we can figure out which ones to prioritize resolving.

[polastre]

This is great @esgoodman. I particularly like the part about satisfaction vs completion. The UK has similar metrics looking at both satisfaction and completion. Knowing what we're aiming for is helpful as we design and re-design MyUSA to figure out what impact our changes are making.

[esgoodman]

Note: moved the text about satisfaction vs completion to comments on #500 because that's the issue that it really applies to.

[esgoodman]

Took out "a first application" from the title because I want this epic to cover not just the first authorization on but also subsequent ones.

[esgoodman]

Related question: #53: How do users respond to auth mechanisms that don't use passwords?

MyUSA currently uses a passwordless sign-on mechanism, so answering #53 helps us answer #513 as well.

[esgoodman]

FOR CONTEXT: This is a broader research question which I anticipated answering gradually, over time. I am still working out how best to document those evolving understandings in a way that's easy for me, accessible to the team, and visible to the public.

[esgoodman]

New research findings on sign-in flow with federal employees.