Include request for system feedback at various interaction points in Echo

[tannerwelsh]

Critical Strategic Goals

• !Echo usability measured

  Benefits

What are the benefits of this change, and whom do they impact?

• Learners are able to give direct feedback about the usability of the system in the appropriate places and times (i.e. right after a significant action is taken, and in a contextually appropriate view)
• LOS receives constant quality feedback about the systems we build

  Description

Describe the change, and provide any needed context.

At key points in the UI of Echo, add a component to the view that requests a single piece of user feedback on the interaction the user has just experienced.

This component should include one multiple choice question with each answer being a button, similar to how the SUS Typeform works: https://lguild.typeform.com/to/Hfffz0

The questions should rotate through a set, so that each feedback request event asks the user for a different piece of feedback.

After a user clicks an answer button, they are thanked for their input and directed to the issues page of the los repo for more long-form feedback.

When and where we'd ask for feedback:

• After a retro is submitted, in the submit completion view
• After every 5th review completed per player, as a DM to the player from Echo
• After teams are formed, as a DM to the player from Echo

Since there is no review or team/project view, these two would have to be messages sent by Echo for now. Might be best to wait for a UI for each of these interactions before requesting feedback on them.

UI

Feedback question

On the last page of the retro, show the following form (multiple options provided, looking for feedback on both).

Note that the questions/statements would rotate between a set of 10+ (initially pulled from the SUS), so that learners would see a different one each time until their 11th retro, at which point the cycle would repeat.

Option A: Likert-style statements

Do you agree with the following statement: "I thought the system was easy to use."
[strongly disagree] [disagree] [neutral] [agree] [strongly agree]

Option B: Question with custom responses

How easy is the retro tool to use?
[ super easy ] [ good enough ] [ could be improved ] [ difficult ]

Post-response confirmation & follow-up

After the user clicks the button, the form would be replaced with the following text:

Thank you for your feedback! If you'd like to say more, please [log an issue](https://github.com/LearnersGuild/los/issues).

Rationale

If we want to be collecting continuous usability feedback, we will get more and better data if we make many small feedback requests rather than a few large ones.

In other words, ask for small bits of feedback often (i.e. answer a single multiple-choice question) rather than big amounts of feedback at less frequent intervals (i.e. fill out this survey every few weeks).

This reduces the perceived load on users, and reduces the friction of giving feedback. With enough volume in users, we should be able to build comprehensive reports based on many individual responses.

[tannerwelsh]

Created in response to: https://app.asana.com/0/184273089013827/191014686405316

[tannerwelsh]

would love your feedback especially @prattsj (as User Research) and everyone else in @LearnersGuild/software

[jeffreywescott]

I love this idea!

It seems like it's potentially quite a lot of development work, which makes me slightly nervous with everything we're trying to pull-off in Q4. Hopefully this can be some kind of MVP. I think this is a hugely important goal for the quarter.

[bundacia]

How much value would we get out of just adding some questions about usability to the retro survey? Everyone's already filling that out weekly, so having a question or two extra on it wouldn't increase the load on the learners too much I would think, and it would be dramatically simpler to implement, which means we could have it in place sooner without bumping other priorities as much. That might be a better value in terms of reward/effort.

[tannerwelsh]

Thanks for the suggestion @bundacia - I like it as a first step. I still think having a general-purpose micro-polling tool would be useful, so I don't want to change this issue.

I will create a new card that can be a higher-priority MVP to satisfy our Q4 goals.

[tannerwelsh]

Ok, moved the MVP version to #99 and de-prioritized this one.

Moving this back to backlog for future UI work.

[heyheyjp]

I'm feeling pretty resistant to the idea of prompting every player to submit responses to yet another survey. We're pushing the boundaries of reasonable/effective as it is, IMHO, between all of the COS, space, LOS and whatever other surveys already being administered.

I'd like to propose we see how far we can get first with passive usability data collection using things like google analytics, log analysis, etc.

[heyheyjp]

Taking a more detailed crack at this:

I'd like to challenge the idea that collecting this feedback at the end of every project is the most valuable way for us to measure Echo usability.

We're already asking a heck of a lot of questions. In most cases, such as questions about the COS, the space, and their teammates, the survey approach seems to get us the most bang for the buck. The cognitive load placed on the players/learners is justified IMHO. I don't currently believe this to be the case when it comes to usability, because (A) I don't expect the answers to change significantly between surveys because the rate at which new features are developed is slower than the rate at which projects are completed/retros are submitted and (B) we actually do have other - arguably richer, and certainly less subjective - data available to us for analysis in the form of trackable actions/events in the web app and in Echo, server logs, already submitted LOS issues and already submitted responses to the usability survey.

Consider that this method of collecting usability data could itself negatively impact usability by exacerbating survey fatigue and whether or not other sources of data might yield more value? :)

[heyheyjp]

That said, I do think that it would be wonderful if there were less friction involved in the process for users providing feedback about issues encountered while using the system. I just feel like we're headed down the wrong path by asking specific multiple choice questions and certainly by prompting for feedback so frequently in the app. I'm getting tired just thinking about it. 😛

I like the idea of having a small control visible at all times in the web app(s) that makes it easy to send feedback. A super simple approach would be to have it open the new LOS issue page in a browser window. To get fancier, we could have the system automatically show a dialog of some sort any time an error is captured, asking that the user optionally provide feedback about the experience they just had. Also, the system might ideally be able to transfer a rich set of information about the context in which the user was using the system any time an error occurs.

[tannerwelsh]

@prattsj I saw your feedback is duplicated here and in #99 - is that intentional? Just want to know if there are specific points you'd like to address here or there, so that I know where to respond. Don't want to maintain two parallel discussions :)

[heyheyjp]

@tannerwelsh: yeah, I'm actually confused by these two issues. Is there a clearer distinction we can make between them in the descriptions? One that ideally eliminates any overlap? :)

[tannerwelsh]

@prattsj based on yesterday's convo, let's freeze this issue for now as well.

They are similar in their intent, just different suggested implementations (one an independent survey component, the other an extra question on the retro). Both may be moot depending on how we decide to measure usability.

[tannerwelsh]

Non-essential, closing. Trusting that tension will re-arise.