Collect user feedback when Stuff Goes Wrong©

[anders314159]

I think that it is valuable to solicit and collect user feedback whenever Stuff Goes Wrong© for the user in Racetrack.

For example, a Racetrack user deploys, but the deployment Goes Wrong©. In the UI, or CLI, a small non-annoying Thing™ appears and asks "Hey, the deploy looks bad - do you have any feedback for the Racetrack admin regarding this?". The user can then select either:

1. "I fucked up", and then Racetrack admin doesn't care too much.
2. "Racetrack fucked up for reason X", and the user can write a comment and a Racetrack admin should probably look at it along with related logs.
3. "Solar radiation flipped a bit, blame the Sun", and Racetrack admin doesn't really know what the Sun is and doesn't care too much.
4. Or ignore the Thing™ entirely.

The Racetrack admin can then collect feedback and make an issue here summarizing frequent problems.

This is a very vibe'y question, and I am not sure if anyone has asked for this, but maybe Ford's Faster Horses quote applies.

[anders314159]

Reasons why it is a bad idea:

1. It creates multiple channels of feedback - the user is unsure if face-to-face feedback to admin is better than Thing™ feedback.
2. The admins might ignore it.
3. This feels a bit like creating Jira tickets, which should be avoided according to this.
4. The users might ignore it.
5. The users might use it, and get frustrated if it feels like their feedback is taking too long to be addressed.
6. etc.
7. We get sampling bias from people who are willing to click the feedback button - meaning they are either very angry with Racetrack, very happy, or just plain bored.
8. Are we gonna need it?
9. Are we gonna use it?
10. Is something else already covering this need, and I just completely overlooked that something?

[JosefAssadERST]

Hm. It's an interesting idea, but my instinct is to push back (I don't like pushing back because I like brainstorms and thinking outside the box, and a blanket no discourages this).

If something goes wrong, I prefer for the software to tell me what the problem was, not the user. Via better logs for example.

I can't really think of any other software that does this kind of thing. Maybe stuff like Microsoft productivity software?

[anders314159]

On balance, I also don't think we should do it right now - the projects using RT right now is fairly small, and I think the feedback loop is okay right now.

But imagine a scenario where a single Racetrack instance is serving millions of users (next year, at the latest) - e.g. if multiple users keep cancelling deployments because they take too long, it would be nice to know that, in addition to logs.

I've seen something like this feature in phone apps, and I heard in a podcast that Facebook (caveat) did this for their ml model deployment platform.

I'll close this, since it isn't really "an issue" and we agree to not do it^{right now}, but where would discussions like this otherwise go?

[JosefAssadERST]

if multiple users keep cancelling deployments because they take too long, it would be nice to know that, in addition to logs

This kind of information should come to the instance admin from RT, the admin shouldn't have to go looking for it. For example, tracking build time and cancellations, having those in Prometheus metrics, and having alerts or Grafana alarms.

That we control and can implement. Users will just send us nudes when they get mad.

[anders314159]

Conjecture: User sentiment is important to the adoption of Racetrack. Conjecture: Logs don't capture user sentiment nearly as well as logs augmented with user feedback. Conjecture: Users aren't malicious (big if true). Contingent-on-the-above conclusion: Gathering user feedback is important to the adoption of Racetrack.

But I don't think this discussion is converging on anything, so let's set it aside for another day.