search

getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
39.05k stars 4.19k forks source link

[WIP] Better alerting about outdated SDKs #27715

Open lobsterkatie opened 3 years ago

lobsterkatie commented 3 years ago

Problem Statement

This is related to the larger project of which https://github.com/getsentry/sentry/pull/27711 is a part - that of improving our alerting about SDK updates - and is my thoughts from the perspective of an SDK maintainer about stuff we might want to consider. Originally I left a few of these questions as a comment there, but that's a frontend PR and in reality most of the work to address these issues should happen on the backend, so I decided to move this to an issue instead.

(Note: I'm sure that some of these things are already being considered, so apologies if that's the case.)

Solution Brainstorm

Thoughts:

1) In the mockup of that PR, the same project is listed multiple times, with different versions as the current version. We should just take the most recent one and show the project once.

2) Can we create a map of internal SDK name (sentry.javascript.browser) to external SDK name (@sentry/browser), so that we can display the latter rather than the former?

3) IDK if this applies to any raven SDKs other than raven-js (maybe the old PHP SDK?), but at least for raven-js, it'd be great if we could mimic what Markus's original in-event banner did and look for indicators that the user is using a framework we specifically support, so that we can tailor our suggestions appropriately. For example, if someone is using raven-js and we can tell from events they send in that they're also using react, we could suggest @sentry/react rather than @sentry/browser.

4) I see a real danger in this being more noise than signal and therefore its overall purpose getting lost. Bottom line, obviously, is that we want to be showing the right information to the right people at the right time so that the upgrades are most likely to happen. Thoughts on this score:

a) This should definitely be (at minimum) scoped to  "My Projects." 

b) Even out of those, for any given user, there are likely only a handful that they pay close attention to/are significant enough players in that they might be able to actually make the upgrade happen.

c) Some upgrades matter more than others, and some kinds of updates happen more frequently than others, and overall the less frequently we bug people about this, and the more we restrict it to just the "important" updates, the more likely they are to take it seriously. We're addressing the largest factor here (deprecated vs not) in the PR linked above, but there are others. Low-hanging fruit here would be to only alert on major and minor updates and ignore patch releases.

d) IDK how feasible various of these possibilities are, both in terms of what data we store in the first place and how easy it is to query that data, but factors we could consider in terms of restricting which projects' alerts are shown to which users and how they're sorted: 

  - Has the user interacted with the project in the last _n_ days/weeks/months? 
  - How recently have they interacted with it? 
  - How often have they interacted with it compared to other projects? 
  - How often have they interacted with it compared to other users?
github-actions[bot] commented 3 years ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

priscilawebdev commented 3 years ago

@lobsterkatie here is some feedback:

  1. In the mockup of that PR, the same project is listed multiple times, with different versions as the current version. We should just take the most recent one and show the project once.

I manually changed the project id for testing purposes, so I don't think we have this issue on production. The screenshot was only to display the new design with "deprecated tags" and the alert

  1. Can we create a map of internal SDK name (sentry.javascript.browser) to external SDK name (@sentry/browser), so that we can display the latter rather than the former?

I agree with you! we are getting the data from the backend, so I think it would be better to update it on the backend and send the already updated data to the frontend

  1. IDK if this applies to any raven SDKs other than raven-js (maybe the old PHP SDK?), but at least for raven-js, it'd be great if we could mimic what Markus's original in-event banner did and look for indicators that the user is using a framework we specifically support, so that we can tailor our suggestions appropriately. For example, if someone is using raven-js and we can tell from events they send in that they're also using react, we could suggest @sentry/react rather than @sentry/browser.

That would be great!

4 . I see a real danger in this being more noise than signal and therefore its overall purpose getting lost. Bottom line, obviously, is that we want to be showing the right information to the right people at the right time so that the upgrades are most likely to happen. Thoughts on this score:

a) This should definitely be (at minimum) scoped to "My Projects."

Not sure what you mean here. In the front-end PR, I updated the code to show updates only to those who are a member of the project or have permission to see it.

b) Even out of those, for any given user, there are likely only a handful that they pay close attention to/are significant enough players in that they might be able to actually make the upgrade happen.

Agree.

c) Some upgrades matter more than others, and some kinds of updates happen more frequently than others, and overall the less frequently we bug people about this, and the more we restrict it to just the "important" updates, the more likely they are to take it seriously. We're addressing the largest factor here (deprecated vs not) in the PR linked above, but there are others. Low-hanging fruit here would be to only alert on major and minor updates and ignore patch releases.

Agree.

d) IDK how feasible various of these possibilities are, both in terms of what data we store in the first place and how easy it is to query that data, but factors we could consider in terms of restricting which projects' alerts are shown to which users and how they're sorted:

It would be awesome if we could do these checks, displaying alerts only to the right users

untitaker commented 3 years ago

Can we create a map of internal SDK name (sentry.javascript.browser) to external SDK name (@sentry/browser), so that we can display the latter rather than the former?

this data exists on the serverside. I have no idea how the endpoints for this look nowadays but we definetly have this mapping, may just not be exposed.

cc @scefali @EvanPurkhiser, not sure who did the backend for this