search

sdss / marvin

Data access and visualization for MaNGA. http://sdss-marvin.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
57 stars 32 forks source link

current top 10 issues on Sentry #454

Open havok2063 opened 6 years ago

havok2063 commented 6 years ago

In the free mode of Sentry, we have an event history of 7 days after which items get removed. We haven't been resolving issues, so we've lost a lot of information. We were gifted 5000 free events until our cycle resets (the 17th). We can currently see some things again. Here are the top ten issues in Sentry:

Issue Title, Event Count, Estimated Date Range

Some of these are just localhost and/or dev errors before I turned on that filter. Some of these are internal errors captured normally by the code by users. Some may indicate real bugs, e.g. the top two issues. If Marvin is working correctly these are endpoints that should never be triggered. Have a look here for the full list. https://sentry.io/manga/marvin/dashboard/

albireox commented 6 years ago

Having a quick look at this, I'm fairly sure the first two are not issues, but Sentry reporting raised exceptions that are caught in a try-except. Every time you use the decision tree it uses the local mode first. If that fails it raises the "this is the end of the road message", which is caught and is the indication to try remote. Which means that we have had 80k requests for data ... not bad. The second is similar, I think. The third may be similar, if we raise and error while trying to check whether sdss_access is installed and, if it is not, fall over to the extern package. I think the "no local database found. Query cannot ..." may fall in the same category. The others seem legit but mostly related bad input parameters.

albireox commented 6 years ago

The "no DB connection found" comes from mangaid2plateifu, btw, and it is try-except'd.

havok2063 commented 6 years ago

80k in the last year at least. This makes sense, given how we utilize Sentry. If we don't want to capture these errors, we should reconsider when we send an error to Sentry. If we detach Sentry from explicitly raised MarvinErrors, then I think we'll only get uncaught exceptions, which is maybe what we ultimately care about. Or we sign up and set up an Inbound Filter to remove these particular errors.

albireox commented 6 years ago

If there is a way of filtering these by the string of the message then I think that will work. There are probably only a handful of these that cause most of the problems. The others are probably valid errors. Another thing is to differentiate errors due to invalid inputs and real errors but that may be harder to do.

Also, if we set an Inbound Filter, will the excluded messages count towards the limit of messages in Sentry?

havok2063 commented 6 years ago

The free Sentry provides the ability to filter out errors

On a paid plan we can filter out errors with specific messages and/or specific software releases. I imagine this would open up if we get approved for access as an open source project.

Messages excluded using Inbound Filters do not count towards the limit of Sentry events. So the more filters we apply that can cut out the crap we know we don't want, the more real errors we'll get.