Detection Work

[wedamija]

The detectors are mostly complete. Mostly we just want scalability things around them, so we need:

• When we mark a url failed, we set a key with a 7 day expiry so that we won't attemp to process it agian. We should just add some jitter around this expiry so that the expirations are more spread out. Maybe the jitter is just 7-10 days instead of exactly 7?
• ~Lots of stats around all the detection code - I haven't had time to instrument it yet.~
• ~A feature flag around actually creating the subscriptions so that we could enable detection and see how it works without creating subscriptions~
• ~Implement robots.txt fetching~
• ~https://github.com/getsentry/sentry/pull/73355 needs thorough tests~
• When we start monitoring a url for a project we want to stop collecting and processing data for that project for a period of time (a week? use jitter)
• When we fail to find any useful urls for a project multiple times in a row we also want to stop collecting and processing data for some period of time.
• ~We need to have a project setting to disable automatic montoring for the project~
• ~We need an org setting to disable automatic monitoring for the whole org~

[evanpurkhiser]

Is this good t oclose @wedamija ?

[wedamija]

Yeah I think all the points are addressed