/auto_triage consistently failing on perf dashboard

[anniesullie]

The /auto_triage endpoint is supposed to do the following:

• Mark untriaged anomalies which have recovered as recovered. (TriageAnomalies)
• Mark anomalies associated with bugs which have recovered as recovered. (TriageBugs)

It attempts to do all of TriageAnomalies and list the bugs for TriageBugs in one post request, with task queue tasks to check each bug. It runs on a cron every 6 hours. It ticks some custom variables when it completes, and they haven't been ticked in the 6 weeks of history I can find in stackdriver.

But looking through the logs, it always fails, timing out in TriageAnomalies.

I think there are three things wrong here:

• auto_triage, TriageAnomalies, and TriageBugs are confusing names. This endpoint should be called something more like mark_recovered_anomalies.
• The check for whether an anomaly is recovered takes too long to do multiple times in one request. Each check should be in a taskqueue task.
• The way we list bugs is to query bugs associated with all anomalies filed in the last 30 days. I think instead we should get the labels auto-applied for each sheriffing rotation, and list the bugs that are still open via monorail query. No point following up on closed bugs, and sometimes things recover after more than 30 days.

[simonhatch]

This has no hope of finishing, there are 18.5k anomalies in the list. Will look closer at options later.

[simonhatch]

@anniesullie did you end up looking at this?

[anniesullie]

Not yet. Likely won't have time this week.

[anniesullie]

This will be fixed with the next dashboard rollout.