search

mozilla / telemetry-analysis-service

Telemetry Analysis Service
https://analysis.telemetry.mozilla.org/
Mozilla Public License 2.0
36 stars 29 forks source link

Job "modules-with-missing-symbols" has due date in the past #888

Open jezdez opened 6 years ago

jezdez commented 6 years ago

from a comment in #553:

This is still happening on my "modules-with-missing-symbols" job.

"It's due next at 2017-12-04 14:25 UTC.", which is yesterday. It was last run on December 1. It is scheduled weekly, so the next date should be December 8.

jezdez commented 6 years ago

/cc @marco-c

jezdez commented 6 years ago

Thanks for the report @marco-c, this seems indeed odd at first, but one thing that isn't obvious to you when looking at the due date is that we had a outage last week that prevented all scheduled jobs to run.

To rectify the situation, I've had to manually run the failed job after fixing the issue causing the outage, including your weekly job.

While doing so it seems as if the next due date was not correctly set and I suspect due to a missing or throttled status update request from AWS that prevented ATMO to save the last run date. In effect the next due date wasn't able to be calculated and ATMO fell back to using the start date as the indicator to calculate it. The displayed due date was wrong in that case.

I've fixed this now by resetting the due date by saving the job once via the web ui, which triggers the due date update mechanism. Note: Since it found a due date in the past it also started the job. Once this run is over the job should be back to the regular schedule.

jezdez commented 6 years ago

I'm leaving this ticket open to follow up once the current run is done.

marco-c commented 6 years ago

@jezdez it's still happening. My job "modules-with-missing-symbols" is "due next at 2017-12-12 19:43 UTC.".