search

m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

Some jobs aren't matching in tracker #252

Open gfr10598 opened 4 years ago

gfr10598 commented 4 years ago

two symptoms:

  1. Sometimes a job appears twice in the status page, but only one of them is being updated.
  2. Sometimes the etl updates are not matching, and the job status in gardener stays stuck in INIT. Seems to have been introduced earlier on April 1. Happy April Fools Day. Grrr.
gfr10598 commented 4 years ago

Note - this is showing up in sandbox, when running u-sandbox-dedup-with-config branch, several different commit points.

gfr10598 commented 4 years ago

Still showing up. When gardener restarts, recent jobs end up with dups in the status. The new jobs never leave the init state, but the old jobs continue to be updated, so it is clear that the old jobs are still synced with the work that etl parsers are doing.

Perhaps this is related to the clock skew in the Date field, which is a time.Time. When K8S restarts the gardener job, perhaps it has a different clock skew, which results in a different map key.

This could be resolved by changing the field to a civil.Date, which seems like a good idea anyway.