Gardener should reprocess jobs on parser failure

[cristinaleonr]

If an ETL parser is interrupted and never completes a job, Gardener marks the job as stale and moves on. Instead, Gardener should try to reprocess the job.

Context: https://github.com/m-lab/dev-tracker/issues/723

[stephen-soltesz]

Looking closer at the current gardener implementation the following are true:

• the job-service has always generated jobs sequentially, either historically or daily (iterators)
• the etl parser maintains no state of its own other than its current job (which can be lost on a restart)
• the etl parser requests jobs from the gardener, and should send Update() and Heartbeat() messages periodically
• the gardener purges "stale" jobs (in any state) after 6h (from k8s config, -job_expiration_time=6h)
• the gardener uses monitor.Watch to regularly marshal jobs through various actions & states
• there is currently no "stale parsing retry" logic (hence this bug).

Using the current architecture, I think these are possible approaches:

• A new date iterator that accepts out of band "stale" parse jobs that should have otherwise been completed
  • this complicates max-retry logic, additional monitoring for degenerate conditions,
• Rearchitect the management of jobs to be a complete, finite set - this could easily fit in RAM or disk
  • At most: 365 days (Now.YEAR - 2009) 20 datatypes == 94900 states
  • This re-architecture could include "make"-like constraint solvers - do only what needs to be done.
    • e.g. is BQ table up to date? new parser version? new GCS files? new archive files?
• TODO: ... others ...

[cristinaleonr]

Update: this issue happened for pcap on 06/10/2024.