search

m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

add load/dedup/copy/delete to state machine #303

Closed gfr10598 closed 3 years ago

gfr10598 commented 3 years ago

This PR requires the ETL parser to be configured in GCS output mode.

  1. updates the ops state machine to include Load from GCS, Dedup, CopyTmpToRaw, and Delete.
  2. adds improved error handling for action errors.
  3. updates the status page and logs to include a bit more detail about each step.
  4. Adds state information to the TasksInFlight metric.

This change is Reviewable

coveralls commented 3 years ago

Coverage Status

Coverage increased (+0.2%) to 59.413% when pulling b4acf175e3eb5a4fb588f57d8a3b458567594b3d on load-from-gcs into df4cdccafd15e13e8f8aebb7a2d4501ec47c9b90 on master.

gfr10598 commented 3 years ago

This PR should be submitted more or less at the same time as m-lab/etl#923