buda-base / ao-workflows

Use DAG platform to define and orchestrate workflows
0 stars 0 forks source link

Handle missing SQS notifications #26

Closed jimk-bdrc closed 2 months ago

jimk-bdrc commented 2 months ago

Fixes #19 Fixes #24

Creates a DB object and an ORM (GlaicerSyncProgress) to track progress. An operator has to use extras.py:add_works_to_project to build the list of works

The GlacierSyncProgress ORM is updated by each step's successful completion (staging_utils:db_phase).

the dataset (underlying table is `drs.glacier_sync_progress) can be queried for time stamps of completed steps.

One shortcoming is that the DB needs to be pre-populated with the universe of work. It can't create records on demand. #25 is a request to fix this.