zooniverse / bajor

Azure Batch Job Runner - BaJoR
Apache License 2.0
2 stars 1 forks source link

Look at adding a job state persistence reporting store #1

Closed camallen closed 2 years ago

camallen commented 2 years ago

Provide job monitoring feature (celery background worker?) to monitor submitted Zoobot batch jobs.

These would track the job state and update as the jobs complete / error.

Also look at adding a polling job status API for requesting this data (via long polling). Similar batch job monitoring code via Azure Cosmos at https://github.com/zooniverse/CameraTraps/blob/master/api/batch_processing/api_core/server_batch_job_manager.py#L127

camallen commented 2 years ago

we're using the azure batch job resources / API as the job tracking store.