`site_pipeline.record_qa`: Observations that fail to process pile up

simonsobs / sotodlib

Simons Observatory: Time-Ordered Data processing library.

MIT License

14 stars 17 forks source link

`site_pipeline.record_qa`: Observations that fail to process pile up #952

Open tristpinsm opened 5 days ago

tristpinsm commented 5 days ago

When an obs_id fails to process its metrics for whatever reason, it isn't added to the InfluxDB such that it show up in the list of unprocessed observations on the next run. Over time, these pile up and will slow down the flow runs.

A solution could be to add an entry in the Influx log that records the failure.

msilvafe commented 5 days ago

I think Matthew envisioned this jobdb being used for that purpose: https://github.com/simonsobs/sotodlib/pull/851 but I have yet to test and review this PR :/

tristpinsm commented 5 days ago

Oh interesting. Could make sense to migrate to that system then rather than continue using Influx to track this.