simonsobs / sotodlib

Simons Observatory: Time-Ordered Data processing library.
MIT License
16 stars 19 forks source link

Preprocess: Error and jobs progression handling #988

Open msilvafe opened 1 month ago

msilvafe commented 1 month ago

We need to implement better error handling and process progress tracking. Currently we write to an "error_log" but whole obs will be skipped if a single group hits an error state. We would like to be able to progress through possible groups and keep track of which ones fail and why. We also want to skip these if we rerun the script, because currently we have to fail on them every time. This could use integration with @mhasself's JobDb currently in PR: https://github.com/simonsobs/sotodlib/pull/851