search

simonsobs / sotodlib

Simons Observatory: Time-Ordered Data processing library.
MIT License
16 stars 19 forks source link

Detector Calibration Refactor #942

Closed jlashner closed 2 months ago

jlashner commented 2 months ago

This PR adds the update_det_cal.py site pipeline script. This contains the det-cal update logic that was was previously in update_smurf_caldbs, along with the ability to run the TES parameter corrections documented here.

Due to the time consuming nature of the detcal correction and its fitting, some care had to be taken in the parallelization process. I've found that there are different optimal ways to parallelize this depending on whether you're running at NERSC or on site computing. IO to NERSC sqlite files is slow, so it is beneficial to parallelize the the obs-info loading in addition to the correction computation. The NERSC method chains together two separate multiprocessing pools, one for sqlite access and the other for param correction, and allows you to set the number of processes in each pool individually.

On site computing, sqlite IO is faster, so we don't need to parallelize the sqlite access. Instead we create a pool of however many processes we want, and run through observations serially, using the pool to parallelize the TES param corrections by running each detector correction in a separate process.

The method run is settable via the cfg object / file.

We can still run without the calibration correction by setting cfg.apply_cal_correction = False.

The TES param correction code itself is in sodetlib here.

jlashner commented 2 months ago

@mhasself do you think you can point out the init.py file I need to change? I don't see where in the site pipeline init or the sotodlib init I should be adding something.

mhasself commented 2 months ago

@mhasself do you think you can point out the init.py file I need to change? I don't see where in the site pipeline init or the sotodlib init I should be adding something.

Ah, I was misremembering. It's actually in so-workflow (here). So, not your job right now.

For compatibility with Prefect and the so-site-pipeline CLI, see https://sotodlib.readthedocs.io/en/latest/site_pipeline.html#module-sotodlib.site_pipeline.cli. Adding stuff to site_pipeline/cli.py should break the CI, as desired...

jlashner commented 2 months ago

Thanks for the comments @mhasself, I think I went through and got everything. I need to rebuild the database with the naive estimates though.