search

simonsobs / sotodlib

Simons Observatory: Time-Ordered Data processing library.
MIT License
16 stars 19 forks source link

depth1 mapmaking - a "preprocess" module that bins a TOD into a map set #562

Open mhasself opened 1 year ago

mhasself commented 1 year ago

As suggested by @tskisner during DRP telecon.

tskisner commented 1 year ago

The reason I asked about this issue (it was more of a question than a suggestion) was in the context of a toast SitePipeline operator that I am designing. In this work I had been planning to use a preprocess "Pipeline" and its config on each toast process to operate on its local set of detectors. This assumes that the pre-processing is a set of serial operations. Since your mapmaker things are MPI parallel, this would need some additional logic to see if a preprocess operator was using MPI or not and if so, run that collectively with the toast process group working on a given observation. It sounds like I should at least add a stub for that possibility (that some pre-process ops use MPI).

mhasself commented 1 year ago

The mapmaker does not naturally split single observations across different MPI tasks (because, in the Max-L model anyway, single-obs detectors are correlated and need to be accessible in the single task). Rather, MPI is used in multi-observation mapmaking (especially iterative mapmaking).

So in this problem, where we're incorporating a single obs (or a single obs-wafer) into a map, MPI would not be implicated (at least not in the current plan). I certainly had no plans to run preprocess pipelines in MPI with an observation split across tasks.

Indeed a sensible boundary for problems that preprocess pipelines should solve is "stuff you could do with a single observation, in a single process".

So I wouldn't prioritize making this MPI aware.