Open tskisner opened 1 year ago
This would be an excellent support feature. Is there also a feature/ planned feature to write the detector timestream data to Disk instead of holding all in memory, such that the user can also make maps later from data on disk one observation at a time?
For simulated data, one can already use the SaveHDF
operator to write per-observation HDF5 files with the contents from memory. For real data, there can sometimes be overhead loading data from disk due to file formats (for example, unpacking frame-based data or re-ordering data that is stored as detector values for each sample). In that case it might also be desirable to load raw data and write to HDF5 if the intention is to repeatedly read data from disk.
That's great! thanks
Currently the
MapMaker
operator assumes that the entire detector timestream data volume is in memory. However, with a small change this could support making maps of data while loading one observation at a time and making several passes through the data. This issue outlines the design of that:loader
andpreprocess
traits to the MapMaker class which are optional class traits. The loader operator should populate detector data in observations one at a time when itsexec()
method is called, and also provide "rewind()
" and "purge()
" (or similar) methods.preprocess
trait of the binning operator.Pipeline
containing the filtering operators can be passed as thepreprocess
trait.