hpc4cmb / toast

Time Ordered Astrophysics Scalable Tools
Other
44 stars 39 forks source link

Improve flagging and noise estimation #695

Closed tskisner closed 9 months ago

tskisner commented 1 year ago

We have two primary flag bits for shared and detector data:
"invalid" and "processing". The invalid bit means that the data does not exist or is wrong / corrupted in some way that makes it useless for any application. The processing bit means that the data should not "typically" be used for science results.

This work attempts to change all built-in simulation and reduction operators to follow these basic guidelines:

  1. Operators that load data:

    • Boresight pointing marked as invalid if no pointing solution exists
    • Detector data marked as invalid if it is missing (e.g. dropped packets, etc)
    • In some cases, there may be data that should not routinely be used for science, and these samples should be flagged with the processing bit.
  2. Operators that simulate telescope observing:

    • Unless the operator is also simulating failed pointing solutions or readout problems, no samples should be flagged as invalid.
    • Some samples might be flagged with the processing bit if those features are typically known by low-level data acquisition systems. Examples might be cooler cycles or scan turnarounds, although those samples might also be flagged by downstream operators.
  3. Operators that simulate detector timestreams:

    • Existing invalid detector samples should have their flags passed through and remain invalid
    • Detector samples flagged with the processing bit should have this bit passed through.
    • If the simulation requires data inputs (e.g. pointing), and that data is flagged as invalid, then the output samples of the simulation operator should be flagged as invalid.
    • If the simulation uses inputs flagged with the processing bit, it is up to the operator whether to use those samples.
    • If the simulation is unable to generate some samples, then those samples should be marked as invalid since they do not contain the full expected detector response
    • If the simulation generates poor-quality data for some reason, those samples might be marked with the processing flag
  4. Operators that process detector timestreams:

    • Existing invalid detector samples should have their flags passed through and remain invalid
    • Detector samples flagged with the processing bit should have this bit passed through as well.
    • It is up to the operator whether to use samples flagged with the processing bit.
    • If the operator is modifying the detector data, and fills some samples with garbage, zeros, etc, then those samples should be marked as invalid.
    • If the operator is modifying the detector data, and is unable to use / process some data, then those samples should have the processing bit set.

This PR also include improvements to flagging in the Offset template and a typo fix in fitting noise models.

tskisner commented 1 year ago

Summarizing a discussion from other channels:

Principles

Implementation Plan

tskisner commented 9 months ago

Ok @keskitalo , this PR is ready for re-review. There is a small issue when running on GPU with hybrid pipeline data movement, but that will likely just be a typo fix somewhere. Will not merge until those tests pass (which are not tested by CI anyway).