Add multiprocessing capability to speed up performance as well as robust removal of additional 1/f noise from averaged ramp data used to obtain residuals at the group or integration level:
Add multiprocessing for more efficient 1/f noise modeling. Defaults to 4 processors (diminishing returns for higher numbers) and of course scales to few processes depending on number of available CPUs.
Improved handling of full frame data, such as accounting for the slowaxis being along the x-direction
Subarray datasets also corrects 1/f noise along the appropriate axis depending on value of slowaxis header info. For instance, NIRISS detector orientation is rotated 90 degrees relative to NIRCam, with slow scan direction being horizontal along x-axis.
More robust extraction of 1/f noise model for single frames where one cannot work on residuals (see next bullet for details in that method).
When determining the average slope and building a ideal ramp data or averaged INTS, these averaged data will still have 1/f noise signatures that then get subtracted from the individual integrations when calculating the residuals, so the 1/f noise model never gets to model them. Now, we fit a 1/f noise model to the averaged dataset to first remove it and utilize a "flattening" routine to subtract large variations along the orthogonal direction of the model that are due to over-fitting flux from astrophysical sources. The result is a much cleaner image of the averaged slope image used to determine residuals at the groups and INTs level.
Testing on multiple coronagraphic datasets indicates that modeling on the rateints has slightly better noise performance than at the group level. Models for rateints is now the default via the pipeline.stage_1overf attribute, but can be set to 'groups' if desired. Also means there is less processing time from 1/f correction.
Add multiprocessing capability to speed up performance as well as robust removal of additional 1/f noise from averaged ramp data used to obtain residuals at the group or integration level:
pipeline.stage_1overf
attribute, but can be set to 'groups' if desired. Also means there is less processing time from 1/f correction.