Propagate 1D flat errors

[stscijgbot-jp]

Issue JP-3429 was created on JIRA by Melanie Clarke:

NIRSpec is updating its flat files to include a 1D error array, associated with the fast variation component in the F-flats (JP-3304).  The flat field step needs to be updated to propagate this error from the reference files to the output error arrays.

In particular, the combine_fast_slow function needs to be updated to take an error array as input and interpolate it onto the wavelengths for the observation.  The interpolated error should then be added in quadrature to any error from other components.

Two implementation notes:

1. Both zero and NaN are possible values for all flat error arrays, 2D and 1D.  The S and D flats will not have error arrays for their 1D components, so these values will default to NaN; their 2D error components will be set to zero, since all flat errors are accounted for in the F-flat instead. When combining flat error values, NaN values should be ignored and zero values should be treated as valid numbers.  

2. The 1D flat vector in the combine_fast_slow algorithm is interpolated with a 3-point Gaussian integration.  A rigorous error propagation through that step should take into account cross-correlations between the pixels that are weighted and summed together, but we don't recommend that.  The flat errors are not likely to be known to enough precision to justify the complexity.  Instead, we recommend interpolating the error vector onto the wavelengths with a simple linear interpolation, e.g.:   error_value = np.interp(wavelengths, tab_wl, tab_flat_error, left=np.nan, right=np.nan)

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Melanie Clarke We're getting ready to make our second release candidate of the Cal pipeline for DMS build 10.0. Would it be feasible to try to get these updates to the flat-field step done in time so that they can be included, and hence have the code ready to support the new flat ref files once they become available? Of course that means the code updates would need to be backwards compatible, at least for a while, so that they can continue to use the existing ref files that don't have the 1D error array. Looks like Brett Graham took a stab at the code updates in #7972 , but that PR got closed without merging (not sure why).

[stscijgbot-jp]

Comment by Brett Graham on JIRA:

I had taken an initial stab at the update. There are a few things unfinished with the code and given the small amount of work that went into it and the unfinished state I decided against suggesting it as a starting point. The PR does not:

• the regression test run shows many nan ERR hdus which indicate that the nans returned from the updated code 'polluted' downstream error values. I expect that further updates are required to update downstream error combination code to handle nan values (I'm not sure if these were possible before at this stage in the pipeline).
• combine the error calculated from the reference files with other sources (as described in this ticket). The changes in the PR used the error calculated from the reference file only if no other error was yet calculated in combine_fast_slow. After seeing this ticket I expect that will need to be changed.

If it's useful please feel free to take a look at the changes and I'm happy to answer questions and help with the update.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Thanks Brett Graham . I hadn't considered the potential downstream effects and the fact that some other steps might need updates to change the way they handle error propagation.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Brett Graham Re: nan pollution, see also JP-3250. I'll link it to this ticket, as there is some shared discussion.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Howard Bushouse - I can take a look at implementing this, including the downstream propagation if desired, but it would be helpful to have a decision on the issues in JP-3250.  Greg Sloan - do we expect a decision soon?

If we do not want to update the downstream propagation at this time, I can replace NaN values with zero for this specific purpose, when missing 1D values are expected from the S, D, and older F-flats.  That would at least leave us with the same situation we currently have.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

I put together a PR, starting from Brett Graham's work but moving 2D + 1D error propagation to combine_fast_slow as well, so that components can be added in quadrature. I handled NaN values from either input as zeroes for now, so this should at least do no harm to later propagation assumptions.  This handling can be updated later when decisions are made about how to handle NaN errors in general in the pipeline.

These changes should only affect NIRSpec flats, and are in line with the team's assumptions in generating the flats.

I have spot-checked the code with FS input file jw01125001001_0310g_00001_nrs1_rate.fits, with the current CRDS F-flat and with a test F-flat as override (attached), containing 1D errors that are 0.1 * the flat values.  Results from the current CRDS flat are identical to data processed with the master branch. Results from the test flat appear to propagate appropriate errors at a glance, but I still need to verify the values.

Let me know what you think of this approach.

 

Edited to add:

Output VAR_FLAT values with the test F-flat look good to me.  Wherever data and error are valid, the output flat variance is (data * 0.1)**2, as expected.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

I also tested IFU, MOS, and BOTS data:

• IFU: jw01335004001_03101_00002_nrs1_rate.fits
• MOS: jw01345061001_07101_00003_nrs1_rate.fits
• BOTS: jw01981011001_04102_00001-seg001_nrs1_rateints.fits

with similarly constructed test flat files.  After fixing a few overflow problems in handling the variance values, output looks good to me for these as well, with both the current flats and the test F-flat with non-zero 1D errors.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #7978

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Tested with v1.12.2 and working as expected.