MRS pixel replacement modifications

[stscijgbot-jp]

Issue JP-3345 was created on JIRA by David Law:

https://jira.stsci.edu/browse/JP-3005 implemented pixel replacement for slit spectroscopy, along with a draft implementation that could be used for MRS as well.  While it performed fairly well, there were still clear failure cases in which the replaced values were significantly incorrect.  At the same time, pixel replacement is less critical for the MRS since the cube building algorithm natively does a degree of replacement using neighboring pixels as part of the voxel overlap computation, and the decision was made to keep this step off for MRS for the time being.

More recently, analysis of data from PID 1268 has shown that NaNs in the cal files are the source of small (generally negative) spikes in the extracted spectra.  The issue is far less pronounced than for the LRS.  The root cause seems to be undersampling; when NaNs occur in the middle of a bright point source trace cube build fills in the corresponding cube voxel using adjacent pixels that are significantly lower in flux.  This is mitigated somewhat by additional dither positions, but not fully.

As a result, I've looked back at the pixel replacement approach and am testing a revised implementation to see if it can do a better job.  This minimum gradient approach is super simple: For every NaN in the cal array, calculate the absolute gradient along the spatial and spectral axes (X and Y) using immediately adjacent pixels.  Pick whichever dimension has the minimum absolute gradient and replace the missing pixel with the average of the two adjacent pixels along that dimension.

This aims to make the process extremely local as global approaches will not generally be reliable; near point sources it will do the replacement along the spectral axis avoiding sampling issues, while near bright extended emission line the replacement will be along the spatial axis.  It may still be suboptimal near bright emission lines from unresolved point sources.  There is no attempt at any replacement if a NaN value is bordered by another NaN value along a given axis.

Test results to follow.

[stscijgbot-jp]

Comment by David Law on JIRA:

spectrum.png illustrates performance on an extracted spectrum; both the profile fit and mingrad methods improve upon the original spectrum noticeably

pixels.png shows performance near a bright emission line; profile fit can sometimes incorrectly estimate replacement values while mingrad results look reasonable

Test data sets: PID 1268 Observation 1, Ch3LONG.  PID 1523 jw01523001001_39101_00001_mirifushort

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

I remember seeing some incorrect replacement values when running the profile fitter on MRS data, though in your figure I am curious to know which regions are being incorrectly fit. The "absorption" (?) feature around 16.71 um is reproduced in the mingrad output but is flattened in the profile fit - do you have an idea what the truth value is expected to be in that region?

[stscijgbot-jp]

Comment by David Law on JIRA:

Tyler Pauly  See what you think of #7823

The problem areas are in the pixels.png image; bright dots right around the tilted emission lines.

Interesting question on the 16.71 um feature (which is not real).  It looks like that's pixel 973,524 in jw01268001001_02101_00003_mirifulong.  The NaN-valued pixel is exactly atop a trace that's extremely well centered in one pixel.  Ordinarily it would use the spectral-domain interpolator but it's so well centered that the spatial difference is slightly less than the spectral difference so it interpolates in the less-optimal direction.

[stscijgbot-jp]

Comment by David Law on JIRA:

Adding a note on relative runtimes for spec2:

No pixel replacement: 400 seconds

profile_fit replacement: 1778 seconds

mingrad replacement: 419 seconds

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Code looks good, I'll give it a proper review. Interesting that you don't exclude neighbors with DO_NOT_USE set, though perhaps they're expected to be NaN? A large gradient would not be selected if the neighboring bad pixel is finite but wonky, though if both neighbors happen to be finite but wonky, that might pose a problem? Probably a very small edge case, if anything at all.

Interesting that the mingrad 2d figure appears to be clearly superior, but the largest discrepancy I see in the 1d plotted spectra lies in the mingrad spectrum. Perhaps just the nature of the regions plotted.

The time taken to iterate through IFU slices is a problem, but because they all have curved traces and/or sharp features that dissuade profile fitting, I suppose that algorithm will likely not ever be activated for them.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Also, I'm curious about the values/stretch on the 2d image for the pixel on the edge of the far right slice, just under the emission feature. It appears that profile fitting overestimates the value, while the mingrad may be underestimating it - presumably the vertical gradient is quite large due to the emission, so it averages a non_science pixel with the other neighbor?

[stscijgbot-jp]

Comment by David Law on JIRA:

I went back and forth on excluding things with DO_NOT_USE.  The issue I ran into is that all between-slice pixels are flagged DO_NOT_USE, and it's nice to clean those up at the same time even if they don't currently feed into the final data cubes.  Nominally everything that's DO_NOT_USE within the slices should already have np.nan flux anyway.

On the far-right pixel: looks like the profile fit value is 297374 and the mingrad fit is 37168.  It should probably be about 60000.  Spectral interpolation would give it about 151000, while spatial interpolation with the off-slice pixel gives it the value of 37168.  Seems like a weird case where it probably shouldn't interpolate in that way, but doing so gives a better answer than anything else...

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #7823

[stscijgbot-jp]

Comment by David Law on JIRA:

Tested and works as expected.