Anomalous resample_spec rectification of some MIR_LRS-FIXEDSLIT data

[stscijgbot-jp]

Issue JP-3322 was created on JIRA by Tyler Pauly:

-Some LRS-FIXEDSLIT data is being rotated by resample_spec, causing the narrow vertical trace to be drizzled across multiple pixels in what appears to be quite a noisy process. The rotation is unexpected and seems to cause a poor spectral extraction.-

Edit: The tilted trace is expected, and not specific to this dataset nor an issue to investigate. However, the resampling of the trace during rectification appears to be much noisier than expected, and is the focus of this ticket.

Reported by Help Desk ticket, for jw02044-o006.

[stscijgbot-jp]

Comment by Greg Sloan on JIRA:

Hi Tyler: I have a bunch of questions for you. First up, is this dataset unusual, or is this problem more widespread than we have recognized before? And do you have any thoughts on what is happening when we rectify the data? I didn't quite follow "rotated". There's more, but we can start here.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Greg Sloan I might have to defer to you, or the MIRI team generally, on if this is unusual data - I have not seen something like this before. The rectification during resample_spec uses the backward transform of the wcs, i.e. it provides the wcs with the target RA/Dec location and a wavelength array, and the wcs returns an array of (x,y) positions of length equal to the wavelength array. This position list shows the source moving in the cross-dispersion axis, i.e. it predicts the trace should be tilted, and so the step attempts to rectify this. Why the wcs predicts the trace to have this tilt is what I'm currently looking into.

[stscijgbot-jp]

Comment by Greg Sloan on JIRA:

My look at the data indicated that the signal had been scrambled, not transformed to some incorrect trace shape. In other words, it had been verticalized (if that's a word!), but the spatial profiles in each wavelength element were no longer smooth.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Tyler Pauly if the resampling algorithm is using the target RA, Dec as an input, can this be the cause of the problem, seeing as we know the target RA, Dec does not always accurately reflect the actual RA, Dec of the pointing (see also spectral extraction & path loss problems)?

[stscijgbot-jp]

Comment by Greg Sloan on JIRA:

Sarah Kendrew, this possibility came up in a recent discussion. Also of note is that the target is relatively close, making proper motions important and absolute position important.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

I wanted to record an update (for future me and/or others) on the symptoms presented in the dataset so that we can better understand how to diagnose the bug.

Looking at program ID 2044, observations 5 and 6, it appears as though the resampling/rectification of the 2d spectra is introducing significant noise in the spectral trace. I'm including figures showing the results of a routine that fits a gaussian profile to each spatial slice of the spectral trace, then plots the location and width of the gaussian as a function of wavelength. The cal files show a well-behaved fit, while fits to the s2d files show noise in both the location and width of the gaussian. This is evident when examining the 2d spectra by eye, though I refrain from attaching them here due to the EAP attached to the program.

[^jw02044005001_03103_00001_mirimage_cal_fit_byrow.pdf] [^jw02044005001_03103_00001_mirimage_s2d_fit_byrow.pdf] [^jw02044005001_03103_00002_mirimage_cal_fit_byrow.pdf] [^jw02044005001_03103_00002_mirimage_s2d_fit_byrow.pdf] [^jw02044006001_06101_00001_mirimage_cal_fit_byrow.pdf] [^jw02044006001_06101_00001_mirimage_s2d_fit_byrow.pdf] [^jw02044006001_06101_00002_mirimage_cal_fit_byrow.pdf] [^jw02044006001_06101_00002_mirimage_s2d_fit_byrow.pdf]

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

The cause of the noise in the resampling appears to be due to structure in the read noise variance array. The structure looks like a result of long integrations accumulating cosmic rays, which truncate integrations for some pixels.

I checked the readnoise reference to see if any structure is present: [^readnoise_reference.pdf] The scalebar units shows DN here, and the variation between pixels is relatively small. The cutout matches the pixel extent shown in the figures below, and includes the bounding box for the LRS spectral region.

The readnoise variance array for these data (specifically, jw02044006001_06101_00001_mirimage) shows a general value of order ~3.e-4, but with roughly circular patches with much smaller values, ~1.e-6: [^cal_var_rnoise.pdf]

With three integrations, I looked at each variance array generated in the calints file: [^calints_var_rnoise_int0.pdf] [^calints_var_rnoise_int1.pdf] [^calints_var_rnoise_int2.pdf] From these I infer that the holes are caused by cosmic ray/snowball events that are truncating ramps during ramp fitting, such that the read noise values are much lower than the value expected for an integration of 300 groups.

I think there are two solutions we can consider - the simplest would be to set resample_spec to use a weighting of "exptime" rather than "ivm". This would set the weight of each pixel to be equal (for spec2 level resampling, where only exposure is resampled), while "ivm" is weighting pixels using the readnoise variance. The other option, which would require testing and likely some updates to extract_1d, would be to turn off resample_spec and extract directly from the detector image. This may not be the right time to do this, but could happen in the future if/when PSF-based extraction is implemented.

It's worth noting that if there is an issue in how the readnoise array is being generated, these solutions are workarounds and not fixes - unfortunately, I don't think I'm capable of knowing whether or not the readnoise array is reasonable.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Assigning to MIRI LRS folks following the prior comment - not sure if Greg Sloan is the correct LRS lead? Sarah Kendrew or Andreea Petric maybe?

[stscijgbot-jp]

Comment by Greg Sloan on JIRA:

I would be the correct assignee. Before leaving for the AAS meeting, I tested the two weighting options on a faint target, and unfortunately, I did not see any improvement with the exptime option. But the test was of pretty limited scope, so I will need to try again and be a bit more rigorous before I can say anything definitive.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

I thought the Spec2 resample_spec step doesn't use the weighting as it is not combining 2 images? How is the weighting being used on a single file?

 

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Because a rectification is being applied, more than one pixel in the input image is contributing to each output image pixel. They are weighted both by their expected contribution to the output pixel (i.e. overlap fraction) and also by their weighting, set by ivm or exptime.