stscijgbot-jp
commented
4 months ago Comment by Ned Molter on JIRA:
It remains an open question what exactly to do about this. Here's my two cents:
- The ATOCA algorithm should not be attempted on datasets that are saturated in orders 1 and 2. IMO this should be handled by special processing tables/instructions, and not by the pipeline.
- The pipeline should therefore throw an error in these cases; however, the error should be easier to understand, e.g., it should check the reasonableness of the (rotation, xshift, yshift) and raise if these are outside the expected tolerances.
- I haven't checked this yet, but my guess is that the transform parameters are poorly constrained in these cases; that's also something that we could check in the pipeline, although it would be a bit more work. If this were done, we could choose to set the transform parameters to zero and proceed with a warning instead of an error (although my first reaction is that I would rather the pipeline throw an error)
- On the JWSTDMS ticket it was mentioned that Nestor Espinoza's local dev version of ATOCA does not throw this error; if this code is going to be overwritten in the near future, I'd rather not implement a patchwork solution now, given that I expect this failure mode to be relatively rare.
Issue JP-3650 was created on JIRA by Ned Molter:
Error message ValueError: Some pixels do not follow the expected dispersion axis! is seen when attempting run the ATOCA algorithm on saturated, highly flagged, or other bad data. Initial analysis suggests that for these datasets, the soss_solver finds unreasonable solutions for the (rotation, xshift, yshift) transform that attempts to align the expected positions of the spectral trace with the real position. For "good" data, the rotation angle should be much smaller than one degree (of order 1e-3 degrees), but for e.g. the OTB mock data this rotation angle was found to be 3 degrees.
The transform is applied to the wavelength map using an FFT method. The wavelength map is padded by 20 pixels in all directions (by default) to mitigate any artifacts caused by the edge of the detector during the transform. However, for such a large rotation angle, artifacts of the FFT end up in the corners of the frame, and are never masked properly; see attached image. These artifacts have incorrect wavelength values, and this is caught when the code checks that the wavelength values are strictly ascending/descending, leading to the error.
This failure mode has occurred on a dataset where Orders 1 and 2 were purposely saturated to attempt to retrieve a spectrum from Order 3. At present, the pipeline does not support extraction of Order 3. It has also occurred on some OTB data that display nothing resembling a true spectral trace. It appears that this failure mode should not occur for good data in orders 1 and 2, because there would be no situation in which the actual rotation angle of the detector was different from expected by such a large amount, far outside the pointing tolerance.