search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
569 stars 167 forks source link

Bullseye artifacts in NIRSpec sflats #8188

Closed stscijgbot-jp closed 3 months ago

stscijgbot-jp commented 10 months ago

Issue JP-3507 was created on JIRA by David Law:

When helping a colleague with some NIRSpec IFU data analysis I noticed that the sflat for their program (jwst_nirspec_sflat_0194.fits) has numerous bullseye artifacts.  See attached screenshot.  Where most flatfield values are near 1, these artifacts can have values ranging from 0.7 to 4.0.  On running the pipeline, it looks like these artifacts are being imprinted into the science data when the flatfield is applied.

stscijgbot-jp commented 9 months ago

Comment by Melanie Clarke on JIRA:

The NIRSpec calibration team will take a look; Nimisha Kumari will coordinate.

stscijgbot-jp commented 9 months ago

Comment by Nimisha Kumari on JIRA:

The file delivery was discussed in this ticket: https://jira.stsci.edu/browse/NRSCOMM-189 

I am looking for the "meeting notes" of this discussion.

stscijgbot-jp commented 9 months ago

Comment by James Muzerolle on JIRA:

Thanks to a comparison with on-sky observations by Cheryl Pavlovsky , I think the issue is contamination from zeroth-order images of failed open MSA shutters. We should verify from the CAP documentation, if possible, but if this turns out to be the case, there is no way to recover valid S-flat correction factors. However, the DQ array should be adjusted to set all the affected pixels to UNRELIABLE_FLAT so that incorrect values are not applied to science data.

stscijgbot-jp commented 9 months ago

Comment by David Law on JIRA:

One other possibility worth considering would be to interpolate over the artifacts when generating the flat.  This is what I did for the science case above and seemed to work well in the science results.  The interpolated flat would be uncertain at the percent level, but that may be better than rejecting otherwise good-looking science data in those positions.

I'm effectively doing something like this in the MRS flatfields as well; emission lines in our flatfield calibration target contaminate certain regions of the detector so that I can't derive a reliable correction there, so I interpolate the flat over those regions in some reasonable way.

stscijgbot-jp commented 9 months ago

Comment by Nimisha Kumari on JIRA:

https://innerspace.stsci.edu/display/JWST/NIRSpec+CDP5?preview=/357983781/357983801/ESA-JWST-SCI-NRS-RP-2022-003.pdf

James Muzerolle & David Law The issue is discussed in the above document in Section 5.1 and Figure 6. it says that    "Currently there is no solution to this problem but the pixel affected are flagged with 2 (unreliable flat) and can be sorted out during extraction."  

stscijgbot-jp commented 9 months ago

Comment by James Muzerolle on JIRA:

Thanks, Nimisha Kumari ! Unfortunately, the flagging was not done correctly, so we'll have to fix that. In the longer term, we can look into David Law's suggestion of applying an interpolation.

stscijgbot-jp commented 7 months ago

Comment by David Law on JIRA:

Adding some additional notes from looking at this last week (already circulated via email, but included here for posterity):

Draft notebook for modifying sflats is attached, along with a screenshot of the results (pending evaluation by the NIRSpec team)

stscijgbot-jp commented 6 months ago

Comment by Bethan James on JIRA:

Update: The NIRSpec team are still looking into this. There are some concerns about the DO_NOT_USE flagging, which results in NaNs between the slices and on the low throughput edges. We'll be discussing our test results in the calibration forum and will report back when we can.

stscijgbot-jp commented 6 months ago

Comment by David Law on JIRA:

Thanks for the update Bethan James .  Just to clarify, the DO_NOT_USE/NaN values between the slices doesn't change much if anything- the inter-slice regions are already NaN in the cal files, this change just helps make that explicit in the flatfield step instead of relying on the pathloss step to achieve this.  For the low-throughput edges, these edges are the direct cause of many of the outliers in the resulting data cubes, and this change is targeted at cleaning these up.

stscijgbot-jp commented 6 months ago

Comment by David Law on JIRA:

Quick update; after talking to Peter Zeidler I revised the notebook a little to help make sure that it wasn't losing too much very-blue light where the throughput is ok but the cal lamps are faint.  Also cleaned up some of the weird single-row glitches, see glitches.png attached.

Latest notebook and flats that it produces can be found at https://stsci.box.com/s/sft1e3jkffrc7ffsq9zzpjslmabukwf2

stscijgbot-jp commented 5 months ago

Comment by Bethan James on JIRA:

Thank you for the update David Law . I've finished performing a series of tests, inspecting the cal files and cubes reduced with current vs new versions of the IFU S-flats (new versions = those generated using the code that you supplied on 5/14). Specific attention was paid to the blue wavelength edges, and assessing areas with bullseye artifacts in the old vs new pipeline products.    Everything looks good on our end and we have no concerns about implementing these in the pipeline. If you're happy for us to proceed, we can pass the newly generated s-flats to our CRDS team and begin the delivery process.  Peter Zeidler and Nimisha Kumari are also in agreement with moving forward.

stscijgbot-jp commented 5 months ago

Comment by David Law on JIRA:

Great, sounds good Bethan James.  Just to check this was the version that produced files titled '_may2024.fits', right?

stscijgbot-jp commented 5 months ago

Comment by Bethan James on JIRA:

That's correct

 

stscijgbot-jp commented 5 months ago

Comment by David Law on JIRA:

Sounds good.

stscijgbot-jp commented 3 months ago

Comment by David Law on JIRA:

Closing this issue, as I believe all of the updated flats have now been delivered to CRDS in time for 11.0 reprocessing.