search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
58 stars 45 forks source link

Removing 390Hz noise from LRS SLITLESSPRISM data #556

Closed taylorbell57 closed 6 months ago

taylorbell57 commented 1 year ago

Some MIRI subarrays exhibit 390Hz noise in the pixels within a group. We roughly knew about this from the LRS TSO commissioning paper, but Michael Ressler did some great investigative work and showed that there is a clear 390Hz noise in the group-level data which has a fixed waveform but has a varying phase. I did some testing myself and determined the waveform for the LRS SLITLESSPRISM subarray and wrote code to find the phase. Interestingly, I found that the phase remains constant within an integration, and while the phase varies a bit between integrations, I noticed that it has cyclical variations with a period of 8 integrations (in at least one dataset). I couldn't figure out a clean way to deterministically compute the phase however, so I just fit it in my added code.

I had hoped that this would improve the high-frequency noise I've observed in LRS spectra (the noise is in the final transit/eclipse spectra). However, I have found that the 390Hz step seems to have nearly zero impact on the transit/eclipse spectrum, while the related MIRI-specific GLBS code I added at the same time also has little impact. That said, I don't think I'm the only one who would want to investigate this effect, so I am still contributing these edits to allow others to investigate the impact on their data and allow for potential further improvements.

I will note that this code was only built with the LRS SLITLESSPRISM subarray in mind and has been tested on several such observations, but a few other subarrays also exhibit a similar noise pattern. I don't have any such observations to test the code on, so for now this step should only be applied to LRS SLITLESSPRISM data.

taylorbell57 commented 1 year ago

I'll also point out that it's theoretically possible that this 390Hz noise removal could be combined with separate background observations to try to avoid self-subtraction. I believe early attempts at using separate background observations after the science observations were not very successful, but perhaps there'd be more success with the inclusion of this 390Hz noise removal code applied to both the science and calibration observations.

kevin218 commented 8 months ago

What's the timeline on this PR? No hurry, just curious.

taylorbell57 commented 8 months ago

It'll probably only take me an hour or two to make all the required edits, just need to find the time to get around to that

codecov-commenter commented 6 months ago

Codecov Report

Attention: 183 lines in your changes are missing coverage. Please review.

Comparison is base (8c56120) 58.91% compared to head (dd9da00) 56.94%. Report is 4 commits behind head on main.

:exclamation: Current head dd9da00 differs from pull request most recent head 74626e5. Consider uploading reports for the commit 74626e5 to get more accurate results

Files Patch % Lines
src/eureka/S1_detector_processing/remove390.py 9.82% 156 Missing :warning:
src/eureka/S1_detector_processing/ramp_fitting.py 16.66% 15 Missing :warning:
src/eureka/S1_detector_processing/s1_process.py 0.00% 11 Missing :warning:
src/eureka/lib/smooth.py 75.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #556 +/- ## ========================================== - Coverage 58.91% 56.94% -1.98% ========================================== Files 96 97 +1 Lines 11653 11858 +205 ========================================== - Hits 6865 6752 -113 - Misses 4788 5106 +318 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

taylorbell57 commented 6 months ago

Just realized that I still need to add axis labels to Figure 1301 - I'll aim to do that today

sebastian-zieba commented 6 months ago

Do you have an idea how your code compares to the 390 Hz correction, which is implemented in jwst==1.13 (https://github.com/spacetelescope/jwst/pull/7857)? Couldn't we just use what the jwst pipeline does?

taylorbell57 commented 6 months ago

@sebastian-zieba, I haven't done a whole bunch of testing yet, but I did do a test recently with jwst==1.13.4 and found that there was still significant 390Hz noise after starting with my previously downloaded _uncal files. I'll run another test or two to double-check that the emicorr step is being run (and re-download the _uncal files if I need to). I've also already contributed to a few papers that reference this new Eureka! feature, so I still want to add this code (at least for now) so that the results of those papers are reproducible.