Pipeline treatment of 1/f noise

[stscijgbot-jp]

Issue JP-2576 was created on JIRA by Anton Koekemoer:

The detectors generally exhibit a significant high-frequency noise component that is highly correlated in time across outputs and reference pixels, ie 1/f noise, manifesting as linear stripes across the detector, and the current baseline pipeline reference pixel correction does not fully account for removing this. This effect has a particularly significant impact on spectroscopic/ grism data but can also significantly impact imaging data, and can potentially limit the science achieved.

So this ticket is to track pipeline enhancements that would remove 1/f noise to a greater extent. Note that NIRSpec data obtained in IRS2 readout mode generally suffer much less from this effect (if at all), therefore the discussion here primarily concerns NIRCam, NIRISS (and MIRI if affected by 1/f noise), at least initially.

Some initial code exists from both the IDT and the STScI side, so the discussion here will aim at examining the performance of different approaches, with the goal of arriving at a pipeline implementation, if feasible.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Note: some relevant earlier background discussion is in the Baseline reference pixel correction page:

https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Reference+Pixel+Correction

 

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Scheduled 1/f noise algorithm discussion for JWST Cal WG Meeting 2022-05-03

[stscijgbot-jp]

Comment by Chris Willott on JIRA:

An algorithm that I am now using for NIRCam and NIRISS data to remove 1/f noise on level 2 images can be found at https://github.com/chriswillott/jwst/blob/master/image1overf.py

 

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Adding a comment to note that we have gotten a number of questions from users about striping or 1/f noise in data products via the help desk, e.g.: 

INC0163572

INC0179841

INC0167474

INC0181624

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

I just want to bring up the option of updating the SIDECAR code to add reference pixels to the subarrays is an option. Eddie has developed modification to the code that would allow any size subarray to have reference pixels. This seems like the cleanest solution, in the long run.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

Presumably this SIDECAR code update would be something similar to IRS2, polling the reference pixels more frequently.  Mike's comment sounds like it is adding reference pixels to subarrays that do not already have them, at least the way I read it. 

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

I think that's great news for subarrays. However we still have a number of help desk tickets from users regarding the residual 1/f noise in their full frame data. 

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Thanks Michael Regan  – also, for NIRCam there was some discussion a few months back with some of the AZ folks (on slack) about eventually implementing a mode similar to NIRSpec IRS2, and we figured that it would be great to have Eddie and others involved. At the time we left it in the hands of Massimo Robberto  and Marcia Rieke for further followup since commissioning was still heavily underway, but maybe this would be a good time to restart those discussions. If Eddie (or you) would like to give a presentation about this at some point feel free to let me know, to help get things going with this.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

In terms of what to recommend to the community for removing 1/f in their data, we are currently suggesting these 3 options, all public:

• Everett Schlawin's "ROEBA" code (runs on individual groups before ramp fitting): ROEBA - Software documentation, code
• Chris Willott's code (runs on cal.fits files): https://github.com/chriswillott/jwst/blob/master/image1overf.py
• Micaela Bagley's 1/f code, from CEERS team (runs on rate.fits files): https://github.com/spacetelescope/jwebbinar_prep/blob/webbinar13/ceers/nircam/remstriping.py

For the pipeline it was decided (at the 2022-05-03 meeting above) not to include 1/f in the operational pipeline as a general step for all affected data, given its empirical nature and its sensitivity to real sources in the image, though for some specialized modes such as TSO it may be sufficiently well constrained to include (TSO WG will be getting back to us once they're ready to suggest a specific implementation). For other modes in general, we currently recommend pointing the community to the above3 public approaches, and may also eventually supporting a general 1/f removal tool as a DATB effort, perhaps based on one or more of the above public codes.

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

Actually, we're on the agenda for an upcoming TIPS to present a menu of possible enhancements. We (mostly Eddie) have done a lot of work prototyping some interesting enhancements but launch and commissioning precluded any implementations until now. There is a process for us to deliver SIDECAR code changes to the GSFC folks for implementation but we cannot do it alone. We need people to ask for the changes.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

That sounds great, thanks. I can revive the discussion for NIRCam (by email this time, and I can cc you) to see about the next steps for getting things going with this.

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

I have created JDOXCM-424 to ask that the three software packages above be added to JDox as 1/f mitigation options. 

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

That's a good solution at the moment, thanks very much.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

For the more specific case of TSO observations, a possible 1/f pipeline approach was discussed at JWST Cal WG Meeting 2023-04-04

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

This does NOT apply to MIRI. It only works for the NIR.

[stscijgbot-jp]

Comment by Karl Misselt on JIRA:

Anton Koekemoer Re: 'recommend to community'; I'm not privy to the details of the recommendations, but it should be emphasized that all of those approaches depend somewhat on a relatively uncrowded fields, especially in the fast read direction. These always remove source flux on e.g. more extended emission (ask me how I know...) and even with point sources with a mask if the there are enough sources given the large JWST PSF. So they might work at an acceptable level of trade off in many programs, but should be approached cautiously with the knowledge that source signal may be compromised.

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I'm assuming Mike meant that it does not apply to MIRI, so am removing MIRI from the list of affected instruments.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Karl Misselt, indeed these approaches are pretty dependent on the properties of the data, and the information provided with each of them describes relevant limitations and caveats.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

tagging Nestor Espinoza  and Nikolay Nikolov  - we have a question from Howard Bushouse which I'll include below - basically the question is, is your proposed 1/f algorithm for TSOs ready for SCSB to start work on implementation? See also the questions Howard raises about the algorithm details, could you answer those please as well?

Hi all,

I just did a review of the Cal WG meeting notes and TSO 1/f presentation from 4/4/2023 and it wasn’t obvious to me exactly what the state of readiness is for SCSB to start any work on this. For example, the discussion notes end with statements like

“Since a primary concern is the accuracy at which the background must be measured, this is not quite ready yet for running this in a fully automated way in the operational pipeline”

And that further discussion can continue in the JP ticket (but there hasn’t been any further discussion there). Also, the presentation on the technique is fairly high-level, without much in the way of specifics (e.g. in places it refers to removing/subtracting a “scaled median frame from each group” – how exactly is the scaled median frame to be computed – taking the median over what: multiple frames, multiple columns, a simple median filtering of the 2D frame, or what?). So it’s not obvious to me that SCSB has what we need to start any actual coding. Or is it expected that Nestor et al. will provide us with a copy of their prototype code, which we are to then port into the pipeline environment?

Any help and/or direction here would be appreciated. I know this is in the priority list for B10.0 (or at least the next quarter).

Thanks,

Howard

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Thanks for tagging into this Anton Koekemoer. Indeed, Howard Bushouse, the idea is that we would lend you folks a prototype function that you can then implement — after proper testing has happened in our end (I already have a function that does this to my own science data, which you can check out here. You can see the function in action here 

As discussed in the presentation, the technique heavily depends on proper background subtraction because the "template" frame (e.g., the median) removed to each individual frame needs to be scaled to account for any TSO signal present in the data (e.g., a transit). The scaling is done on all pixels in a frame, so if any pixels has background signal (where the TSO signal is not present), that would get (wrongly) scaled too.

We are investigating backgrounds in Cycle 2 calibrations. That's the final piece we need to properly test this algorithm. But it indeed, this is not yet ready for implementation until we understand backgrounds better on the TSO modes. Right now, this technique works nicely for NIRSpec/G395H and, if one has the proper background template, for NIRISS/SOSS. Testing is still pending for NIRCam Grism TS; we do see problems with NIRspec/PRISM due to the unknown background.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Nestor Espinoza for the update, and could you please continue to post updates here in this JP ticket as you make further progress on finalizing this algorithm?

Question for Howard Bushouse  – can you give us please the timeline on which you would need this to be specified so that SCSB can start implementing it for B10.0 or the timeline for builds after that?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Anton Koekemoer Based on a relatively quick scan of the code that Nestor Espinoza provided a link to above, I see that the basic correct_1f function is quite simple and straight-forward, but I also notice that it's used in a couple different ways (once when applied to data that've been rearranged into CDS form and another applied to what appears to be the equivalent of "rateints" data), each of which involves the previous computation of things like estimated white-light curves (for scaling during background subtraction), spectral trace info, etc. It's difficult at this point to estimate how much time would be needed to integrate all of that into the existing pipeline, doing all the necessary translations to use our datamodels (instead of plain numpy arrays holding the data), porting of all the supporting functions that compute and provide the prerequisites like the white-light curve, the spectral trace info, etc. without first doing a very detailed review (line-by-line) of everything that's going on all of those functions. Really crude estimate is that it might take somewhere on the order of a month to do all that work to get it into the production pipeline. Right now the tentative B10.0 schedule has us finishing code development near the end of September, so I would estimate we'd need the final version of the prototype 1/f code no later than mid-August (to allow a little bit of a buffer against the deadline). And of course if we get this and something like the MIRI 390Hz noise routine at the same time, there might not be enough resources to work on both at the same time and get them both done in time.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Thanks Howard Bushouse. I was basically asking about the date when SCSB would need the finalized algorithm (which is still in progress by the TSO WG), in order for Nestor Espinoza  et al to have an idea of how much time is available to them to still work out the details if they were to get it to you for B10.0 – so, sounds like it would need to be by mid-August if it were to go in this build.

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Thanks for all this feedback Howard Bushouse and Anton Koekemoer — I don't think it's feasible to have something by August given the current state of affairs in terms of calibrations needed to understand the background in TSO modes. Will keep you updated on progress.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

hi Nestor Espinoza  just checking back on this – can you let us know please if the TSO WG is still working on a 1/f removal script that you'd like to have incorporated into the pipeline (either for operational use or else for offline use initially), or if you'd like to schedule discussions with SCSB for the next steps to start implementing this, could you let us know too please?

[stscijgbot-jp]

Comment by Karl Misselt on JIRA:

[^oneoftest_10052023.pdf] - a quick look at a test of Chris Willott image1overf code applied to NIRCam data on extended source. There are source residuals but are of order ~< 1%, sometimes significantly so. Don't know if this belongs here, move/delete as needed!

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Karl Misselt these tests are great and thanks for posting it here, this is a perfect place to put your slides (and your comment).

To follow up on your result about these residual net changes in source flux (ie source flux being added or removed during the correction, at the level of <~1%) – have you had a chance to check whether these residuals have a different amplitude (and/or direction) for other regions in a given image? Ie, by defining a few more boxes in each image and examining how their histograms behave – eg, maybe a set of 9 boxes in a 3x3 grid, so we can get a sense of whether the net residuals tend to have the same amplitude (and/or direction) along either x or y, or whether these residuals really behave differently in amplitude and direction, in a given image, based on the underlying source structure?

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Hi all - following up on the discussion of 1/f noise at last week's JP Coordination Team meeting (Feb 21), and also previously last year's JWST Cal meeting (Oct 3), we'd like to now get the effort underway for converging on a solution to this issue in the JWST Pipeline for the NIR instruments (NIRCam, NIRISS, NIRSpec).

A new page has now been set up to help coordinate the discussion in terms of the current status and the path forward: https://outerspace.stsci.edu/display/JWSTCC/JWST+Pipeline+1-over-f+Noise+Removal

As a next step, could we ask those from each instrument please to take a look through this page, and update or add any information that's relevant in terms of available software, including software that might not yet be listed, as well as test datasets?

Once that's in hand, we'd like to ask this group to start discussing the options and arrive at a solution to removing 1/f noise in the JWST Pipeline for the NIR instruments, aiming to report back to the JP and Cal Coord Teams.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Anton Koekemoer I'm going to at least temporarily assign this ticket to you, for the INS discussion phase of the work.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Howard Bushouse.

For the INS discussion phase, the next step involves input from the instrument teams to update (if necessary) the list of available software on the JWST pipeline 1/f noise removal page, and also for the instrument teams to list (/update) on that page the test datasets that cover various instrument modes, where the relevant instrument representatives (incl also TSO) are:

• NIRCam: Bryan Hilbert   Armin Rest 
• NIRISS:  Kevin Volk  Rachel Plesha  André Martel 
• NIRSpec: Melanie Clarke  Cheryl Pavlovsky  Nimisha Kumari 
• TSO (for input on TSO modes for each instrument):  Nikolay Nikolov  Nestor Espinoza 

[stscijgbot-jp]

Comment by David Law on JIRA:

This comment formally links this ticket with JP-3639 which addresses the detailed implementation of an initial 1/f correction for Build 11.1

[stscijgbot-jp]

Comment by David Law on JIRA:

As per JP-3639, a generic treatment step for 1/f noise via the optional clean_flicker_noise step has now been introduced into the pipeline and will be available in build 11.1.  See https://jwst-pipeline.readthedocs.io/en/latest/jwst/clean_flicker_noise/index.html

Work remains to test and modify this step, with the goal of providing specific guidance to end users and potentially activating it by default in certain cases.  Such testing will take place via separate tickets as necessary.