Add min/max wavelength range for flux summation in TSO white light calculation

[stscijgbot-jp]

Issue JP-1469 was created on JIRA by Bryan Hilbert:

During the discussion about the white light step for NIRCam grism TSO data in the Calibration Working Group on 26 May 2020 , it was decided that having a wavelength/pixel range over which the 1d spectrum is totaled would help to reduce the noise currently seen in the white light curve. By including the higher uncertainty flux calibration at the red and blue ends of the bandpass, the white light curves produced by the pipeline in version 0.16.1 and before are very noisy and can end up with negative values. These curves make subsequent data analysis not useful. 

 

 

 

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

Min and max wavelength values should be instrument/filter/mode dependent, and should be provided via parameter reference file.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Suggestions on the content of such a reference file are welcome. Once a consensus is reached, SCSB will create a data model for it. One format/content aspect that SCSB will STRONGLY recommend for this new ref file that will contain relatively little data is that it be in ASDF format (no more FITS BINTABLES!).

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Don't worry, Howard. The parameter reference files are all ASDF. :D  ([https://jwst-pipeline.readthedocs.io/en/latest/jwst/stpipe/config_asdf.html)]

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

Maybe this is for a later discussion, but would this be a set of parameter reference files with one file for each mode? Or more like a 2D table of wavelength bounds vs mode where the pipeline would have to select the appropriate row?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

(y)

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Bryan Hilbert It could be done either way. The first approach means more ref files, but simpler cal pipeline code. The second is just the opposite. There aren't a huge number of spectroscopic TSO modes available: NIRCam TSGRISM, NIRISS SOSS, NIRSpec BrightObj, and MIRI LRS Slitless. This already amounts to only 1 mode per instrument, so by the time you break the ref files down to the instrument level, there's only 1 mode to cover in each. NIRCam TSGRISM and NIRSpec BrightObj may need multiple entries to handle different choices of filter and grating+filter, respectively, but that can easily be accommodated through the use of selectors within the ref file itself.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

after discussion in our team meeting, the MIRI team has assigned this as high. while we agree that the wavelength limits are required to produce the white light time series with the right noise properties, we don't consider this critical for commissioning. As we understand it, the step from the time series of 1-d extracted spectra - the output of calspec2() - to the white light time series doesn't test any particular health aspect of the instrument, nor does it require an essential calibration reference file (apart from algorithmic input from a parameter reference file) so this shouldn't affect our science readiness decision point. it will however affect the quality of the science that can be done with the data, so we do consider it a "high" priority.

 

cc Misty Cracraft

 

 

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

This was discussed in JWST Cal WG meeting 2020-08-25 and is accepted for pipeline version expected by launch, so the "calwg" label has been set and this is assigned to the DMS WG for further tracking, including the relevant reference file work.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

This has been implemented using step parameters in the white_light step. The parameters min_wavelenth and max_wavelength are used to create an array mask for the summation. Instrument/mode-dependent parameter reference files can be created to override the default None values. The pull request is linked here.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

I've attached an example parameter reference file that I used to test the pipeline edits. Note that these wavelength bounds are applied to the complete spectrum, so issues with noise at the ends of spectral orders rather than the complete spectrum may require more work than has been done for this ticket.

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Just a follow-up comment here, for completeness. The instrument reps are discussing the parameter files with their respective teams. When we're ready to deliver them, we'll use the linked ticket CRDS-389 and associated subtasks.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

New functionality added, related ticket CRDS-389 for INS delivery of parameter reference files.

[stscijgbot-jp]

Comment by Nikolay Nikolov on JIRA:

I tested the new implementation using 'strun jwst.white_light.WhiteLightStep' on Mirage NIRCam Grism time series without photometric correction. Running my own code to extract light curves, I find identical results in terms of light curve morphology and scatter to the results of the WhiteLightStep. Here is a slide summarizing my test results: [^JP1469NN.pdf]

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

(y)