NIRSpec FLAT4 lamp exposure crashes in flat_field step

[stscijgbot-jp]

Issue JP-2209 was created on JIRA by Howard Bushouse:

A NIRSpec IFU mode lamp exposure in LRE4 using the FLAT4 lamp and G140H grating crashes during the flat-field step with the following traceback:

Traceback (most recent call last):
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/bin/strun", line 28, in <module>
    step = Step.from_cmdline(sys.argv[1:])
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/stpipe/step.py", line 173, in from_cmdline
    return cmdline.step_from_cmdline(args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/stpipe/cmdline.py", line 339, in step_from_cmdline
    step.run(*positional)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/stpipe/step.py", line 407, in run
    step_result = self.process(*args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/pipeline/calwebb_spec2.py", line 131, in process
    raise RuntimeError('\n'.join(failures))
RuntimeError: Traceback (most recent call last):
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/pipeline/calwebb_spec2.py", line 113, in process
    result = self.process_exposure_product(
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/pipeline/calwebb_spec2.py", line 246, in process_exposure_product
    calibrated = self._process_nirspec_slits(calibrated)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/pipeline/calwebb_spec2.py", line 396, in _process_nirspec_slits
    calibrated = self.flat_field(calibrated)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/stpipe/step.py", line 407, in run
    step_result = self.process(*args)
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/flatfield/flat_field_step.py", line 122, in process
    output_model, flat_applied = flat_field.do_correction(
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/flatfield/flat_field.py", line 78, in do_correction
    flat_applied = do_nirspec_flat_field(output_model, fflat, sflat, dflat,
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/flatfield/flat_field.py", line 297, in do_nirspec_flat_field
    return nirspec_ifu(output_model, f_flat_model, s_flat_model, d_flat_model, dispaxis,
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/flatfield/flat_field.py", line 549, in nirspec_ifu
    flat, flat_dq, flat_err, any_updated = flat_for_nirspec_ifu(
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/jwst/flatfield/flat_field.py", line 1703, in flat_for_nirspec_ifu
    ind = np.indices((dy, dx))
  File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.2.3.20210609-py38/lib/python3.8/site-packages/numpy/core/numeric.py", line 1777, in indices
    res = empty((N,)+dimensions, dtype=dtype)
ValueError: negative dimensions are not allowed

In addition to this hard error, the assign_wcs step also issues the following warning:

2021-07-17 02:35:13,743 - stpipe.Spec2NRSLamp.assign_wcs - INFO - gwa_ytilt is 0.1351139000000003 deg
2021-07-17 02:35:13,744 - stpipe.Spec2NRSLamp.assign_wcs - INFO - gwa_xtilt is 0.3576557639999985 deg
2021-07-17 02:35:13,744 - stpipe.Spec2NRSLamp.assign_wcs - INFO - theta_y correction: 0.005495014592365932 deg
2021-07-17 02:35:13,745 - stpipe.Spec2NRSLamp.assign_wcs - INFO - theta_x correction: 0.0 deg
2021-07-17 02:35:13,760 - stpipe.Spec2NRSLamp.assign_wcs - INFO - Combination FLAT4_G140H missing in wavelengthrange file, setting order to -1 and range to [7e-07, 1.27e-06].

when using ref file jwst_nirspec_wavelengthrange_0004.asdf

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

From James Muzerolle:

Yes, there should be an entry for this combination, as well as for G140M+FLAT4. These are equivalent to the G140M/H+F070LP science settings. We'll have to deliver a corrected version.

The flat lamp processing needs to include the D-flat correction because those data are used to construct S-flat reference files. The only dependency for the D-flat is wavelength, so it should be ok as long as assign_wcs is run first. Maybe the problem stems from the missing lamp in the wavelength range reference file? What happens in that case, are the pixel wavelength values set to NaN?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

According to the log messages above, the assign_wcs step defaulted to a wavelength range of [7e-07, 1.27e-06] (0.7 to 1.27 microns). Does that seem reasonable for the F070LP+G140M/H combination? If not, i.e. if that doesn't cover the actual range of the data, perhaps that what in turn leads to the error during flat-fielding.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Howard Bushouse that default wavelength range is exactly right, so shouldn't be the reason for the flat field error. Looks like it's not finding the correct slice limits in either x or y; which detector is being processed here? NRS2 will not have any valid pixels for this grating/lamp combination.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Ding, ding, ding! Yes, it's an NRS2 exposure. So I guess this is another combination that we need to add to existing level-2b ASN rules, so that we don't create a spec2 ASN and hence no attempt at level-2b (or level-3) processing will be made.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Yes, exactly.  Only for IFU and all FS except S200B1.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

James Muzerolle updating the level-2b ASN rules to fix this. Should the only exclusion be for the specific combination NRS2+G140H+FLAT4? Are other grating+lamp combinations OK for the NRS2 detector, such as G140H+FLAT1, G235H+FLAT2, G395H+FLAT3, G140M+FLAT1, G140M+FLAT4? I see those combinations in other exposures in program 812, obs 10 and 11.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Howard Bushouse no, there are other cases. It's analogous to science data. NRS2 should be excluded for IFU and FS (except S200B1) with G140M, G235M, G395M, or PRISM. NRS1 should be excluded for FS S200B1 with the same gratings.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

So there's no dependence on which lamp is in use (e.g. FLAT1, FLAT4, LINE1, LINE2, TEST, REF, ...), only on grating?

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

The only time the lamp matters with regards to detector coverage is for the combination G140H+FLAT4. Everything else should be unique. (And there is no equivalent among the LINE lamps).

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

I've linked issue jp-1466, which fixed detector coverage for science-based FS and IFU modes. Have not read the above in detail, but do the lamps honor basically the same restrictions?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Yes, that appears to be the case. The solution I'm working on right now is to add the same "valid_detector" constraints to the lamp ASN rules that already appear in the science case rules. Some small-level tweaking may be necessary.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

James Muzerolle using exposures from program 00812, obs 10 and 11 for testing this, which contain a lot of IFU and MSASPEC lamp exposures, taken with a large variety of different lamps and gratings. Zeroth-order attempt is to simply apply all of our existing exclusion rules (actually they're written as inclusion rules, i.e. only create an ASN for certain detector+filter+grating combos) for science data to the lamp exposures. Couple of questions about the results I'm seeing so far:

1) All of the OPMODE=MSASPEC lamp exposures are getting treated the same way as OPMODE=IFU lamp exposures. Is that OK? I assume that a given grating would yield the same results in terms of data falling on the NRS2 detector for both IFU and MSA modes, right?

2) The current science data rules only create ASNs for any exposure using the G140H grating when FILTER=F100LP, or when FILTER=F070LP and DETECTOR=NRS1. But since FILTER='OPAQUE' for lamp exposures, this set of rules is not creating any ASNs for any lamp exposures using GRATING=G140H. I assume that's too restrictive, right? Is it OK to allow G140H lamp exposures (with IFU and MSASPEC mode), except when LAMP=FLAT4 and DETECTOR=NRS2?

3) The current science data rules also only create ASNs for any exposure using the PRISM when FILTER='CLEAR' and DETECTOR=NRS1. Again, since lamp exposures all have FILTER='OPAQUE', this is excluding all lamp exposures that use the PRISM, so I assume that this is also too restrictive. What lamp PRISM modes should be allowed?

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

1) In general, no.  For MSASPEC, there are many shutters that will disperse onto NRS2 with G140+FLAT4 or PRISM+FLAT5/LINE4. It depends on the MSA configuration. assign_wcs already has a check for that, so if there happen to be no shutters that disperse onto either detector, there shouldn't be any problems. This all has to do with the geometry of the aperture positions - all but one of the FS and the IFU entrance aperture are to one side of the instrument FOV, which corresponds to the "blue" side in the dispersion direction and aligns with the NRS1 detector. Both detectors see light only for the high-resolution gratings in these cases, because the higher dispersion results in longer spectra.

2) Yes, we need to be able to extract lamp spectra for G140M/H.

3) PRISM with lamps has rules analogous to the science filter case: no data on NRS2 for IFU or any of the FS except S200B1, and no data on NRS1 for S200B1.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

For the case of pure lamp exposures in FIXEDSLIT mode, i.e. EXP_TYPE=NRS_LAMP, as opposed to an AUTOWAVE or AUTOFLAT taken in conjunction with a science exposure, the user does not specify a slit, only a subarray (FULL, ALLSLITS, SUBS200A1, SUBS200B1, SUB2048, etc.). I assume that all lamps will illuminate all slits and hence there's no decision to be made based on the choice of the S200B1 slit. But will all subarrays result in data on both detectors, i.e. are there some subarrays that may cutoff light from falling on NRS2 from the S200B1 slit? (and vice versa for data on the NRS1 detector)

[nden]

The conversation above is a mixture of ASN rules and spec2 processing. Excluding ASN rules for a moment, I want to mention, in case this helps, that assign_wcs makes a decision about which open slits are "valid" by running the model on each of them and excluding those that do not project on the detector. To get all "valid open" slits after assign_wcs and before extract_2d is run one can use

slit2msa = input_model.meta.wcs.get_transform('slit_frame', 'msa_frame')
open_slits = slit2msa.slits[:]

extract_2d is run only on those slits determined to be "valid" by assign_Wcs.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Nadia Dencheva the case that triggered this ticket in the first place (use of lamp "FLAT4" in IFU mode) is a tricky one that does not currently get handled by the logic you mention in the assign_wcs step, due to the fact that the cutout filer that's used with the FLAT4 lamp prevents data from falling on the NRS2 detector. But assign_wcs doesn't know that, because it has no knowledge of the cutoff filters that are paired with each of the lamps. There's no meta data that tells you that (for all lamp exposures FILTER='OPAQUE'). So this is one corner case where the checks in assign_wcs fail to stop the calwebb_spec2 processing.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Howard Bushouse yes, the subarrays complicate matters.  The slit-specific subarrays will only contain data for that slit, by design. SUB2048, SUB1024A/B, and SUB512 are all meant for the S1600A1 slit.  In all cases, however, data are obtained from both detectors.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

OK, so it sounds to me like the logic that's already built into the assign_wcs will handle things for us when attempting to process FS lamp exposures using subarrays that may cutoff data from one or more slits. It specifically checks to see which slits end up having data falling onto each detector and if there aren't any, it automatically aborts (with a clean shutdown) the rest of calwebb_spec2 processing. This is the same behavior that could occur for lamp exposures in MSASPEC/MOS mode. For that mode, the ASN generator rules have no idea which shutters are open and hence whether data will fall on a given detector, so a level-2b ASN will always be created and it'll be up to the assign_wcs step to determine whether any of the shutters yield data on a given detector and if not, it'll abort. So I don't think we need to do any other special handling in the ASN rules for lamp exposures in either MSASPEC or FS mode.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed in #6304

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

B7.8.2 justification:

1) Why do we need it? To prevent the level-2b pipeline from crashing when trying to process NIRSpec lamp exposures taken in IFU mode with the "FLAT4" calibration lamp. 2) What does it fix? The FLAT4 NIRSpec lamp does not result in any dispersed data falling on the NRS2 detector. When attempting to process NRS2 data using that lamp, the current level-2b pipeline will crash due to the lack of data on the detector. This fix modifies the Association Generator rules so that level-2b processing of these exposures is not attempted to begin with. Processing will quit at the end of level-2a. 3) What happens if we don't include it in B7.8.2? Lamp exposures taken in this mode during commissioning and calibration activities will cause pipeline crashes.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Example datasets contained in LRE4RR-OTB test suite: jw0081201[01]001_0210[9at8rs]_00001_nrs[12]

B7.8.2 (Cal 1.3.2) test run2499 (LRE4RR-OTB on 2021-09-08) produced the same set of jw00812-o010nrslamp-spec2 and jw00812-o011nrslamp-spec2 ASN's as the local tests run for this PR and no error reports for any level-2b processing, confirming that the fix works.