Error abort in extract_1d for NIRISS SOSS with F277W

[stscijgbot-jp]

Issue JP-3150 was created on JIRA by Howard Bushouse:

The latest NIRISS SOSS ATOCA code for extracting 1-D spectra does not currently support observations using the F277W filter, due to that mode not being calibrated (yet?). Right now the extract_1d aborts with an ERROR and terminates the rest of the spec2 pipeline.

Should we consider changing the asn_generator rules to not create asn's for SOSS observations using the F277W filter, which would then stop DMS processing at the end of level-2a and avoid these errors? It would also avoid the unnecessary time spent running the first three-quarters of the spec2 pipeline, only to abort near the end.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Alicia Canipe Need the NIRISS team to weigh in on this question of whether to stop processing at the end of level-2a (rate/rateints) for all SOSS exposures using the F277W filter. If the SOSS group within the IDT thinks they'll have calibration for this filter in the near future, we could leave as is for now and just live with the aborts that happen in Ops.

[stscijgbot-jp]

Comment by Jo Taylor on JIRA:

Nestor Espinoza Joseph Filippazzo Tyler Baines What are you thoughts about F277W processing?

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

Yes, right now the codes that produce the reference files for the SOSS extract1d step do not generate files for the GR700XD+F277W configuration. I'm not sure when this will be updated but the SOSS team can discuss its priority. In my opinion, it would make sense to stop DMS processing at the end of level-2A for now. 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

We could potentially set things up so that level-2b processing is still at least partially applied, but we just skip the extract_1d step, so that a calints product would still be produced, but no extracted spectra (x1dints file). I'd have to check with the DMS folks to make sure that the absence of that expected product would not mess up anything in the higher-level processing.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

I went ahead and wrote a draft PR with Howard Bushouse's suggested change, because it was very simple to implement and I had just written something similar for SOSS data in the FULL subarray.  Can we move forward with this solution, or does the absence of an x1dints file cause problems?

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

I think this is a reasonable fix, pending an okay from DMS folks. There is a possible issue, though, that with a successful result from spec2, that we will expect DMS to process spec3 for these data. extract_1d may fail gracefully, but white_light likely will not. In addition, the only step that spec3 would run (if we fix white_light behavior) would be outlier_detection - it may be worth asking INS team members if this would be useful, but I would suggest that we no longer build level 3 associations for this filter. This could be revised if/when NIRISS calibrates this filter.

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

The SOSS team still feels that processing should be halted at the end of level 2A processing for now (it can be revisited if these reference files are delivered), so there is no need to run the spec3 outlier_detection, and it is fine if no level 3 associations are made for this filter/grism combination. Please let us know if any more information is needed for implementation of this.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Rachel Plesha If processing halts at the end of level-2a, that means you will only get rate/rateints products for these exposures. It also means that the association generator rules should be updated so that they do not create any "spec2" or "spec3" associations for these exposures. So you won't get any products at all from level-2b processing (calints, x1dints) or level-3.

If that's OK, we can proceed with that plan.

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

Howard Bushouse yes, that is my understanding that it is fine to not get the calints of x1dints or level 3 products for this combination. Joseph Filippazzo or Nestor Espinoza please correct me if I'm wrong.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #8275

[stscijgbot-jp]

Comment by David Law on JIRA:

Rachel Plesha Does this look like it's now working as you'd expect?

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

David Law we're making our way through 10.2 build items first, but this is on our list to look over for 11.0

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

To test this, I use the F277W dataset 'jw01935001001_04102_00001-seg001_nis_uncal.fits' and ran it through the DETECTOR1 and SPEC2 pipelines. DET1 ran as expected and SPEC2 errored gracefully as designed on the extract_1d step with the output:

2024-07-24 08:53:00,263 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d running with args (<ImageModel(96, 2048) from jw01935001001_04102_00001-seg001_nis_cal.fits>,).

2024-07-24 08:53:00,268 - stpipe.Spec2Pipeline.extract_1d - INFO - Input is a NIRISS SOSS observation, the specialized SOSS extraction (ATOCA) will be used.

2024-07-24 08:53:00,268 - stpipe.Spec2Pipeline.extract_1d - ERROR - {*}The SOSS extraction is implemented for the CLEAR filter only.Requested filter is F277W{*}.

2024-07-24 08:53:00,268 - stpipe.Spec2Pipeline.extract_1d - ERROR - extract_1d will be skipped.

2024-07-24 08:53:00,269 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d done

2024-07-24 08:53:00,269 - stpipe.Spec2Pipeline - WARNING - Extract_1d did not return a DataModel - skipping photom.

2024-07-24 08:53:00,269 - stpipe.Spec2Pipeline - INFO - Finished processing product jw01935001001_04102_00001-seg001_nis_uncal

2024-07-24 08:53:00,269 - stpipe.Spec2Pipeline - INFO - Ending calwebb_spec2

2024-07-24 08:53:00,269 - stpipe.Spec2Pipeline - INFO - Results used CRDS context: jwst_1241.pmap

2024-07-24 08:53:00,269 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline done

Test is passing and the changes look good. Thanks!