NRS MOS Spec2Pipeline crash for internal lamp data

[stscijgbot-jp]

Issue JP-3487 was created on JIRA by Nikolay Nikolov:

S-flat data of the internal lamp of NIRSpec in MOS mode is not processed by the pipeline beyond Detector one. This can be seen on the MAST archive, as well as if one runs the pipeline locally. The reason for the pipeline not to run is identified to be missing contents of the MSA metadata files. One example includes program 1122, observation 2 with MSA file jw01122002001_03_msa.fits. Running Spec2Pipeline using rate images (available on the MAST) produces an error on the assign_wcs step: No open slits fall on detector NRS1. This is not unexpected due to the missing contents of the MSA file, particularly the SHUTTER_INFO and SOURCE_INFO, which are not clear how should be defined for internal lamp exposures.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Nikolay Nikolov Program 1122, observation 2 contains many different types of exposures, using different modes (MSASPEC vs. IFU) and lamps (FLATn, LINEn, REF, ...). Can you indicate one or more specific exposures that have this problem? I assume they're MSASPEC mode with any of the FLATn lamps?

[stscijgbot-jp]

Comment by Nikolay Nikolov on JIRA:

Howard Bushouse, sorry for not detailing! Yes, you are correct, we are looking for calibration of any of the FLATn lamp exposures in MSASPEC mode. Here are a few:

jw01122002001_02109_00001_nrs1 10 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:12:56.815 OPAQUE G395H 3 jw01122002001_02109_00001_nrs2 10 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:12:56.815 OPAQUE G395H 3 jw01122002001_0210a_00001_nrs1 11 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:20:58.287 OPAQUE G395H 4 jw01122002001_0210a_00001_nrs2 11 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:20:58.287 OPAQUE G395H 4 jw01122002001_0210d_00001_nrs1 14 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:37:44.942 OPAQUE G395H 5 jw01122002001_0210d_00001_nrs2 14 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:37:44.942 OPAQUE G395H 5 jw01122002001_0210e_00001_nrs1 15 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:46:01.005 OPAQUE G395H 6 jw01122002001_0210e_00001_nrs2 15 14 NRS_LAMP MSASPEC FLAT3 2022-05-06 18:46:01.005 OPAQUE G395H 6

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

I've confirmed that when processing jw01122002001_02109_00001_nrs1_rate.fits in calwebb_spec2, using MSA meta data file "jw01122002001_03_msa.fits", it aborts at the assign_wcs step, because no slits are projected onto the NRS1 detector.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The MSA metadata file jw01122002001_03_msa.fits contains all the usual SHUTTER_IMAGE, SHUTTER_INFO, and SOURCE_INFO extensions, but the last two (shutter and source tables) have zero rows that are populated. Hence the lack of slitlet/shutter info for use in assign_wcs. The rate file meta data have "MSASTATE='PRIMARYPARK_CONFIG'", which suggests a default state for the MSA. No idea what shutters, if any, are open for the exposure.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The rate image looks like this, so obviously some MSA shutters were open somewhere, in addition to getting light through the (always open) fixed slits. !Screen Shot 2023-12-14 at 2.32.39 PM.png!

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

So if MSA shutters were in fact open, why is DMS not getting any info for them from the upstream subsystems?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The original specs provided by the NIRSpec team for processing internal lamp exposures (see JP-473), which were implemented via JP-1194, clearly specify that exposures taken with any of the FLATn lamps in MSASPEC mode should be capable of being processed through assign_wcs, msa_flagging, and extract_2d. But to do that we also need MSA meta data, so the case of not having MSA meta data was not envisioned.

[stscijgbot-jp]

Comment by Diane Karakla on JIRA:

Howard Bushouse I do see metadata written out in the sql file for Obs 2.  This config was used commonly in other obs of this program.  There are no sources because a catalog is not used for the lamp template.  The slitlets in this case are a series of adjacent open and closed shutters which are located in different columns of the MSA config.

 

APT is writing some info for the config into the sql file:

insert into nirspec_msa_config ( msa_config_id, observation, program, visit, shutter_column, shutter_quadrant, shutter_row ) values  ( 1, 2, 1122, 1, 122, 2, 9 ),  ( 1, 2, 1122, 1, 124, 2, 9 ),  ( 1, 2, 1122, 1, 126, 2, 9 ),  ( 1, 2, 1122, 1, 128, 2, 9 ),  ( 1, 2, 1122, 1, 12, 1, 10 ),  ( 1, 2, 1122, 1, 14, 1, 10 ),  ( 1, 2, 1122, 1, 16, 1, 10 ),  ( 1, 2, 1122, 1, 18, 1, 10 ),  ( 1, 2, 1122, 1, 132, 1, 10 ),  ( 1, 2, 1122, 1, 134, 1, 10 ),  ( 1, 2, 1122, 1, 136, 1, 10 ),  ( 1, 2, 1122, 1, 138, 1, 10 ),  ( 1, 2, 1122, 1, 8, 1, 11 ),  ( 1, 2, 1122, 1, 10, 1, 11 ),  ( 1, 2, 1122, 1, 72, 1, 11 ),  ( 1, 2, 1122, 1, 74, 1, 11 ),  ( 1, 2, 1122, 1, 76, 1, 11 ),

...