Update apcorr datamodels to allow CHANNEL keyword for MIRI

[stscijgbot-jp]

Issue JP-3710 was created on JIRA by Misty Cracraft:

In 2023, the apcorr selection criteria for the MRS apcorr file was updated to add CHANNEL, as new files were going in that would be channel dependent. The MRS apcorr datamodel allows CHANNEL as a keyword, but the LRS and imager datamodels do not. Since CRDS does not distinguish MRS, LRS and Imager files, just lumps all apcorr files together, when the files for LRS or imager are tested with crds certify, there is an error that the CHANNEL required keyword is missing.

The request is that CHANNEL be added to LRS and Imager mode apcorr datamodels, so that those files can have the keyword listed with N/A (since neither actually use CHANNEL), so that the CRDS system will not complain.

And if we could get a little clarification from CRDS, it would be helpful. Do I have the correct understanding, that if any MIRI apcorr file uses a header keyword as a selection criteria, all files of that type must also contain that header keyword? Rosa Diaz  Meaghan McDonald 

I am linking the CRDS ticket where the update was made to the apcorr selection criteria.

 

[stscijgbot-jp]

Comment by David Law on JIRA:

Misty Cracraft Just to confirm, does this issue cause a pipeline crash when trying to process LRS or imager data, or is it just an error in CRDS certification?

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

We've been adding the keyword to the headers before delivery to CRDS. But I don't think CRDS can make the proper selection without the header keywords. But that's the part I wanted clarification for from either Rossy or Meaghan. We haven't delivered the files without the header (since certify says it's a required keyword), so there has been no chance to see if there would be a crash.

 

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Warning: Long-winded explanation follows.

Dealing with the two sides of this question:

For APCORR reference files, as discussed, the META.INSTRUMENT.CHANNEL keyword should be added to the APCORR datamodel. The APCORR datamodel is a definition of the APCORR reftype and if a keyword is in the parkeys selection, it should be in the reference file datamodel.

Now, what a JWST science datamodel, or any science file for that matter, needs to have in it's header to successfully match an APCORR reftype is up to how the rules are defined. One will find that keywords such as META.INSTRUMENT.NAME are required because none of the mappings allow a wildcard for that keyword.

However, for this particular case of META.INSTRUMENT.CHANNEL, the rmap allows the value "N/A", meaning that to match a particular reference, the value of that keyword is "not applicable". Hence, if the science file does not have that keyword defined, it does not matter, and will match the "N/A" reference, given that all other keywords also match.

As a test, one can do the following:

$ python

import stpipe.crds_client as cc header = dict() header['meta.instrument.name'] = 'MIRI' header['meta.exposure.type'] = 'MIR_IMAGE' header['meta.observation.date'] = '2024-01-01' header['meta.observation.time'] = '09:05:03' cc.get_reference_file(header, 'apcorr', 'jwst') '/blah/jwst_miri_apcorr_0008.asdf'

Note that META.INSTRUMENT.CHANNEL is not defined in the above header and the appropriate reference is returned, where CHANNEL == 'N/A'. If one does define META.INSTRUMENT.CHANNEL in the header, another reference will be returned:

header['meta.instrument.channel'] = '1' cc.get_reference_file(header, 'apcorr', 'jwst') '/blah/jwst_miri_apcorr_0007.asdf'

As a check on a "required" keyword, such as META.INSTRUMENT.NAME, try the following:

del header['meta.instrument.name'] cc.get_reference_file(header, 'apcorr', 'jwst') ....BLAH BLAH BLAH... CrdsError: Missing 'META.INSTRUMENT.NAME' keyword in header for determining instrument.

This is because the mapping, in this case the pmap, does not specify a wildcard for META.INSTRUMENT.NAME

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Just as an example of the messages we see:

CRDS - INFO -  Certifying './jwst_miri_apcorr_flight_31jul24.fits' (1/2) as 'FITS' relative to context 'jwst_1263.pmap' CRDS - INFO -  FITS file 'jwst_miri_apcorr_flight_31jul24.fits' conforms to FITS standards. CRDS - INFO -  Setting 'META.EXPOSURE.TYPE [EXP_TYPE]' = 'MIR_IMAGE' to value of 'P_EXP_TY' = 'MIR_IMAGE|MIR_4QPM|MIR_LYOT|MIR_TACQ|MIR_TACONFIRM|' CRDS - ERROR -  instrument='MIRI' type='APCORR' data='./jwst_miri_apcorr_flight_31jul24.fits' ::  Checking 'META.INSTRUMENT.CHANNEL [CHANNEL]' : Missing required keyword 'META.INSTRUMENT.CHANNEL [CHANNEL]'

 

With an error in crds certify, we won't actually be able to deliver the file to the system, so CRDS is telling us that we have to have the header keyword in order to deliver the file.