LRS Slit mode Spec2Pipeline takes the wrong flatfield

[stscijgbot-jp]

Issue JP-2930 was created on JIRA by Juergen Schreiber:

The flatfield step in Spec2Pipeline takes the wrong flatfield CRDS called jwst_miri_flat_0790.fits which is an imager flatfield for filter 1280. That has the LRS slit region close to zero. This leads to zeroing everything in the resample_spec step. CRDS version used: jwst_0965.pmap

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

When I look at the list of MIRI Flat ref files currently in CRDS-Ops I see this for jwst_miri_flat_0790.fits:

!Screen Shot 2022-09-29 at 7.51.20 AM.png!

and the header of that FITS file says:

FILTER  = 'P750L   '           / Name of the MIRI filter                        
CHANNEL = 'N/A     '           / MRS Channel                                    
BAND    = 'N/A     '           / MRS Band                                       
FILENAME= 'MIRI_FM_MIRIMAGE_P750L_FLAT_09.01.00.fits' / File name               
VERSION = '07.01.01'           / Version number                                 
USEAFTER= '2022-04-01T00:00:00' / Date (+time) after which CDP should be used   
AUTHOR  = 'Oers Hunor Detre, detre@mpia.de' / Author(s) of the data             
ORIGIN  = 'MIRI European Consortium' / Organisation responsible for the data    
REFTYPE = 'FLAT    '           / Reference file type                            
DESCRIP = 'MIRI CDP-9 LRS pixel flats' / Description of the reference file      
PEDIGREE= 'INFLIGHT 2022-06-04 2022-06-04' / Pedigree of data. GROUND for pre-la
GWAXTILT= 'N/A     '           / Grating Y tilt angle relative to mirror        
GWAYTILT= 'N/A     '           / Grating X tilt angle relative to mirror        
EXP_TYPE= 'MIR_FLATIMAGE'      / Type of data in the exposure                   
ORIGFILE= 'MIRI_FM_MIRIMAGE_P750L_FLAT_06.01.00.fits' / Original name of the fil```
So all of the meta data indicate that it is a P750L flat, in full frame, which certainly suggests it is intended for LRS slit exposures.

If the actual data in the file are not for the LRS slit mode, that's a problem with the reference file and its delivery to CRDS, not the cal pipeline. The cal pipeline just uses what it's told to use by CRDS, based on the rmap selectors.

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Misty Cracraft Do you have any idea what's going on here (assuming you may have been involved in the delivery of the ref files)?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Looking at that flat in ds9 I also see that the entire frame is set to 1, except for the expected rectangular cutout region for the LRS slit, which has values ranging from ~0.97 to ~1.03 or so. So it certainly looks like an LRS slit flat.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

All of this info is based on what's in CRDS Ops. If I look in CRDS Pub, on the other hand, jwst_miri_flat_0790.fits is in fact a flat for the F1280W filter, as opposed to LRS slit. So I assume you must be looking at files in the CRDS-Pub server?

[stscijgbot-jp]

Comment by Juergen Schreiber on JIRA:

Yes, I used the CRDS-Pub Server. So, there seems to be some mish-mash in the different servers?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Yes, the same ref file can have different version numbers between CRDS-Ops and  CRDS-Pub, just due to differences in when files get delivered to each server. It's a pain and at times confusing (like now), but that's just the way it works (at least for now).

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

When using CRDS-Pub it appears that the appropriate flat for LRS slit is 785: https://jwst-crds-pub.stsci.edu/browse/jwst_miri_flat_0785.fits

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Thanks all for diagnosing this. Having 2 files with the same name on each server that are different in their content or how they are applied seems like a really dangerous failure mode! 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

So can we close this ticket as having been resolved?

[stscijgbot-jp]

Comment by Juergen Schreiber on JIRA:

I was expecting that the pipeline is also working with CRDS-pub. Also, if you run an LRS pipeline with LRS input files it should not be possible to take an Imager CRDS file as CRDS input file. There should be some mechanism in the steps to prevent this. So, I would not close this ticket though.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The cal pipeline uses whichever CRDS server the user has set in their environment. If nothing has been set, then the default is CRDS-Ops, because that's usually where the latest/greatest ref files get delivered. It will only use crds-pub if you explicitly tell it to (or have a local crds cache that is populated from crds-pub).

To a certain extent the pipeline steps intentionally don't do too much checking of the reference files they've been given in order to allow maximum flexibility for users to override the defaults with whatever they want. For example, if it's been determined that the flat for one filter also works pretty well for another (or someone wants to at least experiment to find out), then we don't want the code to block them from doing that. The most that's done is to make sure that basics like the dimensions of the ref file match the science data.  But this is a philosophical type question that could be discussed by either the Cal or DMS working groups.

[stscijgbot-jp]

Comment by Juergen Schreiber on JIRA:

Yes, I am fine with overwriting the CRDS files, but the pipeline should not automatically take an imager CRDS file while it is in the LRS pipeline mode, isn't it?

[nden]

JP ticket was resolved

[stscijgbot-jp]

Comment by Juergen Schreiber on JIRA:

I get the following problem, (I guess the same) when running with the operational CRDS server: It seems to look for an MRS reference instead of LRS.

2022-11-08 13:20:02,882 - stpipe.Spec2Pipeline - INFO - Prefetching reference files for dataset: 'jw01033001001_03103_00001_mirimage_uncal_nod1_rate.fits' reftypes = ['apcorr', 'area', 'barshadow', 'camera', 'collimator', 'cubepar', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'fringe', 'ifufore', 'ifupost', 'ifuslicer', 'mrsxartcorr', 'msa', 'msaoper', 'ote', 'pathloss', 'photom', 'regions', 'sflat', 'speckernel', 'specprofile', 'spectrace', 'specwcs', 'wavecorr', 'wavelengthrange', 'wavemap', 'wfssbkg']

2022-11-08 13:20:02,888 - CRDS - ERROR - Error determining best reference for 'mrsxartcorr' = parameter='META.EXPOSURE.TYPE [EXP_TYPE]' value='MIR_LRS-FIXEDSLIT' is not in ['MIR_MRS']

CrdsLookupError Traceback (most recent call last) Cell In [21], line 1 ----> 1 extract_spec_nod1 = Spec2Pipeline.call('extract1D_ref1.json', save_bsub=True, save_results=True, output_dir = output_dir, steps = steps1) 2 extract_spec_nod2 = Spec2Pipeline.call('extract1D_ref2.json', save_bsub=True, save_results=True, output_dir = output_dir, steps = steps2) File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/step.py:662, in Step.call(cls, *args, *kwargs) 658 name = config.get('name', None) 659 instance = cls.from_config_section(config, 660 name=name, config_file=config_file, param_args=kwargs) --> 662 return instance.run(args) File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/step.py:481, in Step.run(self, args) 479 else: 480 if self.prefetch_references: --> 481 self.prefetch(args) 482 try: 483 step_result = self.process(args) File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/step.py:598, in Step.prefetch(self, args) 596 # prefetch truly occurs at the Pipeline (or subclass) level. 597 if len(args) and len(self.reference_file_types) and not self.skip: --> 598 self._precache_references(args[0]) File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py:313, in Pipeline._precache_references(self, input_file) 310 try: 311 with self.open_model(input_file, asn_n_members=1, 312 asn_exptypes=["science"]) as model: --> 313 self._precache_references_opened(model) 314 except (ValueError, TypeError, IOError): 315 self.log.info( 316 'First argument {0} does not appear to be a ' 317 'model'.format(input_file)) File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py:333, in Pipeline._precache_references_opened(self, model_or_container) 330 if isinstance(model_or_container, Sequence): 331 # recurse on each contained model 332 for contained_model in model_or_container: --> 333 self._precache_references_opened(contained_model) 334 else: 335 # precache a single model object 336 self._precache_references_impl(model_or_container) File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py:336, in Pipeline._precache_references_opened(self, model_or_container) 333 self._precache_references_opened(contained_model) 334 else: 335 # precache a single model object --> 336 self._precache_references_impl(model_or_container) File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py:360, in Pipeline._precache_references_impl(self, model) 356 fetch_types = sorted(set(self.reference_file_types) - set(ovr_refs.keys())) 358 self.log.info("Prefetching reference files for dataset: " + repr(model.meta.filename) + 359 " reftypes = " + repr(fetch_types)) --> 360 crds_refs = crds_client.get_multiple_reference_paths(model.get_crds_parameters(), fetch_types, model.crds_observatory) 362 ref_path_map = dict(list(crds_refs.items()) + list(ovr_refs.items())) 364 for (reftype, refpath) in sorted(ref_path_map.items()): File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/crds_client.py:55, in get_multiple_reference_paths(parameters, reference_file_types, observatory) 52 raise TypeError("First argument must be a dict of parameters") 54 log.set_log_time(True) ---> 55 refpaths = _get_refpaths(parameters, tuple(reference_file_types), observatory) 56 return refpaths File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/stpipe/crds_client.py:68, in _get_refpaths(data_dict, reference_file_types, observatory) 66 return {} 67 with crds_cache_locking.get_cache_lock(): ---> 68 bestrefs = crds.getreferences( 69 data_dict, reftypes=reference_file_types, observatory=observatory) 70 refpaths = {filetype: filepath if "N/A" not in filepath.upper() else "N/A" 71 for (filetype, filepath) in bestrefs.items()} 72 return refpaths File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/crds/core/heavy_client.py:127, in getreferences(parameters, reftypes, context, ignore_cache, observatory, fast) 122 final_context, bestrefs = _initial_recommendations("getreferences", 123 parameters, reftypes, context, ignore_cache, observatory, fast) 125 # Attempt to cache the recommended references, which unlike dump_mappings 126 # should work without network access if files are already cached. --> 127 best_refs_paths = api.cache_references( 128 final_context, bestrefs, ignore_cache=ignore_cache) 130 return best_refs_paths File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/crds/client/api.py:848, in cache_references(pipeline_context, bestrefs, ignore_cache) 840 def cache_references(pipeline_context, bestrefs, ignore_cache=False): 841 """Given a CRDS context pipeline_context and bestrefs dictionary, download missing 842 reference files and cache them on the local file system. 843 (...) 846 Returns: { reference_keyword : reference_local_filepath ... } 847 """ --> 848 wanted = _get_cache_filelist_and_report_errors(bestrefs) 850 if config.S3_RETURN_URI: 851 localrefs = {name: get_flex_uri(name) for name in wanted} File ~/anaconda3/envs/jwst/lib/python3.9/site-packages/crds/client/api.py:891, in _get_cache_filelist_and_report_errors(bestrefs) 889 log.error(str(last_error)) 890 if last_error is not None: --> 891 raise last_error 892 return wanted CrdsLookupError: Error determining best reference for 'mrsxartcorr' = parameter='META.EXPOSURE.TYPE' value='MIR_LRS-FIXEDSLIT' is not in ['MIR_MRS']

 

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Juergen Schreiber  The latest CRDS rmap for the mrsxartcorr ref file has a couple of entries for EXP_TYPE='MIR_MRS' and a "N/A" entry for all other EXP_TYPEs. So when processing an MIR_LRS-FIXEDSLIT exposure it should default to the N/A entry without trouble. Could this be an issue with your CRDS cache, in that it might have an older mrsxartcorr rmap in it, like jwst_miri_mrsxartcorr_0003.rmap, which only has an entry for MIR_MRS and no others?

[stscijgbot-jp]

Comment by Juergen Schreiber on JIRA:

That is the newest in my cache: jwst_miri_mrsxartcorr_0004.rmap

I used jwst version 1.8.2 with jwst_1013.pmap

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I think we're again mixing contexts, for one. In OPS, we're only up to jwst_miri_mrxxartcorr_0002.rmap, while PUB is up to 0004. However, the OPS pmap is jwst_1013.pmap, while PUB is 0991.pmap. Juergen, I think we're recommending people only use OPS now. If you cleared out your CRDS cache and just re-downloaded files from OPS, that should clear out the mis-matched files and mappings. However, since both of those cross-artifact reference file mappings only have valid options for MRS and N/A for anything else, I'm still not sure what's going on there.

[stscijgbot-jp]

Comment by Juergen Schreiber on JIRA:

OK, thanks Misty.

I will delete the whole CRDS cache and try again.

[stscijgbot-jp]

Comment by Juergen Schreiber on JIRA:

OK, deleting the CRDS cache, using the ops CRDS server, and activating the jwst pipeline from scratch seemed to solve the problem.

Well, but the reason for such glitches I do not understand, I must confess.

Should I close the ticket or do you want to still work on it?

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

Issue was confusion between CRDS OPS and PUB servers, which has been sorted out. No further work needed.