search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

CrdsLookupError for F322W2 filter #5977

Closed stscijgbot-jp closed 3 years ago

stscijgbot-jp commented 3 years ago

Issue JP-2053 was created on JIRA by Everett Schlawin:

## I get the following error with the F322W2 filter

from jwst.pipeline.calwebb_spec2 import Spec2Pipeline
import jwst
rateints_files = ['jw00042001001_01101_00002-seg021_nrca5_rateints.fits']

pipeline2 = Spec2Pipeline()
pipeline2.save_results = True
pipeline2.output_dir = '.'
pipeline2.run(rateints_files)

2021-04-21 16:00:11,485 - stpipe.Spec2Pipeline - INFO - Spec2Pipeline instance created.
2021-04-21 16:00:11,486 - stpipe.Spec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2021-04-21 16:00:11,487 - stpipe.Spec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2021-04-21 16:00:11,488 - stpipe.Spec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.
2021-04-21 16:00:11,489 - stpipe.Spec2Pipeline.msa_flagging - INFO - MSAFlagOpenStep instance created.
2021-04-21 16:00:11,491 - stpipe.Spec2Pipeline.extract_2d - INFO - Extract2dStep instance created.
2021-04-21 16:00:11,495 - stpipe.Spec2Pipeline.master_background - INFO - MasterBackgroundNrsSlitsStep instance created.
2021-04-21 16:00:11,496 - stpipe.Spec2Pipeline.master_background.flat_field - INFO - FlatFieldStep instance created.
2021-04-21 16:00:11,497 - stpipe.Spec2Pipeline.master_background.pathloss - INFO - PathLossStep instance created.
2021-04-21 16:00:11,498 - stpipe.Spec2Pipeline.master_background.barshadow - INFO - BarShadowStep instance created.
2021-04-21 16:00:11,500 - stpipe.Spec2Pipeline.master_background.photom - INFO - PhotomStep instance created.
2021-04-21 16:00:11,502 - stpipe.Spec2Pipeline.wavecorr - INFO - WavecorrStep instance created.
2021-04-21 16:00:11,503 - stpipe.Spec2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2021-04-21 16:00:11,504 - stpipe.Spec2Pipeline.srctype - INFO - SourceTypeStep instance created.
2021-04-21 16:00:11,506 - stpipe.Spec2Pipeline.straylight - INFO - StraylightStep instance created.
2021-04-21 16:00:11,507 - stpipe.Spec2Pipeline.fringe - INFO - FringeStep instance created.
2021-04-21 16:00:11,509 - stpipe.Spec2Pipeline.pathloss - INFO - PathLossStep instance created.
2021-04-21 16:00:11,510 - stpipe.Spec2Pipeline.barshadow - INFO - BarShadowStep instance created.
2021-04-21 16:00:11,512 - stpipe.Spec2Pipeline.photom - INFO - PhotomStep instance created.
2021-04-21 16:00:11,513 - stpipe.Spec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2021-04-21 16:00:11,515 - stpipe.Spec2Pipeline.cube_build - INFO - CubeBuildStep instance created.
2021-04-21 16:00:11,517 - stpipe.Spec2Pipeline.extract_1d - INFO - Extract1dStep instance created.
2021-04-21 16:00:11,557 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline running with args (['jw00042001001_01101_00002-seg021_nrca5_rateints.fits'],).
2021-04-21 16:00:11,579 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': '.', 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'fail_on_exception': True, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'imprint_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'msa_flagging': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'extract_2d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'slit_name': None, 'extract_orders': None, 'tsgrism_extract_height': None, 'wfss_extract_half_height': 5, 'grism_objects': None, 'mmag_extract': 99.0}, 'master_background': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'force_subtract': False, 'save_background': False, 'user_background': None, 'inverse': False, 'steps': {'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}}}, 'wavecorr': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'srctype': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'straylight': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'method': 'ModShepard', 'roi': 50, 'power': 1.0}, 'fringe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample_spec': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'pixel_scale_ratio': 1.0, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'cube_build': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'channel': 'all', 'band': 'all', 'grating': 'all', 'filter': 'all', 'output_type': 'band', 'scale1': 0.0, 'scale2': 0.0, 'scalew': 0.0, 'weighting': 'emsm', 'coord_system': 'skyalign', 'rois': 0.0, 'roiw': 0.0, 'weight_power': 2.0, 'wavemin': None, 'wavemax': None, 'single': False, 'xdebug': None, 'ydebug': None, 'zdebug': None, 'skip_dqflagging': False}, 'extract_1d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'smoothing_length': None, 'bkg_fit': 'poly', 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'apply_apcorr': True}}}
2021-04-21 16:00:12,046 - stpipe.Spec2Pipeline - INFO - Prefetching reference files for dataset: 'jw00042001001_01101_00002-seg021_nrca5_rateints.fits' reftypes = ['apcorr', 'area', 'barshadow', 'camera', 'collimator', 'cubepar', 'dflat', 'disperser', 'distortion', 'drizpars', 'extract1d', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'fringe', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'msaoper', 'ote', 'pathloss', 'photom', 'regions', 'resol', 'sflat', 'specwcs', 'wavecorr', 'wavelengthrange', 'wfssbkg']
2021-04-21 16:00:14,389 - CRDS - ERROR -  Error determining best reference for 'extract1d'  =    parameter='META.INSTRUMENT.FILTER [FILTER]' value='F322W2' is not in ['F444W']
Traceback (most recent call last):
  File "/Users/everettschlawin/Documents/jwst/pipeline/pipeline_bugs/crds_lookup_error/run_stage2.py", line 22, in <module>
    pipeline2.run(rateints_files)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/step.py", line 405, in run
    self.prefetch(*args)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/step.py", line 521, in prefetch
    self._precache_references(args[0])
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py", line 235, in _precache_references
    self._precache_references_opened(model)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py", line 255, in _precache_references_opened
    self._precache_references_opened(contained_model)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py", line 258, in _precache_references_opened
    self._precache_references_impl(model_or_container)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/pipeline.py", line 282, in _precache_references_impl
    crds_refs = crds_client.get_multiple_reference_paths(model.get_crds_parameters(), fetch_types, model.crds_observatory)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/crds_client.py", line 55, in get_multiple_reference_paths
    refpaths = _get_refpaths(parameters, tuple(reference_file_types), observatory)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/stpipe/crds_client.py", line 68, in _get_refpaths
    bestrefs = crds.getreferences(
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/crds/core/heavy_client.py", line 127, in getreferences
    best_refs_paths = api.cache_references(
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/crds/client/api.py", line 848, in cache_references
    wanted = _get_cache_filelist_and_report_errors(bestrefs)
  File "/Users/everettschlawin/anaconda/envs/jwst/lib/python3.9/site-packages/crds/client/api.py", line 891, in _get_cache_filelist_and_report_errors
    raise last_error
crds.core.exceptions.CrdsLookupError: Error determining best reference for 'extract1d'  =    parameter='META.INSTRUMENT.FILTER' value='F322W2' is not in ['F444W']
eas342 commented 3 years ago

The example rateints file can be downloaded at: https://stsci.box.com/s/fwksd2ijlbfqka7mhcbtprh3j5vf2mp5

eas342 commented 3 years ago

I just realized that file might get updated for a different reason, so please use this link instead for testing: https://arizona.box.com/s/ad4eecex9vc94tbmcsiupheg9m16lbw1

stscijgbot-jp commented 3 years ago

Comment by Howard Bushouse on JIRA:

This is due to the problem with the latest NIRCam extract1d rmap reported in https://jira.stsci.edu/browse/CRDS-435.  An update was delivered just yesterday and should be fixed in CRDS_CONTEXT 718  https://jwst-crds.stsci.edu/context_table/jwst_0718.pmap which adds an "N/A" entry to the rmap for modes and filters not covered by the single TSO entry.

stscijgbot-jp commented 3 years ago

Comment by Everett Schlawin on JIRA:

Thanks for the quick reply Howard Bushouse! Indeed when I run this, the error goes away!

export CRDS_CONTEXT=jwst_0718.pmap

stscijgbot-jp commented 3 years ago

Comment by Howard Bushouse on JIRA:

(y)