search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
563 stars 167 forks source link

spec2pipeline saying assign_wcs not done when it is #951

Closed stscieisenhamer closed 7 years ago

stscieisenhamer commented 7 years ago

Running Spec2Pipeline on the MIR_MRS dataset provided by K.Murray in

/grp/jwst/ssb/test_build7.1/development/miri/groundtestdata/mrs

errors with the log below. The last message is

jwst.cube_build.file_table.ErrorNoAssignWCS: ('Assign WCS has not been run on file %s', <jwst.datamodels.image.ImageModel object at 0x1215a4128>)

SloperPipeline ran to completion to create the 2a files. And, as is seen in the log below, Spec2Pipeline claims to successfully run the assign_wcs step.

Could be an incomplete dataset, but putting up in case something else is happening.

Log:

strun ../configs/calwebb_spec2.cfg det_image_seq1_MIRIFUSHORT_12LONGexp1_rate.fits
2017-06-05 14:13:24,762 - stpipe.Spec2Pipeline - INFO - Spec2Pipeline instance created.
2017-06-05 14:13:24,763 - stpipe.Spec2Pipeline.pathloss - INFO - PathLossStep instance created.
2017-06-05 14:13:24,764 - stpipe.Spec2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2017-06-05 14:13:24,764 - stpipe.Spec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2017-06-05 14:13:24,765 - stpipe.Spec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2017-06-05 14:13:24,767 - stpipe.Spec2Pipeline.cube_build - INFO - CubeBuildStep instance created.
2017-06-05 14:13:24,767 - stpipe.Spec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.
2017-06-05 14:13:24,768 - stpipe.Spec2Pipeline.photom - INFO - PhotomStep instance created.
2017-06-05 14:13:24,768 - stpipe.Spec2Pipeline.extract_1d - INFO - Extract1dStep instance created.
2017-06-05 14:13:24,769 - stpipe.Spec2Pipeline.srctype - INFO - SourceTypeStep instance created.
2017-06-05 14:13:24,770 - stpipe.Spec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2017-06-05 14:13:24,770 - stpipe.Spec2Pipeline.fringe - INFO - FringeStep instance created.
2017-06-05 14:13:24,771 - stpipe.Spec2Pipeline.extract_2d - INFO - Extract2dStep instance created.
2017-06-05 14:13:24,771 - stpipe.Spec2Pipeline.straylight - INFO - StraylightStep instance created.
2017-06-05 14:13:24,772 - stpipe - INFO - Hostname: inanna2.local.stsci.edu
2017-06-05 14:13:24,772 - stpipe - INFO - OS: Darwin
2017-06-05 14:13:26,150 - stpipe.Spec2Pipeline.pathloss - INFO - Prefetch for PATHLOSS reference file is 'N/A'.
2017-06-05 14:13:26,594 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for DFLAT reference file is 'N/A'.
2017-06-05 14:13:26,594 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for FFLAT reference file is 'N/A'.
2017-06-05 14:13:26,595 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for FLAT reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_flat_0246.fits'.
2017-06-05 14:13:26,595 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for SFLAT reference file is 'N/A'.
2017-06-05 14:13:27,252 - stpipe.Spec2Pipeline.resample_spec - INFO - Prefetch for DRIZPARS reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_drizpars_0001.fits'.
2017-06-05 14:13:27,711 - stpipe.Spec2Pipeline.cube_build - INFO - Prefetch for CUBEPAR reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_cubepar_0001.fits'.
2017-06-05 14:13:27,711 - stpipe.Spec2Pipeline.cube_build - INFO - Prefetch for RESOL reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_resol_0003.fits'.
2017-06-05 14:13:28,368 - stpipe.Spec2Pipeline.photom - INFO - Prefetch for AREA reference file is 'N/A'.
2017-06-05 14:13:28,368 - stpipe.Spec2Pipeline.photom - INFO - Prefetch for PHOTOM reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_photom_0018.fits'.
2017-06-05 14:13:28,824 - stpipe.Spec2Pipeline.extract_1d - INFO - Prefetch for EXTRACT1D reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_extract1d_0003.json'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for CAMERA reference file is 'N/A'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISTORTION reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_distortion_0025.asdf'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FORE reference file is 'N/A'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FPA reference file is 'N/A'.
2017-06-05 14:13:29,465 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for MSA reference file is 'N/A'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for OTE reference file is 'N/A'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for REGIONS reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_regions_0018.asdf'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for SPECWCS reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_specwcs_0014.asdf'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for V2V3 reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_v2v3_0007.asdf'.
2017-06-05 14:13:29,466 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for WAVELENGTHRANGE reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_wavelengthrange_0003.asdf'.
2017-06-05 14:13:29,932 - stpipe.Spec2Pipeline.fringe - INFO - Prefetch for FRINGE reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_fringe_0028.fits'.
2017-06-05 14:13:30,589 - stpipe.Spec2Pipeline.straylight - INFO - Prefetch for STRAYMASK reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_straymask_0004.fits'.
2017-06-05 14:13:30,661 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline running with args ('det_image_seq1_MIRIFUSHORT_12LONGexp1_rate.fits',).
2017-06-05 14:13:30,661 - stpipe.Spec2Pipeline - INFO - Starting calwebb_spec2 ...
2017-06-05 14:13:30,699 - stpipe.Spec2Pipeline - INFO - Processing product det_image_seq1_MIRIFUSHORT_12LONGexp1
2017-06-05 14:13:30,699 - stpipe.Spec2Pipeline - INFO - Working on input det_image_seq1_MIRIFUSHORT_12LONGexp1_rate.fits ...
2017-06-05 14:13:31,076 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for CAMERA reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISTORTION reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_distortion_0025.asdf'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FORE reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FPA reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for MSA reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for OTE reference file is 'N/A'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for REGIONS reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_regions_0018.asdf'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for SPECWCS reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_specwcs_0014.asdf'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for V2V3 reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_v2v3_0007.asdf'.
2017-06-05 14:13:31,077 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for WAVELENGTHRANGE reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_wavelengthrange_0003.asdf'.
2017-06-05 14:13:31,113 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<jwst.datamodels.image.ImageModel object at 0x11d804630>,).
2017-06-05 14:13:40,814 - stpipe.Spec2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2017-06-05 14:13:41,079 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs done
2017-06-05 14:13:43,361 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for DFLAT reference file is 'N/A'.
2017-06-05 14:13:43,361 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for FFLAT reference file is 'N/A'.
2017-06-05 14:13:43,361 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for FLAT reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_flat_0246.fits'.
2017-06-05 14:13:43,361 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for SFLAT reference file is 'N/A'.
2017-06-05 14:13:43,491 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field running with args (<jwst.datamodels.image.ImageModel object at 0x1133868d0>,).
2017-06-05 14:13:45,640 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2017-06-05 14:13:46,433 - stpipe.Spec2Pipeline.flat_field - WARNING - /Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/flatfield/flat_field.py:193: RuntimeWarning: divide by zero encountered in true_divide
  science.data /= flat_data

2017-06-05 14:13:46,568 - stpipe.Spec2Pipeline.flat_field - WARNING - /Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/flatfield/flat_field.py:194: RuntimeWarning: divide by zero encountered in true_divide
  science.err /= flat_data

2017-06-05 14:13:47,174 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field done
2017-06-05 14:13:47,546 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype running with args (<jwst.datamodels.image.ImageModel object at 0x11d808d68>,).
2017-06-05 14:13:48,334 - stpipe.Spec2Pipeline.srctype - INFO - Input EXP_TYPE is MIR_MRS
2017-06-05 14:13:48,334 - stpipe.Spec2Pipeline.srctype - INFO - Input SRCTYPE is unknown. Setting to default value of EXTENDED
2017-06-05 14:13:48,604 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype done
2017-06-05 14:13:50,672 - stpipe.Spec2Pipeline.straylight - INFO - Prefetch for STRAYMASK reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_straymask_0004.fits'.
2017-06-05 14:13:50,797 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight running with args (<jwst.datamodels.image.ImageModel object at 0x1187cb128>,).
2017-06-05 14:13:52,801 - stpipe.Spec2Pipeline.straylight - INFO - Using STRAYLIGHT reference file /Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_straymask_0004.fits
2017-06-05 14:14:04,461 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight done
2017-06-05 14:14:06,673 - stpipe.Spec2Pipeline.fringe - INFO - Prefetch for FRINGE reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_fringe_0028.fits'.
2017-06-05 14:14:06,812 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe running with args (<jwst.datamodels.image.ImageModel object at 0x11d7ecbe0>,).
2017-06-05 14:14:08,985 - stpipe.Spec2Pipeline.fringe - INFO - Using FRINGE reference file: /Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_fringe_0028.fits
2017-06-05 14:14:09,051 - stpipe.Spec2Pipeline.fringe - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2017-06-05 14:14:09,051 - stpipe.Spec2Pipeline.fringe - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2017-06-05 14:14:09,051 - stpipe.Spec2Pipeline.fringe - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2017-06-05 14:14:09,927 - stpipe.Spec2Pipeline.fringe - INFO - The DQ values in the reference file will NOT be used to update the    DQ values in the output DQ array.
2017-06-05 14:14:10,198 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe done
2017-06-05 14:14:12,360 - stpipe.Spec2Pipeline.photom - INFO - Prefetch for AREA reference file is 'N/A'.
2017-06-05 14:14:12,360 - stpipe.Spec2Pipeline.photom - INFO - Prefetch for PHOTOM reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_photom_0018.fits'.
2017-06-05 14:14:12,498 - stpipe.Spec2Pipeline.photom - INFO - Step photom running with args (<jwst.datamodels.image.ImageModel object at 0x1218ccb00>,).
2017-06-05 14:14:14,503 - stpipe.Spec2Pipeline.photom - INFO - Using photom reference file: /Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_photom_0018.fits
2017-06-05 14:14:15,746 - stpipe.Spec2Pipeline.photom - INFO - Using area reference file: N/A
2017-06-05 14:14:15,746 - stpipe.Spec2Pipeline.photom - INFO - Using instrument: MIRI
2017-06-05 14:14:15,746 - stpipe.Spec2Pipeline.photom - INFO -  detector: MIRIFUSHORT
2017-06-05 14:14:15,746 - stpipe.Spec2Pipeline.photom - INFO -  exp_type: MIR_MRS
2017-06-05 14:14:15,746 - stpipe.Spec2Pipeline.photom - INFO -  band: LONG
2017-06-05 14:14:15,830 - stpipe.Spec2Pipeline.photom - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2017-06-05 14:14:16,769 - stpipe.Spec2Pipeline.photom - INFO - Writing PHOTMJSR with value: 42.5
2017-06-05 14:14:17,081 - stpipe.Spec2Pipeline.photom - INFO - Step photom done
2017-06-05 14:14:19,272 - stpipe.Spec2Pipeline.cube_build - INFO - Prefetch for CUBEPAR reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_cubepar_0001.fits'.
2017-06-05 14:14:19,273 - stpipe.Spec2Pipeline.cube_build - INFO - Prefetch for RESOL reference file is '/Users/eisenham/Documents/ssbdev/testdata/crds-cache/references/jwst/miri/jwst_miri_resol_0003.fits'.
2017-06-05 14:14:19,396 - stpipe.Spec2Pipeline.cube_build - INFO - Step cube_build running with args (<jwst.datamodels.image.ImageModel object at 0x1215a4128>,).
2017-06-05 14:14:19,396 - stpipe.Spec2Pipeline.cube_build - INFO - Starting IFU Cube Building Step
2017-06-05 14:14:19,396 - stpipe.Spec2Pipeline.cube_build - INFO - Input interpolation: pointcloud
2017-06-05 14:14:19,396 - stpipe.Spec2Pipeline.cube_build - INFO - Coordinate system to use: ra-dec
2017-06-05 14:14:19,396 - stpipe.Spec2Pipeline.cube_build - INFO - Weighting method for point cloud: standard
2017-06-05 14:14:19,396 - stpipe.Spec2Pipeline.cube_build - INFO - Power Weighting distance : 2.000000
----------------------------------------------------------------------
ERROR RUNNING STEP 'Spec2Pipeline':
    ('Assign WCS has not been run on file %s',
    <jwst.datamodels.image.ImageModel object at 0x1215a4128>)
Traceback (most recent call last):
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 320, in step_from_cmdline
    step.run(*positional)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 356, in run
    result = self.process(*args)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 94, in process
    asn.filename
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 249, in process_exposure_product
    cube = self.cube_build(input)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 356, in run
    result = self.process(*args)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/cube_build/cube_build_step.py", line 202, in process
    self.output_file = cubeinfo.setup()
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/cube_build/cube_build.py", line 159, in setup
    self.dec_offset)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/cube_build/file_table.py", line 211, in set_file_table
    ifile)
jwst.cube_build.file_table.ErrorNoAssignWCS: ('Assign WCS has not been run on file %s', <jwst.datamodels.image.ImageModel object at 0x1215a4128>)

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/EGG-INFO/scripts/strun", line 14, in <module>
    step = Step.from_cmdline(sys.argv[1:])
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 142, in from_cmdline
    return cmdline.step_from_cmdline(args)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/stpipe/cmdline.py", line 320, in step_from_cmdline
    step.run(*positional)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 356, in run
    result = self.process(*args)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 94, in process
    asn.filename
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/pipeline/calwebb_spec2.py", line 249, in process_exposure_product
    cube = self.cube_build(input)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/stpipe/step.py", line 356, in run
    result = self.process(*args)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/cube_build/cube_build_step.py", line 202, in process
    self.output_file = cubeinfo.setup()
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/cube_build/cube_build.py", line 159, in setup
    self.dec_offset)
  File "/Users/eisenham/anaconda3/envs/spec3/lib/python3.5/site-packages/jwst-0.7.7.dev756-py3.5-macosx-10.7-x86_64.egg/jwst/cube_build/file_table.py", line 211, in set_file_table
    ifile)
jwst.cube_build.file_table.ErrorNoAssignWCS: ('Assign WCS has not been run on file %s', <jwst.datamodels.image.ImageModel object at 0x1215a4128>)
jemorrison commented 7 years ago

Ok I get a different error. I took Katie's det image data and ran calwebb_sloper.cfg and then took the rate file and ran calwebb_spec2.cfg This is the error I get (I have the most updated pipeline, astroconda and condo-dev)

dev3) arizonajane2:MIRI morrison$ strun calwebb_spec2.cfg det_image_seq1_MIRIFUSHORT_12LONGexp1_rate.fits

2017-06-06 00:03:21,779 - stpipe.Spec2Pipeline - INFO - Spec2Pipeline instance created.

2017-06-06 00:03:21,779 - stpipe.Spec2Pipeline.extract_1d - INFO - Extract1dStep instance created.

2017-06-06 00:03:21,780 - stpipe.Spec2Pipeline.photom - INFO - PhotomStep instance created.

2017-06-06 00:03:21,781 - stpipe.Spec2Pipeline.extract_2d - INFO - Extract2dStep instance created.

2017-06-06 00:03:21,782 - stpipe.Spec2Pipeline.cube_build - INFO - CubeBuildStep instance created.

2017-06-06 00:03:21,782 - stpipe.Spec2Pipeline.flat_field - INFO - FlatFieldStep instance created.

2017-06-06 00:03:21,783 - stpipe.Spec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.

2017-06-06 00:03:21,784 - stpipe.Spec2Pipeline.fringe - INFO - FringeStep instance created.

2017-06-06 00:03:21,784 - stpipe.Spec2Pipeline.pathloss - INFO - PathLossStep instance created.

2017-06-06 00:03:21,785 - stpipe.Spec2Pipeline.straylight - INFO - StraylightStep instance created.

2017-06-06 00:03:21,785 - stpipe.Spec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.

2017-06-06 00:03:21,786 - stpipe.Spec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.

2017-06-06 00:03:21,786 - stpipe.Spec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.

2017-06-06 00:03:21,787 - stpipe.Spec2Pipeline.srctype - INFO - SourceTypeStep instance created.

2017-06-06 00:03:21,787 - stpipe - INFO - Hostname: arizonajane2.local.stsci.edu

2017-06-06 00:03:21,787 - stpipe - INFO - OS: Darwin

2017-06-06 00:03:34,481 - stpipe.Spec2Pipeline.photom - INFO - Prefetch for AREA reference file is 'N/A'.

2017-06-06 00:03:34,481 - stpipe.Spec2Pipeline.photom - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0018.fits'.

2017-06-06 00:03:35,181 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for DFLAT reference file is 'N/A'.

2017-06-06 00:03:35,181 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for FFLAT reference file is 'N/A'.

2017-06-06 00:03:35,181 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0246.fits'.

2017-06-06 00:03:35,318 - stpipe.Spec2Pipeline.flat_field - INFO - Prefetch for SFLAT reference file is 'N/A'.

2017-06-06 00:03:35,827 - stpipe.Spec2Pipeline.fringe - INFO - Prefetch for FRINGE reference file is '/grp/crds/cache/references/jwst/jwst_miri_fringe_0028.fits'.

2017-06-06 00:03:36,402 - stpipe.Spec2Pipeline.pathloss - INFO - Prefetch for PATHLOSS reference file is 'N/A'.

2017-06-06 00:03:36,916 - stpipe.Spec2Pipeline.straylight - INFO - Prefetch for STRAYMASK reference file is '/grp/crds/cache/references/jwst/jwst_miri_straymask_0004.fits'.

2017-06-06 00:03:37,849 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for CAMERA reference file is 'N/A'.

2017-06-06 00:03:37,849 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.

2017-06-06 00:03:37,849 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISPERSER reference file is 'N/A'.

2017-06-06 00:03:37,849 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0014.asdf'.

2017-06-06 00:03:37,985 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'.

2017-06-06 00:03:37,985 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FORE reference file is 'N/A'.

2017-06-06 00:03:37,985 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FPA reference file is 'N/A'.

2017-06-06 00:03:37,985 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUFORE reference file is 'N/A'.

2017-06-06 00:03:37,985 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUPOST reference file is 'N/A'.

2017-06-06 00:03:37,985 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUSLICER reference file is 'N/A'.

2017-06-06 00:03:37,986 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for MSA reference file is 'N/A'.

2017-06-06 00:03:37,986 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for OTE reference file is 'N/A'.

2017-06-06 00:03:37,986 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for REGIONS reference file is '/grp/crds/cache/references/jwst/jwst_miri_regions_0007.asdf'.

2017-06-06 00:03:38,121 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for SPECWCS reference file is '/grp/crds/cache/references/jwst/jwst_miri_specwcs_0007.asdf'.

2017-06-06 00:03:38,258 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for V2V3 reference file is '/grp/crds/cache/references/jwst/jwst_miri_v2v3_0007.asdf'.

2017-06-06 00:03:38,394 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for WAVELENGTHRANGE reference file is '/grp/crds/cache/references/jwst/jwst_miri_wavelengthrange_0002.asdf'.

2017-06-06 00:03:38,785 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline running with args ('det_image_seq1_MIRIFUSHORT_12LONGexp1_rate.fits',).

2017-06-06 00:03:38,785 - stpipe.Spec2Pipeline - INFO - Starting calwebb_spec2 ...

2017-06-06 00:03:38,817 - stpipe.Spec2Pipeline - INFO - Processing product det_image_seq1_MIRIFUSHORT_12LONGexp1

2017-06-06 00:03:38,817 - stpipe.Spec2Pipeline - INFO - Working on input det_image_seq1_MIRIFUSHORT_12LONGexp1_rate.fits ...

2017-06-06 00:03:39,194 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for CAMERA reference file is 'N/A'.

2017-06-06 00:03:39,194 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.

2017-06-06 00:03:39,194 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISPERSER reference file is 'N/A'.

2017-06-06 00:03:39,194 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0014.asdf'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FORE reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for FPA reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUFORE reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUPOST reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for IFUSLICER reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for MSA reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for OTE reference file is 'N/A'.

2017-06-06 00:03:39,264 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for REGIONS reference file is '/grp/crds/cache/references/jwst/jwst_miri_regions_0007.asdf'.

2017-06-06 00:03:39,333 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for SPECWCS reference file is '/grp/crds/cache/references/jwst/jwst_miri_specwcs_0007.asdf'.

2017-06-06 00:03:39,402 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for V2V3 reference file is '/grp/crds/cache/references/jwst/jwst_miri_v2v3_0007.asdf'.

2017-06-06 00:03:39,473 - stpipe.Spec2Pipeline.assign_wcs - INFO - Prefetch for WAVELENGTHRANGE reference file is '/grp/crds/cache/references/jwst/jwst_miri_wavelengthrange_0002.asdf'.

2017-06-06 00:03:39,588 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<jwst.datamodels.image.ImageModel object at 0x11650e6a0>,).

ERROR RUNNING STEP 'Spec2Pipeline':

'1LONG'

Traceback (most recent call last):

File "/Users/morrison/git/jwst/jwst/stpipe/cmdline.py", line 320, in step_from_cmdline

step.run(*positional)

File "/Users/morrison/git/jwst/jwst/stpipe/step.py", line 355, in run

result = self.process(*args)

File "/Users/morrison/git/jwst/jwst/pipeline/calwebb_spec2.py", line 94, in process

asn.filename

File "/Users/morrison/git/jwst/jwst/pipeline/calwebb_spec2.py", line 153, in process_exposure_product

input = self.assign_wcs(input)

File "/Users/morrison/git/jwst/jwst/stpipe/step.py", line 355, in run

result = self.process(*args)

File "/Users/morrison/git/jwst/jwst/assign_wcs/assign_wcs_step.py", line 60, in process

result = load_wcs(input_model, reference_file_names)

File "/Users/morrison/git/jwst/jwst/assign_wcs/assign_wcs.py", line 30, in load_wcs

pipeline = mod.create_pipeline(input_model, reference_files)

File "/Users/morrison/git/jwst/jwst/assign_wcs/miri.py", line 38, in create_pipeline

pipeline = exp_type2transform[exp_type](input_model, reference_files)

File "/Users/morrison/git/jwst/jwst/assign_wcs/miri.py", line 262, in ifu

det2abl = (detector_to_abl(input_model, reference_files)).rename(

File "/Users/morrison/git/jwst/jwst/assign_wcs/miri.py", line 334, in detector_to_abl

mapper = jwmodels.MIRI_AB2Slice(bzero[cb], bdel[cb], c)

KeyError: '1LONG'

hbushouse commented 7 years ago

Yep, that's the error the regression tests have been showing recently too. I pinged @nden about it, but haven't had a response yet.

nden commented 7 years ago

I'll take a look today, had no internet yesterday.

nden commented 7 years ago

Are you sure you have the latest CRDS_CONTEXT? The MRS files were added in 0337. If I use 0337 I can't reproduce the error. With 0336 I see it.

hbushouse commented 7 years ago

Could someone log-on to jwcalibdev and make sure that the reg tests are set to use context 337? It's set in the file /data1/jwst_rt/pdk_environment. I think the last time I saw it, it was still set to 336. Didn't know 337 existed.

philhodge commented 7 years ago

It's set to 337, but the date on the file is 8:11 this morning. The pdk_environment~ file sets it to 336.

On 06/06/2017 09:47 AM, Howard Bushouse wrote:

Could someone log-on to jwcalibdev and make sure that the reg tests are set to use context 337? It's set in the file /data1/jwst_rt/pdk_environment. I think the last time I saw it, it was still set to 336. Didn't know 337 existed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/STScI-JWST/jwst/issues/951#issuecomment-306491090, or mute the thread https://github.com/notifications/unsubscribe-auth/AB-FDLFyQDLGOpsCHtLO8XMDVRmqHpHLks5sBVhUgaJpZM4NwvAG.

stscieisenhamer commented 7 years ago

I have mine set to to 337. If not, I get the cubepars missing error.

nden commented 7 years ago

I ran the above file successfully through assign_wcs and cube_build. Make sure you have the latest ref files and code (although the error indicates missing reference files).

nden commented 7 years ago

More specifically, the distortion reference file listed here is /grp/crds/cache/references/jwst/jwst_miri_distortion_0014.asdf . The one that I get is /grp/crds/cache/references/jwst/jwst_miri_distortion_0025.asdf.

hbushouse commented 7 years ago

Could it be CRDS selection issues based on DATE-OBS value in science data relative to USEAFTER in reference files?

hbushouse commented 7 years ago

Oh, never mind. In context 337 jwst_miri_distortion_0014.asdf is not even an active/available ref file anymore, and all the ref files have very early useafter dates (either 1900 or 2000), so it must be an issue with the context being used.

stscieisenhamer commented 7 years ago

So, created new conda-dev, made sure context is 337, and still get the unassigned wcs error. Note that in my original message, the distortion file is the _0025 one. Though, after clearing out my local cache, I'm now getting _0024. Note, I'm also using jwst-crds.stsci.edu as the server.

jdavies-st commented 7 years ago

If I don't have a CRDS_CONTEXT set, why doesn't our local environment just pick up the latest context available? Often when testing locally, I am not aware of reffile deliveries or context changes in parts of the jwst package which I haven't contributed to.

nden commented 7 years ago

@jdavies-st There's a "default" CRDS_CONTEXT which can be earlier than the latest for flexibility. This allows testing different features before making them default.

nden commented 7 years ago

@stscieisenhamer When you got 0024 for the distortion, did yuo use the same input data file? 0024 is for 34SHORT files, while the input is a 12LONG file.

nden commented 7 years ago

BTW the original error that assign_wcs was not run comes from cube_build. file_table. @jemorrison What are the input parameters in this case? Apparently cube_build.file_table thinks assign_wcs was not run on the file. However, I verified te file has assign_wcs set to "COMPLETE".

jdavies-st commented 7 years ago

Thanks @nden. I see. A little more complicated than I realized.

Perhaps we need Howard or a Context Master General™ to set the default for master when a new branch is merged in that requires it. Often the person doing the PR knows that the new context is required. Perhaps there's a milestone or label we could add which would indicate a new context is required?

nden commented 7 years ago

Todd and Howard are the CRDS Context tzars. I guess we are still testing new ref files.

hbushouse commented 7 years ago

You're right that if a new context doesn't contain any changes that will break old code, then we should just go ahead and update the default at the same time the new context is created and the new code is merged. The times that you don't want to update the default context are cases where some ref files have had their format or contents changed that are not compatible with folks, say in INS, running older (not the current conda-dev) versions of the pipeline. In situations where a new context contains new ref files that are needed by the conda-dev version of the pipeline, then we should update the default so that anyone running conda-dev will get those new necessary ref files.

stscieisenhamer commented 7 years ago

Believe I found the error; argument swap on a method call. PR shortly...