search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
568 stars 167 forks source link

Image2 fail with rateints exposure #2899

Closed stscijgbot closed 5 years ago

stscijgbot commented 5 years ago

Issue JP-457 was created by Alicia Canipe:

Hi,

I was trying to run the rateints output from Detector1 through the Image2 pipeline but it's failing. My data is here: /grp/jwst/wit/nircam/canipe/forBuild7.2Testing/

I tried running the single exposure and an association file, but I get the error below. I'm not sure if it's an issue with the file or how I'm trying to run it. Thanks for the help.

  {code:java} In [1]: from jwst.pipeline import calwebb_detector1, calwebb_image2, calwebb_coron3 In [2]: img2 = calwebb_image2.Image2Pipeline() 2018-11-27 15:42:08,530 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created. 2018-11-27 15:42:08,531 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created. 2018-11-27 15:42:08,532 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created. 2018-11-27 15:42:08,534 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created. 2018-11-27 15:42:08,535 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created. 2018-11-27 15:42:08,537 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created. In [3]: img2.run("jw42424001003_01103_00001_nrca3_uncal_rateints.fits") 2018-11-27 15:42:17,267 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args ('jw42424001003_01103_00001_nrca3_uncal_rateints.fits',). 2018-11-27 15:42:24,030 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'jw42424001003_01103_00001_nrca3_uncal_rateints.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg'] 2018-11-27 15:42:24,587 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_nircam_area_0007.fits'. 2018-11-27 15:42:24,592 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'. 2018-11-27 15:42:24,592 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'. 2018-11-27 15:42:24,592 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'. 2018-11-27 15:42:24,592 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'. 2018-11-27 15:42:24,592 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_nircam_distortion_0058.asdf'. 2018-11-27 15:42:24,596 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_nircam_drizpars_0001.fits'. 2018-11-27 15:42:24,601 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'. 2018-11-27 15:42:24,601 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'. 2018-11-27 15:42:24,601 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_nircam_flat_0003.fits'. 2018-11-27 15:42:24,605 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'. 2018-11-27 15:42:24,606 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'. 2018-11-27 15:42:24,606 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'. 2018-11-27 15:42:24,606 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'. 2018-11-27 15:42:24,606 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'. 2018-11-27 15:42:24,606 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'. 2018-11-27 15:42:24,606 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'. 2018-11-27 15:42:24,606 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_nircam_photom_0047.fits'. 2018-11-27 15:42:24,611 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'. 2018-11-27 15:42:24,611 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'. 2018-11-27 15:42:24,611 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'. 2018-11-27 15:42:24,611 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'. 2018-11-27 15:42:24,611 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'. 2018-11-27 15:42:24,612 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ... 2018-11-27 15:42:54,074 - stpipe.Image2Pipeline - INFO - Processing product jw42424001003_01103_00001_nrca3_uncal 2018-11-27 15:42:54,074 - stpipe.Image2Pipeline - INFO - Working on input jw42424001003_01103_00001_nrca3_uncal_rateints.fits ... 2018-11-27 15:42:54,393 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<CubeModel(4, 2048, 2048) from jw42424001003_01103_00001_nrca3_uncal_rateints.fits>,). 2018-11-27 15:42:54,974 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 204.299514551 -29.890961509 204.299435685 -29.873102056 204.279038828 -29.873064103 204.278826369 -29.891019964 2018-11-27 15:42:54,975 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 204.299514551 -29.890961509 204.299435685 -29.873102056 204.279038828 -29.873064103 204.278826369 -29.891019964 2018-11-27 15:42:54,975 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs 2018-11-27 15:42:54,983 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done 2018-11-27 15:42:55,082 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<CubeModel(4, 2048, 2048) from jw42424001003_01103_00001_nrca3_uncal_rateints.fits>,). 2018-11-27 15:42:56,151 - stpipe.Image2Pipeline.flat_field - WARNING - Missing subarray corner/size keywords in reference file 2018-11-27 15:42:56,151 - stpipe.Image2Pipeline.flat_field - WARNING - Setting them to full-frame default values 2018-11-27 15:42:57,580 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done 2018-11-27 15:42:57,638 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<CubeModel(4, 2048, 2048) from jw42424001003_01103_00001_nrca3_uncal_rateints.fits>,). 2018-11-27 15:42:57,668 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_nircam_photom_0047.fits 2018-11-27 15:42:57,672 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_nircam_area_0007.fits 2018-11-27 15:42:57,673 - stpipe.Image2Pipeline.photom - INFO - Using instrument: NIRCAM 2018-11-27 15:42:57,673 - stpipe.Image2Pipeline.photom - INFO - detector: NRCA3 2018-11-27 15:42:57,673 - stpipe.Image2Pipeline.photom - INFO - exp_type: NRC_IMAGE 2018-11-27 15:42:57,673 - stpipe.Image2Pipeline.photom - INFO - filter: F090W 2018-11-27 15:42:57,673 - stpipe.Image2Pipeline.photom - INFO - pupil: CLEAR 2018-11-27 15:42:57,864 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 1e-15 2018-11-27 15:42:58,074 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output. 2018-11-27 15:42:58,075 - stpipe.Image2Pipeline.photom - WARNING - The relative difference between the values for 2018-11-27 15:42:58,075 - stpipe.Image2Pipeline.photom - WARNING - the pixel area in sq arcsec (1.0) 2018-11-27 15:42:58,075 - stpipe.Image2Pipeline.photom - WARNING - exceeds the defined tolerance (0.001) 2018-11-27 15:42:58,079 - stpipe.Image2Pipeline.photom - INFO - Step photom done 2018-11-27 15:42:58,147 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<CubeModel(4, 2048, 2048) from jw42424001003_01103_00001_nrca3_uncal_rateints.fits>,).

RuntimeError Traceback (most recent call last)

in () ----> 1 img2.run("jw42424001003_01103_00001_nrca3_uncal_rateints.fits") ~/miniconda3/envs/jwstB72rc1/lib/python3.5/site-packages/jwst-0.12.0-py3.5-macosx-10.6-x86_64.egg/jwst/stpipe/step.py in run(self, *args) 398 else: 399 try: --> 400 step_result = self.process(*args) 401 except TypeError as e: 402 if "process() takes exactly" in str(e): ~/miniconda3/envs/jwstB72rc1/lib/python3.5/site-packages/jwst-0.12.0-py3.5-macosx-10.6-x86_64.egg/jwst/pipeline/calwebb_image2.py in process(self, input) 60 product, 61 asn['asn_pool'], ---> 62 op.basename(asn.filename) 63 ) 64 ~/miniconda3/envs/jwstB72rc1/lib/python3.5/site-packages/jwst-0.12.0-py3.5-macosx-10.6-x86_64.egg/jwst/pipeline/calwebb_image2.py in process_exposure_product(self, exp_product, pool_name, asn_file) 148 # regular science image types. 149 if input.meta.exposure.type.upper() in self.image_exptypes: --> 150 self.resample(input) 151 152 # That's all folks ~/miniconda3/envs/jwstB72rc1/lib/python3.5/site-packages/jwst-0.12.0-py3.5-macosx-10.6-x86_64.egg/jwst/stpipe/step.py in run(self, *args) 398 else: 399 try: --> 400 step_result = self.process(*args) 401 except TypeError as e: 402 if "process() takes exactly" in str(e): ~/miniconda3/envs/jwstB72rc1/lib/python3.5/site-packages/jwst-0.12.0-py3.5-macosx-10.6-x86_64.egg/jwst/resample/resample_step.py in process(self, input) 53 if len(input_models[0].data.shape) != 2: 54 # resample can only handle 2D images, not 3D cubes, etc ---> 55 raise RuntimeError("Input {} is not a 2D image.".format(input_models[0])) 56 57 # Get drizzle parameters reference file RuntimeError: Input is not a 2D image. {code}  
stscijgbot commented 5 years ago

Comment by James Davies: This is time-series data? It should not be resampled.

You want to use {{calwebb_tso-image2.cfg}} I believe for TSO data.

stscijgbot commented 5 years ago

Comment by James Davies: And it looks like we need to update our documentation to indicate this.

stscijgbot commented 5 years ago

Comment by Alicia Canipe: This is not time-series data, this is for a regular NIRCam imaging simulation with multiple integrations. I originally encountered it when trying to create a coronagraphy simulation with multiple integrations, so I was trying to double check that I also got the error for a regular NIRCam exposure with multiple integrations.

stscijgbot commented 5 years ago

Comment by James Davies: I'm using the TSO term here loosely here, sorry. I mean that the {{_rateints}} are 3D cubes with multiple integrations which correspond to measurements in time in the 3rd dimension and can be thought of as a time series, even if the actual pixel values are not diffs from the previous integration. {{Detector1Pipeline}} produces an {{_rate}} output that should be run through {{Image2Pipeline}}.

Anyway, resampling is only supported for 2D images, not 3D cubes. So running a {{_rateints}} file through all steps of the {{Image2Pipeline}} is not supported I believe. Perhaps [~bushouse] can comment?

Perhaps we should have a more graceful check in the resample step to be sure only 2D data is being passed to the drizzle algorithm.

hbushouse commented 5 years ago

@jdavies-st is correct. The resample step in the calwebb_image2 pipeline was not setup to handle anything other than 2-D images (not a 3-D stack of images). There is a hook in the calwebb_image2 pipeline that only executes the resample step for exposures with EXP_TYPE="MIR_IMAGE", "NRC_IMAGE", or "NIS_IMAGE", with the assumption being that any such exposures will consist of a simple 2-D image (i.e. the _rate product as input, not the _rateints). Hence the _rateints products for coronagraphy exposures, with EXP_TYPE="NRC_CORON" or "MIR_CORON", should pass through OK, because the resample step should get skipped (resampling will be applied later in the calwebb_coron3 pipeline).

So perhaps the hook in calwebb_image2 should be made more generic to check for 2-D vs. 3-D data, rather than EXP_TYPE value.

stscijgbot commented 5 years ago

Comment by James Davies: I'm using the TSO term here loosely here, sorry. I mean that the {{_rateints}} are 3D cubes with multiple integrations which correspond to measurements in time in the 3rd dimension and can be thought of as a time series, even if the actual pixel values are not diffs from the previous integration. {{Detector1Pipeline}} produces an {{_cal}} output that should be run through {{Image2Pipeline}}.

Anyway, resampling is only supported for 2D images, not 3D cubes. So running a {{_calints}} file through all steps of the {{Image2Pipeline}} is not supported I believe. Perhaps [~bushouse] can comment?

Perhaps we should have a more graceful check in the resample step to be sure only 2D data is being passed to the drizzle algorithm.

stscijgbot commented 5 years ago

Comment by Alicia Canipe: Thanks, Howard and James. I had misinterpreted the Image2 pipeline documentation (this is what I was reading: [https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/description.html#stage-2-imaging-pipeline-step-flow-calwebb-image2)] and didn't realize NRC_IMAGE rateints exposures would cause the resample step to fail. NRC_CORON rateints exposures run all the way through Image2, automatically skipping the resample step and generating the calints as expected.

stscieisenhamer commented 5 years ago

I de-assigned myself from the jira ticket. This particular issue seems to be resolved, but will let those involved with the conversation make the final determination.

stscijgbot commented 5 years ago

Comment by Howard Bushouse: Reviewing this again, I see that the resample step did issue the RuntimeError that was inserted for B7.2 that gets triggered when the input data are not 2-D, so that worked properly (rather than having the step crash later). The calwebb_spec2 pipeline also got updated to check for 2-D data before calling the resample_spec step, but the calwebb_image2 pipeline still only checks the value of EXP_TYPE when deciding whether to call the resample step. So that's why it makes it to the RuntimeError in resample before stopping. I'm wondering if it would be more user friendly to:

1) Make the same update to calwebb_image2 that was done to calwebb_spec2 to check the data dimensions before calling resample, so that 3-D data don't result in an error.

2) Change the RuntimeError in resample and resample_spec to be a warning and just return without producing an output.

Thoughts?

nden commented 5 years ago

it may be good to have a consistent way in the pipeline to deal with data which fails the expectations in some way. Currently it may be a mixture of the two solutions above. This consistency should be kept also at the step level. Other examples are: missing msa configuration file or passing a level1a file to assign_wcs. If we go with 2) above the Warning can be easily missed by users. My preference would be to go with 1) and catch the error if necessary to deal with it in a proper way.

stscijgbot commented 5 years ago

Comment by Howard Bushouse: I agree that at least implementing #1 will prevent the RuntimeError from being raised in the resample step, because the step will not be called, at least in the context of running the whole calwebb_image2 pipeline, which is the case in this particular instance.