search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
567 stars 167 forks source link

Outlier detection's save_intermediate_results errors #4163

Closed stscijgbot closed 4 years ago

stscijgbot commented 5 years ago

Issue JP-1078 was created by Shannon Osborne:

For an input ASN file, setting save_intermediate_results to True for the outlier detection step of the Image3Pipeline causes it to fail with the error "AttributeError: 'OutlierDetection' object has no attribute 'save_model' ". The same does not happen for _cal.fits inputs.

 

Example code:

{color:#cc7832}from {color}jwst.pipeline {color:#cc7832}import {color}Image3Pipeline

cal_file = {color:#6a8759}"/grp/jwst/wit/fgs/pipeline/build7.3_testing/files/im3_image_cal_asn.json" {color}

pipe = Image3Pipeline()

pipe.outlier_detection.save_intermediate_results = {color:#cc7832}True{color}

out = pipe.run(cal_file)

stscijgbot commented 5 years ago

Comment by Shannon Osborne: Full traceback:

 

2019-10-22 13:27:05,213 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created. 2019-10-22 13:27:05,220 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created. 2019-10-22 13:27:05,225 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created. 2019-10-22 13:27:05,230 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created. 2019-10-22 13:27:05,235 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created. 2019-10-22 13:27:05,239 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created. 2019-10-22 13:27:05,363 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('/grp/jwst/wit/fgs/pipeline/build7.3_testing/files/im3_image_cal_asn.json',). 2019-10-22 13:27:06,495 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00729011001_02101_00001_guider2_cal.fits' reftypes = ['drizpars'] 2019-10-22 13:27:06,498 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 2019-10-22 13:27:06,498 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'jw00729011001_02102_00002_guider2_cal.fits' reftypes = ['drizpars'] 2019-10-22 13:27:06,501 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is 'N/A'. 2019-10-22 13:27:06,501 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ... 2019-10-22 13:27:07,157 - stpipe.Image3Pipeline - INFO - Aligning input images... 2019-10-22 13:27:07,229 - stpipe.Image3Pipeline - WARNING - /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/step.py:350: ResourceWarning: unclosed file <_io.FileIO name='/user/sosborne/fgs_pipeline_testing/build_7.3/files/jw00729011001_02101_00001_guider2_cal.fits' mode='rb' closefd=True> gc.collect()

2019-10-22 13:27:07,229 - stpipe.Image3Pipeline - WARNING - /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/step.py:350: ResourceWarning: unclosed file <_io.FileIO name='/user/sosborne/fgs_pipeline_testing/build_7.3/files/jw00729011001_02102_00002_guider2_cal.fits' mode='rb' closefd=True> gc.collect()

2019-10-22 13:27:07,232 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (,). 2019-10-22 13:27:10,838 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 27 sources in jw00729011001_02101_00001_guider2_cal.fits. 2019-10-22 13:27:14,438 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 27 sources in jw00729011001_02102_00002_guider2_cal.fits. 2019-10-22 13:27:14,785 - stpipe.Image3Pipeline.tweakreg - INFO - 2019-10-22 13:27:14,785 - stpipe.Image3Pipeline.tweakreg - INFO - ***** jwst.tweakreg.imalign.align() started on 2019-10-22 13:27:14.785482 2019-10-22 13:27:14,785 - stpipe.Image3Pipeline.tweakreg - INFO - 2019-10-22 13:27:14,785 - stpipe.Image3Pipeline.tweakreg - INFO - Selected image 'jw00729011001_02101_00001_guider2_cal.fits' as reference image 2019-10-22 13:27:14,790 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'jw00729011001_02102_00002_guider2_cal.fits' to the reference catalog. 2019-10-22 13:27:14,801 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'jw00729011001_02102_00002_guider2_cal.fits' with sources from reference image 'jw00729011001_02101_00001_guider2_cal.fits' 2019-10-22 13:27:14,801 - stpipe.Image3Pipeline.tweakreg - INFO - Computing initial guess for X and Y shifts... 2019-10-22 13:27:14,801 - stpipe.Image3Pipeline.tweakreg - WARNING - /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/numpy/lib/histograms.py:1032: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer. edges[i] = np.linspace(smin, smax, bins[i] + 1)

2019-10-22 13:27:14,801 - stpipe.Image3Pipeline.tweakreg - INFO - Found initial X and Y shifts of 0, 0 based on a single non-zero bin and 27 matches 2019-10-22 13:27:14,803 - stpipe.Image3Pipeline.tweakreg - INFO - Found 27 matches for 'jw00729011001_02102_00002_guider2_cal.fits'... 2019-10-22 13:27:14,803 - stpipe.Image3Pipeline.tweakreg - INFO - Performing 'general' fit 2019-10-22 13:27:14,805 - stpipe.Image3Pipeline.tweakreg - INFO - Computed 'general' fit for jw00729011001_02102_00002_guider2_cal.fits: 2019-10-22 13:27:14,806 - stpipe.Image3Pipeline.tweakreg - INFO - XSH: -4.17309e-16 YSH: -1.60896e-16 PROPER ROT: 360 2019-10-22 13:27:14,806 - stpipe.Image3Pipeline.tweakreg - INFO - : 180 SKEW: -360 ROT_X: 360 ROT_Y: 6.01013e-16 2019-10-22 13:27:14,806 - stpipe.Image3Pipeline.tweakreg - INFO - : 1 SCALE_X: 1 SCALE_Y: 1 2019-10-22 13:27:14,806 - stpipe.Image3Pipeline.tweakreg - INFO - 2019-10-22 13:27:14,806 - stpipe.Image3Pipeline.tweakreg - INFO - XRMS: 8.81396e-15 YRMS: 5.61757e-16 2019-10-22 13:27:14,806 - stpipe.Image3Pipeline.tweakreg - INFO - Final solution based on 26 objects. 2019-10-22 13:27:14,863 - stpipe.Image3Pipeline.tweakreg - INFO - 2019-10-22 13:27:14,863 - stpipe.Image3Pipeline.tweakreg - INFO - jwst.tweakreg.imalign.align() ended on 2019-10-22 13:27:14.863448 2019-10-22 13:27:14,863 - stpipe.Image3Pipeline.tweakreg - INFO - jwst.tweakreg.imalign.align() TOTAL RUN TIME: 0:00:00.077966 2019-10-22 13:27:14,863 - stpipe.Image3Pipeline.tweakreg - INFO - 2019-10-22 13:27:14,865 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done 2019-10-22 13:27:14,865 - stpipe.Image3Pipeline - INFO - Matching sky values across all input images... 2019-10-22 13:27:14,936 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (,). 2019-10-22 13:27:15,039 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:15,039 - stpipe.Image3Pipeline.skymatch - INFO - jwst.skymatch.skymatch.match() started on 2019-10-22 13:27:15.039547 2019-10-22 13:27:15,039 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:15,039 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'global+match' 2019-10-22 13:27:15,039 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: DOWN 2019-10-22 13:27:15,039 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: OFF 2019-10-22 13:27:15,040 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:15,040 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions. 2019-10-22 13:27:16,347 - stpipe.Image3Pipeline.skymatch - INFO - Image ID=jw00729011001_02101_00001_guider2_cal.fits. Sky background: 0 2019-10-22 13:27:16,348 - stpipe.Image3Pipeline.skymatch - INFO - Image ID=jw00729011001_02102_00002_guider2_cal.fits. Sky background: 0 2019-10-22 13:27:16,348 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:16,348 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing "global" sky - smallest sky value across all input images. 2019-10-22 13:27:16,728 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - "Global" sky value correction: -10.17807253743414 [not converted] 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - ---- Final (match+global) sky for: 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - Image ID=jw00729011001_02101_00001_guider2_cal.fits. Sky background: -10.1781 (old=0, delta=-10.1781) 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - Image ID=jw00729011001_02102_00002_guider2_cal.fits. Sky background: -10.1781 (old=0, delta=-10.1781) 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - jwst.skymatch.skymatch.match() ended on 2019-10-22 13:27:16.729487 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:01.689940 2019-10-22 13:27:16,729 - stpipe.Image3Pipeline.skymatch - INFO - 2019-10-22 13:27:16,734 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done 2019-10-22 13:27:16,734 - stpipe.Image3Pipeline - INFO - Performing outlier detection on input images... 2019-10-22 13:27:16,808 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (,). 2019-10-22 13:27:16,811 - stpipe.Image3Pipeline.outlier_detection - INFO - Performing outlier detection on 2 inputs 2019-10-22 13:27:19,081 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (2048, 2048) --> (2111, 2048) 2019-10-22 13:27:24,850 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (2048, 2048) --> (2111, 2048) 2019-10-22 13:27:28,011 - stpipe.Image3Pipeline.outlier_detection - INFO - Writing out resampled exposures...

AttributeError Traceback (most recent call last)

in 5 pipe = Image3Pipeline() 6 pipe.outlier_detection.save_intermediate_results = True ----> 7 out = pipe.run(cal_file) /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/step.py in run(self, *args) 392 self.prefetch(*args) 393 try: --> 394 step_result = self.process(*args) 395 except TypeError as e: 396 if "process() takes exactly" in str(e): /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/pipeline/calwebb_image3.py in process(self, input) 74 75 self.log.info("Performing outlier detection on input images...") ---> 76 input_models = self.outlier_detection(input_models) 77 78 if input_models[0].meta.cal_step.outlier_detection == 'COMPLETE': /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/stpipe/step.py in run(self, *args) 392 self.prefetch(*args) 393 try: --> 394 step_result = self.process(*args) 395 except TypeError as e: 396 if "process() takes exactly" in str(e): /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/outlier_detection/outlier_detection_step.py in process(self, input) 164 # Set up outlier detection, then do detection 165 step = detection_step(self.input_models, reffiles=reffiles, **pars) --> 166 step.do_detection() 167 168 if self.input_container: /user/sosborne/miniconda3/envs/jwstdev73/lib/python3.6/site-packages/jwst-0.13.7-py3.6.egg/jwst/outlier_detection/outlier_detection.py in do_detection(self) 193 if save_intermediate_results: 194 log.info("Writing out resampled exposures...") --> 195 self.save_model( 196 model, 197 output_file=model.meta.filename, AttributeError: 'OutlierDetection' object has no attribute 'save_model'
stscijgbot commented 4 years ago

Comment by Stacy (Anastasia) Smith: Previously contained component: "validation_testing".

stscijgbot commented 4 years ago

Comment by Alicia Canipe: [~bushouse] other related tickets are closed... do you know if this issue has been resolved too?

stscijgbot commented 4 years ago

Comment by Jonathan Eisenhamer: This should be resolved by the same PR that resolved JP-1283.

stscijgbot commented 4 years ago

Comment by Alicia Canipe: I believe this should be fixed based on the status of the other tickets and Jonathan's comment. Will move this to "Ready for Testing".

stscijgbot commented 3 years ago

Comment by Alicia Canipe: Just confirming that this is now resolved. [~sosborne] FYI {code:java} 2020-11-11 13:00:17,535 - stpipe.Image3Pipeline.outlier_detection - INFO - Writing out MEDIAN image to: jw01069002001_01101_00001_nrca5_a3001_median.fits 2020-11-11 13:00:17,712 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting median... 2020-11-11 13:00:19,527 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:22,144 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:24,712 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:27,265 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:29,767 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:32,331 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:34,899 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:37,444 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (3219, 3066) 2020-11-11 13:00:37,840 - stpipe.Image3Pipeline.outlier_detection - INFO - Writing out BLOT images... 2020-11-11 13:00:44,741 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done {code} and the files are there: {code:java} -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002001_01101_00001_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002001_01101_00002_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002002_01101_00005_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002002_01101_00006_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002003_01101_00009_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002003_01101_00010_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002004_01101_00013_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff  118480320 Nov 11 13:00 jw01069002004_01101_00014_a3001_outlier_i2d.fits -rw-r--r--  1 acanipe  staff   39516480 Nov 11 13:00 jw01069002001_01101_00001_nrca5_a3001_median.fits -rw-r--r--  1 acanipe  staff  117547200 Nov 11 13:00 level3_F277W_asn_jw01069002001_01101_00001_nrca5_blot.fits_a3001_blot.fits -rw-r--r--  1 acanipe  staff  117552960 Nov 11 13:00 level3_F277W_asn_jw01069002001_01101_00002_nrca5_blot.fits_a3001_blot.fits -rw-r--r--  1 acanipe  staff  117552960 Nov 11 13:00 level3_F277W_asn_jw01069002002_01101_00005_nrca5_blot.fits_a3001_blot.fits -rw-r--r--  1 acanipe  staff  117552960 Nov 11 13:00 level3_F277W_asn_jw01069002002_01101_00006_nrca5_blot.fits_a3001_blot.fits -rw-r--r--  1 acanipe  staff  117552960 Nov 11 13:00 level3_F277W_asn_jw01069002003_01101_00009_nrca5_blot.fits_a3001_blot.fits -rw-r--r--  1 acanipe  staff  117552960 Nov 11 13:00 level3_F277W_asn_jw01069002003_01101_00010_nrca5_blot.fits_a3001_blot.fits -rw-r--r--  1 acanipe  staff  117552960 Nov 11 13:00 level3_F277W_asn_jw01069002004_01101_00013_nrca5_blot.fits_a3001_blot.fits -rw-r--r--  1 acanipe  staff  117552960 Nov 11 13:00 level3_F277W_asn_jw01069002004_01101_00014_nrca5_blot.fits_a3001_blot.fits {code}

stscijgbot commented 3 years ago

Comment by Alicia Canipe: Confirmed that this has been resolved, closing the ticket.