search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
568 stars 167 forks source link

extract_1d: does it work for LRS subarray SLITLESSPRISM. #3477

Closed JuergenSchreiber closed 4 years ago

JuergenSchreiber commented 5 years ago

This is might be more a question than an issue since I tried it only with an old jwst pipeline version (0.9.6). I tried to apply Spec2Pipeline on the output of a MIRISim LRS slltless simulation and failed. (there was no error message but there was only output some wavelengths at the very long end) I guess/speculate the reason is that the simulator output is a SLITLESSPRISM SUBARRAY with size of about 416 x 78 pixels and not the full 1024 x 1032 array which extract_1d seems to expect. So the question would be: What is the frame size the pipeline expects from MIRI in case of an LRS slitless prism observation? As said this is only a question and the issue might already be solved with the recent pipeline versions.

JuergenSchreiber commented 5 years ago

the logging output: "2019-05-14 16:27:56,857 - stpipe.Spec2Pipeline - INFO - Working on input results/det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits ...\n", "2019-05-14 16:27:57,023 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,).\n", "2019-05-14 16:27:58,293 - stpipe.Spec2Pipeline.assign_wcs - WARNING - /home/schreiber/anaconda/envs/miricle.devel/lib/python3.5/site-packages/jwst-0.9.6-py3.5-linux-x86_64.egg/jwst/assign_wcs/util.py:561: RuntimeWarning: invalid value encountered in less\n", " negative_ind = footprint[0] < 0\n", "\n", "2019-05-14 16:27:58,294 - stpipe.Spec2Pipeline.assign_wcs - INFO - There are NaNs in s_region\n", "2019-05-14 16:27:58,294 - stpipe.Spec2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs\n", "2019-05-14 16:27:58,298 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs done\n", "2019-05-14 16:27:58,376 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,).\n", "2019-05-14 16:27:58,447 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored\n", "2019-05-14 16:27:58,449 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored\n", "2019-05-14 16:27:58,450 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored\n", "2019-05-14 16:27:58,452 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored\n", "2019-05-14 16:27:58,535 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field done\n", "2019-05-14 16:27:58,618 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,).\n", "2019-05-14 16:27:58,642 - stpipe.Spec2Pipeline.srctype - INFO - Input EXP_TYPE is MIR_LRS-SLITLESS\n", "2019-05-14 16:27:58,643 - stpipe.Spec2Pipeline.srctype - INFO - Input SRCTYPE is unknown. Setting to default value of POINT\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2019-05-14 16:27:58,644 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype done\n", "2019-05-14 16:27:58,723 - stpipe.Spec2Pipeline.photom - INFO - Step photom running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,).\n", "2019-05-14 16:27:58,751 - stpipe.Spec2Pipeline.photom - INFO - Using photom reference file: /home/schreiber/CRDS/references/jwst/miri/jwst_miri_photom_0014.fits\n", "2019-05-14 16:27:58,757 - stpipe.Spec2Pipeline.photom - INFO - Using area reference file: N/A\n", "2019-05-14 16:27:58,758 - stpipe.Spec2Pipeline.photom - INFO - Using instrument: MIRI\n", "2019-05-14 16:27:58,759 - stpipe.Spec2Pipeline.photom - INFO - detector: MIRIMAGE\n", "2019-05-14 16:27:58,759 - stpipe.Spec2Pipeline.photom - INFO - exp_type: MIR_LRS-SLITLESS\n", "2019-05-14 16:27:58,760 - stpipe.Spec2Pipeline.photom - INFO - filter: P750L\n", "2019-05-14 16:27:58,808 - stpipe.Spec2Pipeline.photom - INFO - subarray: SUBPRISM\n", "2019-05-14 16:27:58,811 - stpipe.Spec2Pipeline.photom - INFO - PHOTMJSR value: 1\n", "2019-05-14 16:27:58,814 - stpipe.Spec2Pipeline.photom - INFO - Storing relative response table\n", "2019-05-14 16:27:58,818 - stpipe.Spec2Pipeline.photom - INFO - Step photom done\n", "2019-05-14 16:27:58,895 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,).\n", "2019-05-14 16:27:58,925 - stpipe.Spec2Pipeline.extract_1d - INFO - Using EXTRACT1D reference file /home/schreiber/miricap302/extract1D_ref1.json\n", "2019-05-14 16:27:58,958 - stpipe.Spec2Pipeline.extract_1d - WARNING - spectral_order is None; using 1\n", "2019-05-14 16:27:58,975 - stpipe.Spec2Pipeline.extract_1d - INFO - Total number of dither points = None; assuming no nod/dither offset\n", "2019-05-14 16:27:59,021 - stpipe.Spec2Pipeline.extract_1d - WARNING - Using RELSENS, 265 elements were extrapolated; these values will be set to 1.\n", "2019-05-14 16:27:59,087 - stpipe.Spec2Pipeline.extract_1d - WARNING - INTSTART not found; assuming a value of 1.\n", "2019-05-14 16:27:59,088 - stpipe.Spec2Pipeline.extract_1d - WARNING - INTEND not found; assuming a value of 10.\n", "2019-05-14 16:27:59,088 - stpipe.Spec2Pipeline.extract_1d - WARNING - There is no INT_TIMES table in the input file.\n", "2019-05-14 16:27:59,175 - stpipe.Spec2Pipeline.extract_1d - INFO - Saved model in results/det_image_seq1_MIRIMAGE_P750Lexp1_x1dints.fits\n", "2019-05-14 16:27:59,176 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d done\n", "2019-05-14 16:27:59,177 - stpipe.Spec2Pipeline - INFO - Finished processing product det_image_seq1_MIRIMAGE_P750Lexp1_x1d\n", "2019-05-14 16:27:59,343 - stpipe.Spec2Pipeline - INFO - Saved model in results/det_image_seq1_MIRIMAGE_P750Lexp1_calints.fits\n", "2019-05-14 16:27:59,418 - stpipe.Spec2Pipeline - INFO - Ending calwebb_spec2\n", "2019-05-14 16:27:59,421 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline done\n", "2019-05-14 16:27:59,441 - stpipe.Spec2Pipeline - INFO - Spec2Pipeline instance created.\n", "2019-05-14 16:27:59,444 - stpipe.Spec2Pipeline.extract_1d - INFO - Extract1dStep instance created.\n",

philhodge commented 5 years ago

Could I have a copy of your _rate.fits file and your extract1D_ref1.json reference file? I'll check it out. I suspect it might have to do with the bounding box, since you said that a portion of the spectrum was extracted. I'll try running the current version of the spec2 pipeline. There have been many changes recently.

JuergenSchreiber commented 5 years ago

Hi Phil, thanks for looking into it. yes thats right, especially with the updated specwcs CRDS files, there are now completely different IMXSLTL, IMYSLTL reference point values in the header which alone make a big difference I send the requested files to your email adress, since github does not like those kind of files....

philhodge commented 5 years ago

The extract_1d step can work with subarrays; that was not the problem. The photom reference file jwst_miri_photom_0014.fits only includes data for wavelengths between 4.973034 and 12.010071 microns for SUBPRISM, so the conversion from net count rate to flux was only done for that range of wavelengths. The problem is that extract_1d should have set the flux to zero for pixels outside that range, but instead it left them in units of count rate. Since the count rate is numerically much larger than the flux, it looked as if the spectrum was populated only for wavelengths longer than 12 microns. As a workaround, you can extract a slice from the arrays of wavelength and flux in the x1d file. The exact slice limits can depend on your version of jwst, but the slice would be approximately [139:373].

In the latest version of the JWST pipeline code ('0.13.3a.dev22'), the photom step is the one that actually applies the conversion from count rate to flux (rather than just adding a RELSENS extension to the data file), and photom now does set the data to zero for pixels outside the range of wavelengths in the photom reference file.

I also have a suggestion regarding the extract_1d reference file. In extract1D_ref1.json, you specified "xstart": 26, "xstop": 42, and "extract_width": 16. The xstart and xstop limits are not Python slice indices, they're inclusive limits (but zero-based indices, like Python). extract_width takes precedence for the number of pixels to include in the extraction region, but if there's a conflict, xstart and xstop are still used to determine the center of the extraction region. Since you specified 16 for extract_width, I'll guess that you intended the extraction to be from x = 26 through x = 41, inclusive. If that's the case, I would suggest that you change "xstop" to 41; other than that, your reference file looks fine.

JuergenSchreiber commented 5 years ago

Hi Phil, now I installed version 0.13.2 and tried it again. It uses the newest reference files jwst_miri_specwcs_0003.fits, jwst_miri_distortion_0028.asdf etc. Also the values of the ...cal.fits output look fine. I applied it with the photom step skipped, but the extracted ...x1d.fits has much too low values, I would say about a factor of 1e6 too low. The wavelength range now looks fine. ( I also applied it to slit mode and the values are much too high....) I copy the log:

2019-06-03 12:38:10,063 - stpipe.Spec2Pipeline - INFO - Spec2Pipeline instance created. 2019-06-03 12:38:10,065 - stpipe.Spec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created. 2019-06-03 12:38:10,075 - stpipe.Spec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created. 2019-06-03 12:38:10,081 - stpipe.Spec2Pipeline.imprint_subtract - INFO - ImprintStep instance created. 2019-06-03 12:38:10,086 - stpipe.Spec2Pipeline.msa_flagging - INFO - MSAFlagOpenStep instance created. 2019-06-03 12:38:10,090 - stpipe.Spec2Pipeline.extract_2d - INFO - Extract2dStep instance created. 2019-06-03 12:38:10,094 - stpipe.Spec2Pipeline.flat_field - INFO - FlatFieldStep instance created. 2019-06-03 12:38:10,097 - stpipe.Spec2Pipeline.srctype - INFO - SourceTypeStep instance created. 2019-06-03 12:38:10,101 - stpipe.Spec2Pipeline.straylight - INFO - StraylightStep instance created. 2019-06-03 12:38:10,108 - stpipe.Spec2Pipeline.fringe - INFO - FringeStep instance created. 2019-06-03 12:38:10,112 - stpipe.Spec2Pipeline.pathloss - INFO - PathLossStep instance created. 2019-06-03 12:38:10,115 - stpipe.Spec2Pipeline.barshadow - INFO - BarShadowStep instance created. 2019-06-03 12:38:10,117 - stpipe.Spec2Pipeline.photom - INFO - PhotomStep instance created. 2019-06-03 12:38:10,122 - stpipe.Spec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created. 2019-06-03 12:38:10,125 - stpipe.Spec2Pipeline.cube_build - INFO - CubeBuildStep instance created. 2019-06-03 12:38:10,129 - stpipe.Spec2Pipeline.extract_1d - INFO - Extract1dStep instance created. 2019-06-03 12:38:10,247 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline running with args ('results/det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits',). 2019-06-03 12:38:10,382 - stpipe.Spec2Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits' reftypes = ['barshadow', 'camera', 'collimator', 'cubepar', 'dflat', 'disperser', 'distortion', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'fringe', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'msaoper', 'ote', 'pathloss', 'regions', 'resol', 'sflat', 'specwcs', 'straymask', 'wavecorr', 'wavelengthrange', 'wfssbkg'] 2019-06-03 12:38:10,390 - stpipe.Spec2Pipeline - INFO - Prefetch for BARSHADOW reference file is 'N/A'. 2019-06-03 12:38:10,391 - stpipe.Spec2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'. 2019-06-03 12:38:10,392 - stpipe.Spec2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'. 2019-06-03 12:38:10,392 - stpipe.Spec2Pipeline - INFO - Prefetch for CUBEPAR reference file is 'N/A'. 2019-06-03 12:38:10,393 - stpipe.Spec2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'. 2019-06-03 12:38:10,393 - stpipe.Spec2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'. 2019-06-03 12:38:10,394 - stpipe.Spec2Pipeline - INFO - Prefetch for DISTORTION reference file is '/home/schreiber/CRDS/references/jwst/miri/jwst_miri_distortion_0028.asdf'. 2019-06-03 12:38:10,395 - stpipe.Spec2Pipeline - INFO - Override for EXTRACT1D reference file is '/home/schreiber/miricap302/extract1D_ref1.json'. 2019-06-03 12:38:10,396 - stpipe.Spec2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'. 2019-06-03 12:38:10,397 - stpipe.Spec2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'. 2019-06-03 12:38:10,397 - stpipe.Spec2Pipeline - INFO - Prefetch for FLAT reference file is '/home/schreiber/CRDS/references/jwst/miri/jwst_miri_flat_0245.fits'. 2019-06-03 12:38:10,399 - stpipe.Spec2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'. 2019-06-03 12:38:10,399 - stpipe.Spec2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'. 2019-06-03 12:38:10,400 - stpipe.Spec2Pipeline - INFO - Prefetch for FRINGE reference file is 'N/A'. 2019-06-03 12:38:10,401 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'. 2019-06-03 12:38:10,401 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'. 2019-06-03 12:38:10,402 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'. 2019-06-03 12:38:10,405 - stpipe.Spec2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'. 2019-06-03 12:38:10,406 - stpipe.Spec2Pipeline - INFO - Prefetch for MSAOPER reference file is 'N/A'. 2019-06-03 12:38:10,407 - stpipe.Spec2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'. 2019-06-03 12:38:10,408 - stpipe.Spec2Pipeline - INFO - Prefetch for PATHLOSS reference file is 'N/A'. 2019-06-03 12:38:10,409 - stpipe.Spec2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'. 2019-06-03 12:38:10,409 - stpipe.Spec2Pipeline - INFO - Prefetch for RESOL reference file is 'N/A'. 2019-06-03 12:38:10,410 - stpipe.Spec2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'. 2019-06-03 12:38:10,411 - stpipe.Spec2Pipeline - INFO - Prefetch for SPECWCS reference file is '/home/schreiber/CRDS/references/jwst/miri/jwst_miri_specwcs_0003.fits'. 2019-06-03 12:38:10,420 - stpipe.Spec2Pipeline - INFO - Prefetch for STRAYMASK reference file is 'N/A'. 2019-06-03 12:38:10,421 - stpipe.Spec2Pipeline - INFO - Prefetch for WAVECORR reference file is 'N/A'. 2019-06-03 12:38:10,422 - stpipe.Spec2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'. 2019-06-03 12:38:10,423 - stpipe.Spec2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'. 2019-06-03 12:38:10,425 - stpipe.Spec2Pipeline - INFO - Starting calwebb_spec2 ... 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 2019-06-03 12:38:10,443 - stpipe.Spec2Pipeline - WARNING - /home/schreiber/anaconda/envs/jwst/lib/python3.6/site-packages/jwst-0.13.2-py3.6.egg/jwst/associations/association_io.py:116: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. asn = yaml_lib.load(serialized)

201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte"

201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 17280: invalid start byte" 201906031238:DEBUG:jwst.associations.association_io.load:Error unserializing: "'utf-8' codec can't decode byte 0xbf in position 896: invalid start byte" 2019-06-03 12:38:11,034 - stpipe.Spec2Pipeline - INFO - Processing product det_image_seq1_MIRIMAGE_P750Lexp1_x1d 2019-06-03 12:38:11,036 - stpipe.Spec2Pipeline - INFO - Working on input results/det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits ... 2019-06-03 12:38:11,279 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,). 2019-06-03 12:38:11,598 - stpipe.Spec2Pipeline.assign_wcs - WARNING - /home/schreiber/anaconda/envs/jwst/lib/python3.6/site-packages/asdf/asdf.py:202: UserWarning: File 'file:///home/schreiber/CRDS/references/jwst/miri/jwst_miri_distortion_0028.asdf' was created with extension 'asdf.extension.BuiltinExtension' from package asdf-2.4.0.dev1705, but older version asdf-2.4.0.dev57+g178194c is installed warnings.warn(fmt_msg)

2019-06-03 12:38:12,621 - stpipe.Spec2Pipeline.assign_wcs - INFO - Created a MIRI mir_lrs-slitless pipeline with references {'distortion': '/home/schreiber/CRDS/references/jwst/miri/jwst_miri_distortion_0028.asdf', 'filteroffset': None, 'specwcs': '/home/schreiber/CRDS/references/jwst/miri/jwst_miri_specwcs_0003.fits', 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None} 2019-06-03 12:38:12,828 - stpipe.Spec2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 359.998913541 -0.000094318 359.998913541 0.000098846 0.001031224 0.000098846 0.001031224 -0.000094318 2019-06-03 12:38:12,829 - stpipe.Spec2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 359.998913541 -0.000094318 359.998913541 0.000098846 0.001031224 0.000098846 0.001031224 -0.000094318 2019-06-03 12:38:12,829 - stpipe.Spec2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs 2019-06-03 12:38:12,842 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs done 2019-06-03 12:38:12,926 - stpipe.Spec2Pipeline - WARNING - /home/schreiber/anaconda/envs/jwst/lib/python3.6/site-packages/jwst-0.13.2-py3.6.egg/jwst/stpipe/step.py:350: ResourceWarning: unclosed file <_io.FileIO name='results/det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits' mode='rb' closefd=True> gc.collect()

2019-06-03 12:38:12,931 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,). 2019-06-03 12:38:13,001 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored 2019-06-03 12:38:13,004 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored 2019-06-03 12:38:13,007 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored 2019-06-03 12:38:13,009 - stpipe.Spec2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored 2019-06-03 12:38:13,147 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field done 2019-06-03 12:38:13,221 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,). 2019-06-03 12:38:13,243 - stpipe.Spec2Pipeline.srctype - INFO - Input EXP_TYPE is MIR_LRS-SLITLESS 2019-06-03 12:38:13,244 - stpipe.Spec2Pipeline.srctype - INFO - Input SRCTYPE is unknown; setting default SRCTYPE = POINT 2019-06-03 12:38:13,252 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype done 2019-06-03 12:38:13,339 - stpipe.Spec2Pipeline.photom - INFO - Step photom running with args (<ImageModel(416, 72) from det_image_seq1_MIRIMAGE_P750Lexp1_rate.fits>,). 2019-06-03 12:38:13,342 - stpipe.Spec2Pipeline.photom - INFO - Step skipped. 2019-06-03 12:38:13,344 - stpipe.Spec2Pipeline.photom - INFO - Step photom done 2019-06-03 12:38:13,432 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d running with args (<ImageModel(416, 72) from results/det_image_seq1_MIRIMAGE_P750Lexp1_cal.fits>,). 2019-06-03 12:38:13,458 - stpipe.Spec2Pipeline.extract_1d - INFO - Using EXTRACT1D reference file /home/schreiber/miricap302/extract1D_ref1.json 2019-06-03 12:38:13,525 - stpipe.Spec2Pipeline.extract_1d - WARNING - spectral_order is None; using 1 2019-06-03 12:38:13,756 - stpipe.Spec2Pipeline.extract_1d - WARNING - Nod/dither offset is NaN; setting it to 0. 2019-06-03 12:38:13,864 - stpipe.Spec2Pipeline.extract_1d - WARNING - 4 NaNs in wavelength array 2019-06-03 12:38:13,937 - stpipe.Spec2Pipeline.extract_1d - INFO - Output arrays have been trimmed by 4 elements 2019-06-03 12:38:14,097 - stpipe.Spec2Pipeline.extract_1d - WARNING - Using cdelt1 = 0.1099 arcsec for both dimensions of a pixel 2019-06-03 12:38:14,298 - stpipe.Spec2Pipeline.extract_1d - INFO - Saved model in results/det_image_seq1_MIRIMAGE_P750Lexp1_x1d.fits 2019-06-03 12:38:14,299 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d done 2019-06-03 12:38:14,303 - stpipe.Spec2Pipeline - INFO - Finished processing product det_image_seq1_MIRIMAGE_P750Lexp1_x1d 2019-06-03 12:38:14,304 - stpipe.Spec2Pipeline - INFO - Ending calwebb_spec2 2019-06-03 12:38:14,691 - stpipe.Spec2Pipeline - INFO - Saved model in results/det_image_seq1_MIRIMAGE_P750Lexp1_cal.fits 2019-06-03 12:38:14,693 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline done

philhodge commented 5 years ago

The extract_1d step no longer writes NET or NERROR columns. The FLUX is in units of janskys (or will be, after the photom reference files are updated), and extract_1d also writes a SURF_BRIGHT (surface brightness) column. After running photom, the _cal.fits data will be in units of surface brightness, MJy / steradian. extract_1d populates the SURF_BRIGHT column with the sum of the pixel values divided by the number of elements that were added together. The value for the FLUX column is gotten by multiplying the sum of the pixel values by the solid angle of a pixel (thereby converting MJy / sr to MJy), and then multiplying by 1.e6 (to convert MJy to Jy). Doing that manually, I get:

fd = fits.open("det_image_seq1_MIRIMAGE_P750Lexp1_x1d.fits")
tbl = fd[1].data
wl = tbl.field("wavelength")
flux = tbl.field("flux")
sb = tbl.field("surf_bright")
npixels = tbl.field("npixels")
fd_cal = fits.open("det_image_seq1_MIRIMAGE_P750Lexp1_cal.fits")
cal = fd_cal[1].data
cal.shape
(416, 72)
wl.shape
(387,)
cdelt = 0.1099 / 3600.
pixel_solid_angle = cdelt**2 * (math.pi / 180.)**2
temp = cal[9:396, 26:42].sum(axis=1)
sb_cal = temp / 16.
flux_cal = temp * pixel_solid_angle * 1.e6

The value for cdelt is a little off; I took the value from what was printed in the log output.

Comparing the output from extract_1d with the values calculated above, I get:

r_sb = sb_cal / sb
r_flux = flux_cal / flux
r_sb.min(), r_sb.mean(), r_sb.max()
(0.9999998557510801, 1.0000000052468327, 1.000001114846342)
r_flux.min(), r_flux.mean(), r_flux.max()
(0.99974759933795, 0.9997477528529608, 0.9997487813331171)

The imperfect flux values are, I believe, due to the limited precision of cdelt.

JuergenSchreiber commented 5 years ago

Hi Phil, Thanks for your check, but that is still related to the old 0.9.6 version. sorry, I did not send you the new files generated with the 0.13.2 version, now in attachment. I ran it skipping the photom step, so I do not expect Jy output, but e/s, so in principle the sum of the pixels values of the cal file for each wavelength. (This is done, since I want to determine the PHOTOM CDS and need a "raw" non-calibrated spectrum). As you can see, the x1d "flux" values are much too low. Cheers, Juergen

On 2019-06-07 17:31, Phil Hodge wrote:

The extract_1d step no longer writes NET or NERROR columns. The FLUX is in units of janskys (or will be, after the photom reference files are updated), and extract_1d also writes a SURF_BRIGHT (surface brightness) column. After running photom, the _cal.fits data will be in units of surface brightness, MJy / steradian. extract_1d populates the SURF_BRIGHT column with the sum of the pixel values divided by the number of elements that were added together. The value for the FLUX column is gotten by multiplying the sum of the pixel values by the solid angle of a pixel (thereby converting MJy / sr to MJy), and then multiplying by 1.e6 (to convert MJy to Jy). Doing that manually, I get:

fd = fits.open("det_image_seq1_MIRIMAGE_P750Lexp1_x1d.fits") tbl = fd[1].data wl = tbl.field("wavelength") flux = tbl.field("flux") sb = tbl.field("surf_bright") npixels = tbl.field("npixels") fd_cal = fits.open("det_image_seq1_MIRIMAGE_P750Lexp1_cal.fits") cal = fd_cal[1].data cal.shape (416, 72) wl.shape (387,) cdelt = 0.1099 / 3600. pixel_solid_angle = cdelt*2 (math.pi / 180.)*2 temp = cal[9:396, 26:42].sum(axis=1) sb_cal = temp / 16. flux_cal = temp pixel_solid_angle * 1.e6

The value for cdelt is a little off; I took the value from what was printed in the log output.

Comparing the output from extract_1d with the values calculated above, I get:

r_sb = sb_cal / sb r_flux = flux_cal / flux r_sb.min(), r_sb.mean(), r_sb.max() (0.9999998557510801, 1.0000000052468327, 1.000001114846342) r_flux.min(), r_flux.mean(), r_flux.max() (0.99974759933795, 0.9997477528529608, 0.9997487813331171)

The imperfect flux values are, I believe, due to the limited precision of cdelt.

-- You are receiving this because you commented. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/spacetelescope/jwst/issues/3477?email_source=notifications&amp;email_token=AE2ADKHJHX7IRM4O4SANYRTPZJ5NVA5CNFSM4HNNCPOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXGFTDQ#issuecomment-499931534 [2] https://github.com/notifications/unsubscribe-auth/AE2ADKFATYJAJVNBHN7TUFDPZJ5NVANCNFSM4HNNCPOA

philhodge commented 5 years ago

Juergen,

I don't see the attachments, but three weeks ago I did run calwebb_spec2 using the 0.13.4 version, and my reply at that time should still be relevant. I see your point: it would make more sense if extract_1d did not multiply the flux by the solid angle of a pixel if the photom step has not been run. But currently extract_1d does multiply the flux by the solid angle of a pixel (and by 1.e6), so you should expect the values in the flux column to be much smaller (by about the factor 3.e-7) than the sum of the count rates.

As a workaround, you can use the values in the surf_bright column multiplied by the values in the npixels column. surf_bright is supposed to be the surface brightness (and it would be, if photom had been run), so it's the sum of the pixel values in the input file, divided by the number of pixels that were added together. That number of pixels is saved in the npixels column, so by multiplying surf_bright by npixels you will have the sum of the count rate pixel values, which (if I understand correctly) is what you want.

Phil

JuergenSchreiber commented 5 years ago

Phil, thanks a lot, that sounds reasonable, I'll try the workaround

JuergenSchreiber commented 5 years ago

The workaround works, but there is one caveat which should be documented much more detailed. The aperture selection parameters in the extract1d json file xstart, xstop refer to the resampled slit region (43x387 pixels) not the full 1032x1024 frame which is not documented at all in the jdocs.

philhodge commented 5 years ago

Thank you for pointing that out. That's a serious oversight, and I'll fix it.

hbushouse commented 5 years ago

So to summarize, it appears we want to do at least 2 things as a result of this discussion:

1) Update the extract_1d step to NOT multiply the FLUX values by the solid angle of a pixel if the photom step has not been run (i.e. if the data are still in units of countrate)

2) Update the extract_1d documentation to make it more obvious that the extraction region specified in the extract1d reference file (via xstart/xstop and ystart/ystop values) refer to the subarray frame and not the original 1032x1024 detector frame.

Is that correct?

philhodge commented 5 years ago

Yes, I agree.

JuergenSchreiber commented 5 years ago

Me too...

On 2019-07-15 16:56, Phil Hodge wrote:

Yes, I agree.

-- You are receiving this because you commented. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/spacetelescope/jwst/issues/3477?email_source=notifications&amp;email_token=AE2ADKFRNYJ7LYZQWY64HPDP7SFY7A5CNFSM4HNNCPOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ56S5Q#issuecomment-511437174 [2] https://github.com/notifications/unsubscribe-auth/AE2ADKFTL3OI7GMSLAXLRULP7SFY7ANCNFSM4HNNCPOA

hbushouse commented 5 years ago

Tracking in Jira at https://jira.stsci.edu/browse/JP-859

hbushouse commented 4 years ago

Fixed by #4369