search

oirlab / liger_iris_pipeline

Data Reduction System (DRS) for the Thirty Meter Telescope IRIS imager/spectrograph
https://oirlab.github.io/liger-iris-pipeline
0 stars 4 forks source link

Simulated data at 4096 #5

Closed zonca closed 5 years ago

zonca commented 5 years ago

@nils-erik @arunsurya77 can you please prepare some simulated data at 4096x4096? The older test files I had are 2048x2048 but now I am testing flags at 4096x4096.

You can paste here the path on Galactica. Thanks!

arunsurya77 commented 5 years ago

Hi Andrea, I have added 2 sample files at 4k. Its at /data/group/packages/idlpro/iris/readout_iris/

sample_image.fits sample_flat.fits

It doesn't have the headers though. Is that required? Cheers Arun

On Wed, 17 Apr 2019 at 17:07, Andrea Zonca notifications@github.com wrote:

@nils-erik https://github.com/nils-erik @arunsurya77 https://github.com/arunsurya77 can you please prepare some simulated data at 4096x4096? The older test files I had are 2048x2048 but now I am testing flags at 4096x4096.

You can paste here the path on Galactica. Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/5, or mute the thread https://github.com/notifications/unsubscribe-auth/AKB37LVNBOSKZ55YICFPAA3PQ63UNANCNFSM4HGYXF4Q .

nils-erik commented 5 years ago

Hi Andrea, I've generated some sample calibration files and stored them at the following directory on galactica: /data/group/data/iris/dev/iris_cal_files/example_fits/ Let me know if there's anything about these files that should be changed or if there is important corresponding metadata missing. Best, Nils

zonca commented 5 years ago

@nils-erik for the "DQ" extension, you can have some saturated pixels, so I can test how I can merge the bad pixel mask with "per-frame" flags like saturation.

the data type should be float32, good pixel is 0, saturated is 2, see all flags at https://jwst-pipeline.readthedocs.io/en/latest/jwst/references_general/references_general.html#data-quality-flags

nils-erik commented 5 years ago

@zonca , I've generated new FITS images in the same directory in Galactica: /data/group/data/iris/dev/iris_cal_files/example_fits/ I've added a "science_frame_reduced" image to help give a target for what the reduction process should achieve. It should be noted that this "reduced" image is processed with essentially no noise properties (readnoise, background, darkcurrent, etc.) and so it is a fairly unphysical expectation to achieve through the reduction process. In the actual reduced frames there will be residuals and small variations that don't exist in this image. For both the "reduced" and "raw" science frames, I've attached a "DQ" extension with the value for "2" for a few saturated pixels.
Let me know if you need anything else or clarification on these images! Nils

zonca commented 5 years ago

@nils-erik astropy.io.fits reports a few illegal keywords:

2019-05-29 16:07:53,896 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IMGTEMP<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,898 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Note: astropy.io.fits uses zero-based indexing.

  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,899 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IMGNAME<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,901 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IFSTEMP<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,902 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IFSNAME<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,907 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'PRESNAM<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,909 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'PRESSUR<'
  warnings.warn(line, VerifyWarning)

this doesn't affect my tests as I'm stripping those keywords, so no need to rerun the files you created already

zonca commented 5 years ago

@nils-erik moved to https://github.com/oirlab/iris_pipeline/issues/8 for future reference

zonca commented 5 years ago

Required modifications to FITS for IRISImageModel

The modifications below make the raw science FITS file (after fixing headers as #8) readable as a IRISImageModel:

zonca commented 5 years ago

see #8 for next simulation data release