search

oirlab / liger_iris_pipeline

Data Reduction System (DRS) for the Thirty Meter Telescope IRIS imager/spectrograph
https://oirlab.github.io/iris-pipeline
0 stars 3 forks source link

Make FITS from simulator compatible with iris_pipeline #8

Closed zonca closed 4 years ago

zonca commented 5 years ago

Fix illegal keywords, FIXED in v2

replace < with 0 or something else

@nils-erik astropy.io.fits reports a few illegal keywords:

2019-05-29 16:07:53,896 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IMGTEMP<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,898 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Note: astropy.io.fits uses zero-based indexing.

  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,899 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IMGNAME<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,901 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IFSTEMP<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,902 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'IFSNAME<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,907 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'PRESNAM<'
  warnings.warn(line, VerifyWarning)

2019-05-29 16:07:53,909 - stpipe - WARNING - /home/azonca/anaconda3/envs/jwst_dev/lib/python3.6/site-packages/astropy/io/fits/verify.py:117: VerifyWarning: Unfixable error: Illegal keyword name 'PRESSUR<'
  warnings.warn(line, VerifyWarning)

this doesn't affect my tests as I'm stripping those keywords, so no need to rerun the files you created already

zonca commented 5 years ago

Fix bad pixel extension

also, extension 1 for bad pixels has the wrong order, SIMPLE should be after the others, i.e.:

from:

XTENSION= 'DQ      '           /                                                
SIMPLE  =                    T /                                                
BITPIX  =                  -32 / IEEE single precision floating point           
NAXIS   =                    2 /                                                
NAXIS1  =                 4096 /                                                
NAXIS2  =                 4096 /                                                
SCALE   =           0.00400000 / pixel scale (arcsec)                           
UNITS   = 'BAD PIXELS'         /

to:

XTENSION= 'DQ      '           /                                                
BITPIX  =                  -32 / IEEE single precision floating point           
NAXIS   =                    2 /                                                
NAXIS1  =                 4096 /                                                
NAXIS2  =                 4096 /                                                
PCOUNT  =                    0                                                  
GCOUNT  =                    1                                                  
SIMPLE  =                    T /                                                
SCALE   =           0.00400000 / pixel scale (arcsec)                           
UNITS   = 'BAD PIXELS'         /
zonca commented 5 years ago

NFRAMES should be an integer, not a float

zonca commented 5 years ago

Required headers for the pipeline

this should be only in extension 0, not repeated in extension 1

SIMPLE  =                    T / conforms to FITS standard                      
BITPIX  =                    8 / array data type                                
NAXIS   =                    0 / number of array dimensions                     
EXTEND  =                    T                                                  

        Level 3 Schema Metadata                                                 

DATE    = '2019-01-04T16:06:30.733' / [yyyy-mm-ddThh:mm:ss.ss] UTC date file cre
FILENAME= 'sci.fits'           / Name of the file                               
DATAMODL= 'IRISImageModel'     / Type of data model                             
TELESCOP= 'TMT     '           / Telescope used to acquire the data             

        Observation identifiers                                                 

DATE-OBS= '2019-01-01'         / [yyyy-mm-dd] UTC date at start of exposure     
TIME-OBS= '12:00:00.000'       / [hh:mm:ss.sss] UTC time at start of exposure   

        Instrument configuration information                                    

INSTRUME= 'IRIS    '           / Instrument used to acquire the data            
DETECTOR= 'IRIS1   '           / Name of detector used to acquire the data      
FILTER  = 'K       '           / Name of the filter element used                
PUPIL   = 'CLEAR   '           / Name of the pupil element used                 

        Exposure parameters                                                     

EXP_TYPE= 'IRIS_IMAGE'         / Type of data in the exposure                   

        Subarray parameters                                                     

SUBARRAY= 'FULL    '           / Subarray used                                  
SUBSTRT1=                    1 / Starting pixel in axis 1 direction             
SUBSTRT2=                    1 / Starting pixel in axis 2 direction             
SUBSIZE1=                 2048 / Number of pixels in axis 1 direction           
SUBSIZE2=                 2048 / Number of pixels in axis 2 direction
zonca commented 5 years ago

Move data to extension 1

There should be no data in extension 0, move science to extension 1 named SCI, extension with bad pixels should be named DQ, make sure EXTNAME is DQ, not XTENSION

zonca commented 5 years ago

@nils-erik this should be everything, I have created a title for each modification to better organize this. When these are ready and you want to point me to the files, I can crosscheck.

zonca commented 5 years ago

@nils-erik do you have an estimate of when a new set of FITS file is going to be ready?

nils-erik commented 5 years ago

@zonca I have moved updated FITS images to the directory '/data/group/data/iris/dev/iris_cal_files/example_fits/'. For all of the images I have included a null primary directory with metadata and extensions for the image data with what I believe are appropriate headers. Please let me know what other requirements there are for these images if/when there are any.

zonca commented 5 years ago

thanks @nils-erik, most issues are fixed.

I am checking only raw_science_frame_sci.fits, a problem is that XTENSION is 1, instead it should be IMAGE, otherwise astropy thinks it is a non standard extension instead of an image. If I try to fix that, I get these other errors:

HDU 0:
    'BITPIX' card does not exist.
    'NAXIS' card at the wrong place (card 1).
    'EXTEND' card at the wrong place (card 2).
HDU 1:
    'BITPIX' card at the wrong place (card 2).
    'NAXIS' card at the wrong place (card 3).
    'NAXIS1' card at the wrong place (card 4).
    'NAXIS2' card at the wrong place (card 5).
    'PCOUNT' card does not exist.
    'GCOUNT' card does not exist.
HDU 2:
    'BITPIX' card at the wrong place (card 2).
    'NAXIS' card at the wrong place (card 3).
    'NAXIS1' card at the wrong place (card 4).
    'NAXIS2' card at the wrong place (card 5).
    'PCOUNT' card at the wrong place (card 6).
    'GCOUNT' card at the wrong place (card 7).

Are you using a IDL library to write the FITS files? What about using AstroIDL or something similar https://idlastro.gsfc.nasa.gov/contents.html#C9 ? Those libraries should automatically create those headers properly.

zonca commented 5 years ago

astropy allows to fix those errors automatically, using f.writeto("v2/raw_science_frame_sci.fits", overwrite=True, output_verify="fix") I wrote the fixed file in /home/azonca/p/software/IRIS-data-reduction-tests/201905_iris_pipeline_0.1_calibration_sims/example_fits/v2/raw_science_frame_sci.fits on Galactica

nils-erik commented 5 years ago

@zonca I fixed the 'XTENSION' value to 'IMAGE', and put the images in the same directory. I write the images with 'writefits', which is from the AstroIDL library. I tried earlier to use fits_write (from the same library) but ran into extension overwrite issues. Let me know if there are any other issues though!

zonca commented 5 years ago
nils-erik commented 5 years ago

Should be fixed

zonca commented 5 years ago

thanks!

only remaining issue is the order of the headers, this is not critical as they are only warnings, the software works anyway, so up to you if you want to fix them:

in background:

WARNING: VerifyWarning: Verification reported errors: [astropy.io.fits.verify]
WARNING: VerifyWarning: HDU 1: [astropy.io.fits.verify]
WARNING: VerifyWarning:     'BITPIX' card at the wrong place (card 2).  Fixed by moving it to the right place (card 1). [astropy.io.fits.verify]
WARNING: VerifyWarning:     'NAXIS' card at the wrong place (card 3).  Fixed by moving it to the right place (card 2). [astropy.io.fits.verify]
WARNING: VerifyWarning:     'NAXIS1' card at the wrong place (card 4).  Fixed by moving it to the right place (card 3). [astropy.io.fits.verify]
WARNING: VerifyWarning:     'NAXIS2' card at the wrong place (card 5).  Fixed by moving it to the right place (card 4). [astropy.io.fits.verify]
WARNING: VerifyWarning:     'PCOUNT' card does not exist.  Fixed by inserting a new 'PCOUNT' card. [astropy.io.fits.verify]
WARNING: VerifyWarning:     'GCOUNT' card does not exist.  Fixed by inserting a new 'GCOUNT' card. [astropy.io.fits.verify]
WARNING: VerifyWarning: Note: astropy.io.fits uses zero-based indexing.
 [astropy.io.fits.verify]

in raw_science:

WARNING: VerifyWarning: HDU 0: [astropy.io.fits.verify]
WARNING: VerifyWarning:     'BITPIX' card does not exist.  Fixed by inserting a new 'BITPIX' card. [astropy.io.fits.verify]
WARNING: VerifyWarning: HDU 2: [astropy.io.fits.verify]
WARNING: VerifyWarning:     'PCOUNT' card at the wrong place (card 6).  Fixed by moving it to the right place (card 5). [astropy.io.fits.verify]
WARNING: VerifyWarning:     'GCOUNT' card at the wrong place (card 7).  Fixed by moving it to the right place (card 6). [astropy.io.fits.verify]
zonca commented 5 years ago

Found another issue in all files:

DATAMODL= ' IRISImageModel'    /Type of data model

the initial space prevents stpipe from identifying the model correctly. After that is fixed, the pipeline works fine (without dark, I'm working on that next)

No need to regenerate the files, @nils-erik can you please just fix this for next time and confirm here so I can close the issue?

zonca commented 4 years ago

@nils-erik is this issue with the space at the beginning of DATAMODL fixed?

nils-erik commented 4 years ago

Should be fixed, check '/data/group/data/iris/dev/iris_cal_files/example_fits/' for the images.

On Mon, Aug 12, 2019 at 2:08 PM Andrea Zonca notifications@github.com wrote:

@nils-erik https://github.com/nils-erik is this issue with the space at the beginning of DATAMODL fixed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/8?email_source=notifications&email_token=AJPG6ON3K7EOFIFCNXZLLSTQEHGMHA5CNFSM4HQXW372YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4D2U7I#issuecomment-520596093, or mute the thread https://github.com/notifications/unsubscribe-auth/AJPG6OPWMJ6KY5DVONGNJALQEHGMHANCNFSM4HQXW37Q .

zonca commented 4 years ago

@nils-erik that folder looks empty, can you generate the files again?

nils-erik commented 4 years ago

Should be there now.

Nils

On Wed, Aug 21, 2019 at 4:14 PM Andrea Zonca notifications@github.com wrote:

@nils-erik https://github.com/nils-erik that folder looks empty, can you generate the files again?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oirlab/iris_pipeline/issues/8?email_source=notifications&email_token=AJPG6OOHAIWREO7ACZQFDCLQFXD33A5CNFSM4HQXW372YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD43MU3I#issuecomment-523684461, or mute the thread https://github.com/notifications/unsubscribe-auth/AJPG6OPZ2YJA6S3GVRKKAW3QFXD33ANCNFSM4HQXW37Q .

zonca commented 4 years ago

yes! they work out of the box, thanks!