Unable to open FITS files when certain `GbmHeader` keywords aren't in the corresponding header

[derekocallaghan]

I've noticed that certain triggers/burst FITS files don't contain all GbmHeader subclass keywords, preventing the subclass from opening the file. This can be illustrated with Tcat.open() and a local copy of glg_tcat_all_bn180112842_v00.fit, but it also happens in other situations e.g. GbmTte.open():

In [1]: from astropy.io import fits
   ...: from gdt.missions.fermi.gbm.headers import TcatHeader
   ...: from gdt.missions.fermi.gbm.tcat import Tcat

In [2]: tcat_file = "./glg_tcat_all_bn180112842_v00.fit"

In [3]: Tcat.open(tcat_file)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 Tcat.open(tcat_file)

File ~/gitrepos/gifts-6u/gdt-fermi/src/gdt/missions/fermi/gbm/tcat.py:88, in Tcat.open(cls, file_path, **kwargs)
     86 obj = super().open(file_path, **kwargs)
     87 hdrs = [hdu.header for hdu in obj.hdulist]
---> 88 obj._headers = TcatHeaders.from_headers(hdrs)        
     89 return obj

File ~/gitrepos/gifts-6u/gdt-core/src/gdt/core/headers.py:192, in FileHeaders.from_headers(cls, headers)
    190             hidx += 1
    191         else:
--> 192             obj[i][key] = headers[i][key]
    194 return obj

File ~/anaconda3/envs/gifts/lib/python3.10/site-packages/astropy/io/fits/header.py:170, in Header.__getitem__(self, key)
    167 else:
    168     keyword = key
--> 170 card = self._cards[self._cardindex(key)]
    172 if card.field_specifier is not None and keyword == card.rawkeyword:
    173     # This is RVKC; if only the top-level keyword was specified return
    174     # the raw value, not the parsed out float value
    175     return card.rawvalue

File ~/anaconda3/envs/gifts/lib/python3.10/site-packages/astropy/io/fits/header.py:1771, in Header._cardindex(self, key)
   1768         indices = self._rvkc_indices.get(keyword, None)
   1770 if not indices:
-> 1771     raise KeyError(f"Keyword {keyword!r} not found.")
   1773 try:
   1774     return indices[n]

KeyError: "Keyword 'INFILE01' not found."

The FITS file can be opened directly, where it can be seen that INFILE01 isn't in the header:

In [5]: sorted(TcatHeader().keys())
Out[5]: 
['ADC_HI',
 'ADC_LO',
 'CHAN_HI',
 'CHAN_LO',
 'CLASS',
 'CREATOR',
 'DATE',
 'DATE-END',
 'DATE-OBS',
 'DEC_OBJ',
 'DEC_SCX',
 'DEC_SCZ',
 'DET_MASK',
 'EQUINOX',
 'ERR_RAD',
 'FILENAME',
 'FILETYPE',
 'GCN_FLAG',
 'GEO_LAT',
 'GEO_LONG',
 'HISTORY',
 'HISTORY',
 'HISTORY',
 'HISTORY',
 'INFILE01',
 'INSTRUME',
 'LOC_ENRG',
 'LOC_SRC',
 'LOC_VER',
 'MJDREFF',
 'MJDREFI',
 'OBJECT',
 'OBJ_CLAS',
 'OBSERVER',
 'ORIGIN',
 'PHI',
 'RADECSYS',
 'RA_OBJ',
 'RA_SCX',
 'RA_SCZ',
 'RELIABLT',
 'TELESCOP',
 'THETA',
 'TIMESYS',
 'TIMEUNIT',
 'TRIGSCAL',
 'TRIGTIME',
 'TRIG_ALG',
 'TRIG_SIG',
 'TSTART',
 'TSTOP']

In [7]: tcatf = fits.open("./glg_tcat_all_bn180112842_v00.fit")

In [8]: 'INFILE01' in tcatf["PRIMARY"].header.keys()
Out[8]: False

In [9]: tcatf["PRIMARY"].header
Out[9]: 
SIMPLE  =                    T / file does conform to FITS standard             
BITPIX  =                    8 / number of bits per data pixel                  
NAXIS   =                    0 / number of data axes                            
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
CREATOR = 'DOL+GBM_TRIGDAT_Reader v1.28' / Software and version creating file   
FILETYPE= 'TRIGGER ENTRY'      / Name for this type of FITS file                
TELESCOP= 'GLAST   '           / Name of mission/satellite                      
INSTRUME= 'GBM     '           / Specific instrument used for observation       
OBSERVER= 'Meegan  '           / GLAST Burst Monitor P.I.                       
ORIGIN  = 'GIOC    '           / Name of organization making file               
DATE    = '2018-01-12T20:22:25' / file creation date (YYYY-MM-DDThh:mm:ss UT)   
DATE-OBS= '2018-01-12T20:10:14' / Date of start of observation                  
DATE-END= '2018-01-12T20:20:28' / Date of end of observation                    
TIMESYS = 'TT      '           / Time system used in time keywords              
TIMEUNIT= 's       '           / Time since MJDREF, used in TSTART and TSTOP    
MJDREFI =                51910 / MJD of GLAST reference epoch, integer part     
MJDREFF = 7.428703703703703D-4 / MJD of GLAST reference epoch, fractional part  
TSTART  =     537480614.201896 / Time of first TRIGDAT INFO packet              
TSTOP   =     537481228.613448 / Time of last TRIGDAT INFO packet               
FILENAME= 'glg_tcat_all_bn180112842_v00.fit' / Name of this file                
TRIGTIME=     537480753.340436 / Trigger time relative to MJDREF, double precisi
OBJECT  = 'GRB180112842'       / Burst name in standard format, yymmddfff       
RADECSYS= 'FK5     '           / Stellar reference frame                        
EQUINOX =               2000.0 / Equinox for RA and Dec                         
RA_OBJ  =             173.9200 / Calculated RA of burst                         
DEC_OBJ =              23.0600 / Calculated Dec of burst                        
ERR_RAD =               6.6700 / Calculated Location Error Radius               
TRIG_SIG=                  4.9 / Trigger significance (sigma)                   
LOC_SRC = 'Fermi, GBM'         / Mission/Instrument providing the localization. 
LOC_VER = '3       '           / Version string for localization software.      
LOC_ENRG= '(50.0,300.0)'       / [keV] Energy range used for localization.      
GEO_LONG=              45.6500 / [deg] Spacecraft geographical east longitude   
GEO_LAT =              25.4833 / [deg] Spacecraft geographical north latitude   
RA_SCX  =             298.1209 / [deg] Pointing of spacecraft x-axis: RA        
DEC_SCX =              13.4377 / [deg] Pointing of spacecraft x-axis: Dec       
RA_SCZ  =              94.5567 / [deg] Pointing of spacecraft z-axis: RA        
DEC_SCZ =              75.3901 / [deg] Pointing of spacecraft z-axis: Dec       
THETA   =              65.0000 / [deg] Angle from spacecraft z-axis (zenith)    
PHI     =             243.0000 / [deg] Angle from spacecraft +x-axis toward +y  
CLASS   = 'GRB     '           / Classification of trigger.                     
OBJ_CLAS= 'GRB     '           / GBM internal classification.                   
RELIABLT=               0.9529 / Reliability of classification.                 
TRIGSCAL=                  256 / [ms] Triggered timescale                       
TRIG_ALG=                    9 / Triggered algorithm number                     
CHAN_LO =                    3 / Trigger channel: low                           
CHAN_HI =                    4 / Trigger channel: high                          
ADC_LO  =                  259 / Trigger channel: low (ADC: 0 - 4095)           
ADC_HI  =                 1352 / Trigger channel: high (ADC: 0 - 4095)          
DET_MASK= '00000011000000'     / Triggered detectors: (0-13)                    
GCN_FLAG= 'No      '           / Was this file used for auto GCN Notice?        
CHECKSUM= 'ZEmKc9jHZCjHb9jH'   / HDU checksum updated 2018-01-12T20:22:25       
DATASUM = '         0'         / data unit checksum updated 2018-01-12T20:22:25 

I'm currently using a local gdt-core workaround as follows, where any expected keywords missing from the header are ignored. Although I can create a PR with this modification, I'm unsure whether you always want to reject any FITS files with missing keywords, so I've held off for now and am logging this issue instead.

diff --git a/src/gdt/core/headers.py b/src/gdt/core/headers.py
index 65e39a4..37c61e3 100644
--- a/src/gdt/core/headers.py
+++ b/src/gdt/core/headers.py
@@ -180,7 +180,8 @@ class FileHeaders():
         for i in range(num_headers):
             cidx = 0
             hidx = 0
-            for key in obj[i].keys():
+            #for key in obj[i].keys():
+            for key in [k for k in obj[i].keys() if k in headers[i]]:
                 if (key == 'COMMENT'):
                     obj[i][key][cidx] = headers[i][key][cidx]
                     cidx += 1

[derekocallaghan]

Hi @AdamGoldstein-USRA,

Just wanted to check whether the suggested fix/workaround is okay with you? If so, I'll create a PR.

Thanks

[derekocallaghan]

I've confirmed that this issue has been fixed with this recent gdt-core PR that displays a warning for missing headers: https://github.com/USRA-STI/gdt-core/pull/25

In [1]: from gdt.missions.fermi.gbm.tcat import Tcat

In [2]: tcat = Tcat.open("./glg_tcat_all_bn180112842_v00.fit")
/home/derek/gitrepos/gifts-6u/gdt-fermi/src/gdt/missions/fermi/gbm/tcat.py:88: RuntimeWarning: INFILE01 not found in header PRIMARY
  obj._headers = TcatHeaders.from_headers(hdrs)
/home/derek/gitrepos/gifts-6u/gdt-fermi/src/gdt/missions/fermi/gbm/tcat.py:88: RuntimeWarning: HISTORY not found in header PRIMARY
  obj._headers = TcatHeaders.from_headers(hdrs)

In [3]: tcat.headers[0]
Out[3]: WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]

CREATOR = 'DOL+GBM_TRIGDAT_Reader v1.28' / Software and version creating file   
FILETYPE= 'TRIGGER ENTRY'      / Name for this type of FITS file                
TELESCOP= 'GLAST   '           / Name of mission/satellite                      
INSTRUME= 'GBM     '           / Specific instrument used for observation       
OBSERVER= 'Meegan  '           / GLAST Burst Monitor P.I.                       
ORIGIN  = 'GIOC    '           / Name of organization making file               
DATE    = '2018-01-12T20:22:25' / file creation date (YYYY-MM-DDThh:mm:ss UT)   
DATE-OBS= '2018-01-12 20:11:18.386' / Date of start of observation              
DATE-END= '2018-01-12 20:21:32.797' / Date of end of observation                
TIMESYS = 'TT      '           / Time system used in time keywords              
TIMEUNIT= 's       '           / Time since MJDREF, used in TSTART and TSTOP    
MJDREFI =                51910 / MJD of GLAST reference epoch, integer part     
MJDREFF = '0.0007428703703703703' / MJD of GLAST reference epoch, fractional par
TSTART  =     537480614.201896 / [GLAST MET] Observation start time             
TSTOP   =     537481228.613448 / [GLAST MET] Observation stop time              
FILENAME= 'glg_tcat_all_bn180112842_v00.fit' / Name of this file                
TRIGTIME=     537480753.340436 / Trigger time relative to MJDREF, double precisi
OBJECT  = 'GRB180112842'       / Burst name in standard format, yymmddfff       
RADECSYS= 'FK5     '           / Stellar reference frame                        
EQUINOX =               2000.0 / Equinox for RA and Dec                         
RA_OBJ  =               173.92 / Calculated RA of burst                         
DEC_OBJ =                23.06 / Calculated Dec of burst                        
ERR_RAD =                 6.67 / Calculated Location Error Radius               
THETA   =                 65.0 / [deg] Angle from spacecraft zenith             
PHI     =                243.0 / [deg] Angle from spacecraft +X axis toward +Y  
LOC_SRC = 'Fermi, GBM'         / Mission/Instrument providing the localization  
CLASS   = 'GRB     '           / Classification of trigger                      
OBJ_CLAS= 'GRB     '           / Classification of trigger                      
TRIGSCAL=                  256 / [ms] Triggered timescale                       
TRIG_ALG=                    9 / Triggered algorithm number                     
CHAN_LO =                    3 / Trigger channel: low                           
CHAN_HI =                    4 / Trigger channel: high                          
ADC_LO  =                  259 / Trigger channel: low (ADC: 0 - 4095)           
ADC_HI  =                 1352 / Trigger channel: high (ADC: 0 - 4095)          
TRIG_SIG=                  4.9 / Trigger significance (sigma)                   
GEO_LONG=                45.65 / [deg] Spacecraft geographical east longitude   
GEO_LAT =              25.4833 / [deg] Spacecraft geographical north latitude   
DET_MASK= '00000011000000'     / Triggered detectors: (0-13)                    
RA_SCX  =             298.1209 / [deg] Pointing of spacecraft x-axis: RA        
DEC_SCX =              13.4377 / [deg] Pointing of spacecraft x-axis: Dec       
RA_SCZ  =              94.5567 / [deg] Pointing of spacecraft z-axis: RA        
DEC_SCZ =              75.3901 / [deg] Pointing of spacecraft z-axis: Dec       
INFILE01= '' / Level 0 input data file                                          
LOC_VER = '3       '           / Version string of localizing software          
LOC_ENRG= '(50.0,300.0)'       / Energy range used for localization             
RELIABLT=               0.9529 / Reliability of classification                  
GCN_FLAG= 'No      '                                                            
HISTORY ()                                                                      
HISTORY BKG_POLY_ORDER=                                                         
HISTORY BKGINT1 = ()                                                            
HISTORY BKGINT2 = ()