Processing iTelescope images fails when telescope metadata is missing or incorrect

[DaveStrickland]

Processing of iTelescope images from T32 and T09, in particular, fails in ap_find_stars.py:

2021-01-16 19:41:12,069 | __main__ | DEBUG | FITS keywords found in image: {'IMG_FILE': ('calibrated-T09-davestrickland-NGC_253-20200813-023100-Red-BIN1-W-060-001.fit', 'Name of image file searched for stars'), 'IMG_COLS': (4096, 'Number of columns in input image'), 'IMG_ROWS': (4096, 'Number of rows in input image'), 'AP_NDET': (2049, 'Number of sources detected in the image.'), 'AP_NPHOT': (200, 'Number of sources final photometry.'), 'AP_NFIT': (25, 'Number of sources used in FWHM fitting.'), 'AP_NSIGM': (7.0, 'Source searching threshold (sigma above background)'), 'EXPOSURE': (60.0, '[seconds] Image exposure time'), 'DATE-OBS': ('2020-08-13T16:31:27', 'Observation date and time'), 'OBJECT': ('NGC 253', 'Target object'), 'TELESCOP': ('iTelescope 9', 'Telescope used'), 'INSTRUME': ('FLI', 'Detector used'), 'CCD-TEMP': (-25.0, 'CCD temperature at start of exposure in C'), 'APTDIA': (90.0, '[mm] Diameter of telescope aperture'), 'RA': ('00 47 33.10', 'Target right ascension'), 'DEC': ('-25 17 15.0', 'Target declination'), 'XPIXSZ': (9.0, '[micrometers] X-axis pixel scale after binning'), 'YPIXSZ': (9.0, '[micrometers] Y-axis pixel scale after binning'), 'FOCALLEN': (417.0, '[mm] Stated telescope focal length'), 'FILTER': ('Red', 'Filter used'), 'LAT-OBS': (-31.2736111111, '[deg +N WGS84] Observatory Geodetic latitude'), 'LONG-OBS': (149.064444444, '[deg +E WGS84] Observatory Geodetic longitude'), 'ALT-OBS': (1165.0, '[metres] Observatort altitude above mean sea level'), 'AIRMASS': (1.02503407597, 'Airmass (multiple of zenithal airmass)')}
2021-01-16 19:41:12,069 | __main__ | DEBUG | FITS keywords missing from image: ['OBJNAME', 'RA-OBJ', 'DEC-OBJ', 'EGAIN']
2021-01-16 19:41:12,202 | __main__ | INFO | Approximate coordinates: ra=0.792528 hours, dec=-25.287500 deg
2021-01-16 19:41:12,202 | __main__ | INFO | Approximate image field of view is 7.163 degrees across.
2021-01-16 19:41:12,202 | __main__ | INFO | Approximate pixel sizes (arcseconds) are x=4.452, y=4.452
2021-01-16 19:41:12,202 | __main__ | CRITICAL | Shutting down due to fatal error
Traceback (most recent call last):
  File "/Users/dks/git/AstroPhotography/src/AstroPhotography/scripts/ap_find_stars.py", line 2180, in <module>
    status = main()
  File "/Users/dks/git/AstroPhotography/src/AstroPhotography/scripts/ap_find_stars.py", line 2161, in main
    find_stars.write_quality_report(p_qual_rprt)
  File "/Users/dks/git/AstroPhotography/src/AstroPhotography/scripts/ap_find_stars.py", line 1039, in write_quality_report
    val = self._kw_dict[fkw][0] 
KeyError: 'EGAIN'

T32 and T09 images from Q3 2020 lack gain information in either the raw or calibrated FITS files, which is present in T05 images using the EGAIN keyword. T09 raw/calibrated FITS files have an incorrect focal length FOCALLEN and APTDIA keywords:

FOCALLEN=   417.00000000000000 /Focal length of telescope in mm                 
APTDIA  =   90.000000000000000 /Aperture diameter of telescope in mm 

Expected values are 680mm and 127mm, see here.

The lack of a gain keyword causes ap_find_stars.py to crash when generating a quality report. Working around the lack of the gain keyword when generating the quality report is one option, but is non-optimal given other processing requires knowing the gain correctly (e.g. cosmic ray correction).

The incorrect focal length for T09 leads to an incorrect estimated pixel size, approximately 70% larger than expected. This would likely cause ap_astrometry.py to fail as the current method limits the solution to be within +/-30% of the approximate pixel size.

Here the keywords are present, but incorrect. The software would need some mechanism of applying corrected FITS header metadata to the raw files and or during calibration.

[DaveStrickland]

We have two alternatives:

1. The user must know what keywords are to be updated, which are to be added, and modifies the raw files using a tool similar to heasoft's fmodhead. It would be up to them to automate the process for all files and all necessary keywords, e.g. with a shell script. Conceptually simple, but more work in the long run, and not very robust.
2. We create a system where AstroPhotography can identify the telescope automatically, and via a hard-wired table or user-supplied configuration file it will apply the corrections, presumably directly to the raw files, if it detects the need to do so. More complicated but a more powerful solution, but only as good as the configuration it uses.

Other considerations: Raw iTelescope files lack the TELESCOP FITS keyword, it is added in their calibrated images by whatever calibration software they use. AstroPhotography already has a tool, ap_add_metadata.py/ApAddMetadata, that adds keywords and detects the iTelescope telescope used based on the file name using a hard-wired internal dictionary. Currently that must be run only on self-calibrated files before performing star detection and astrometry, but it could easily be moved to being run on all files and run before calibration.

Modifying this, with maybe a longer term goal of moving the configuration out of the code into a file and generalizing it for user-defined telescope names, seems like a good place to start.

[DaveStrickland]

Issue #2 opened for end-to-end processing scripts that have a simple configuration mechanism. This could be used to also specify FITS keywords to override/add.

[DaveStrickland]

Need to check if this is still an issue when issue #2 merged.