Standardize header metadata.

[DinoBektesevic]

This is a first attempt at taking the various varied FITS headers and standardizing a select number of header keywords that we can use in our header metadata DB table.

The header keywords we standardize on here will set our table schema and our query tool interface but there are some outstanding issues with standardizing the WCSs. If we can not easily separate WCS data into a standardized set of columns querying that table will be much harder but it should still be possible to save a subset of keywords (center pixel values and coordinates) and the whole WCS as a pickled blob.

This would not be optimal, as optimal as storing all of these values separately, I suspect. Ergo, this draft PR to give some insight into what the difficulties are.

[mrawls]

Metadata still missing: band or filter, exposure duration (grab independently instead of doing end - start if feasible), some kind of processed/reduced flag (probably either "yes" or "unknown")

[DinoBektesevic]

Metadata still missing: band or filter, exposure duration (grab independently instead of doing end - start if feasible), some kind of processed/reduced flag (probably either "yes" or "unknown")

Added filter and exposure time for astro_metadata_translator recognized instruments.

Added test dataset to get an overview of where functionality is right now. Coverage right now is about 50/50 of the dataset you compiled. Images in the dataset have been cropped for size but the WCS values have been shifted to maintain positional accuracy (approximately).

I started gnawing at getting more instruments supported today but nothing that I would like to commit quite yet. I think our discussion was on-point and that whether we want or no we will have to create a HeaderStandardizer and a ImageStandardizer classes that do the atomic work on a single header and image and then see if FitsProcessor can be made to support both single and multi-extension fits files. If not a bit of a redesign of the FitsProcessor into a SingleExtensionFitsProcessor and a MultiExtensionFitsProcessor might be needed so I don't want to promote this into a full PR quite yet.

[DinoBektesevic]

Ok, this is now getting much better. I think someone may take a look at what is in this code and not feel offended.

Added header standardizer classes, essentially maps between what we want in our DB and what we can find in particular instrument's header

Processors are essentially recipes of how particular file should be processed. For example, a compressed archive needs to be uncompressed, "unarchived" then each file needs to be processed; but is this particular file Rubin calexp - then the image is only in 1st HDU - or is it a DECam NOAO image - then all HDUs have headers and data - is it an SDSS....

To do:

• rework the DB schema to be able to accept focal planes.
  • Different pipelines produce different types of FITS files. Some store all CCDs from a focal plane in the same FITS, some split them up. The fake DB model right now accepts everything as a single CCD. We either unravel multiext fits into individual CCDs on a generic basis or we implement a better schema.
  • DB schema could also use a lot of work in terms of recording uploading times, ips, etc...
• Fix the astro_metadata_translator issues with Header https://github.com/lsst/astro_metadata_translator/pull/53
• Fix science validity of the parsed headers.
  • As-is we are not parsing MOA-II fits file WCS data correctly and it's unclear how to. Perhaps this is a good test case for Astrometry.NET integration. Other headers insert data into the DB, but timezones and other details could be an issue.
  • Fix timezone-naive Time.iso output.
    Format for the time-zone aware datetime objects seem to be a bit different and Astropy does not seem to return timezone encoded strings.
  • Standardize database schema values, as is instruments, telescopes, filters etc all kind of return something different. For certain things we can guarantee consistency (f.e. what is in telescope, what in instrument etc. for others, i.e. filter or science-program, we can do our best to be as close to something standard but really can not be consistent). Perhaps make these fields something like "identifier" and then adopt the instruments conventional naming scheme (f.e. run-calcol-filter-field for SDSS or run-field-filter-chip for MOA-II etc.)
  • This is literally an easy free-for-all problem, it just takes emailing people in cases like MOA-II or, for bigger instruments, reading documentation. Las Cumbres standardizer even has a comment to paper with the lookup table.
• Figure out what I want to do with the dataclasses
  • they are very helpful when debugging etc and it is nice when you don't have to traverse highly nested dicts, but their function would ideally be handled by the model classes. Unfortunately the schema is not very reflective of processing steps and will complain for mandatory values for example.
  • no tests for now, until we decide we like or don't like them
• Add a register method to processors and standardizers?
  • gets rid of the import funkery that has to happen right now.
• Add a (manual, or automatic) priority levels to processors/standardizers
  • or remember procs./stand. that volunteered and try to reprocess on fail....
• better naming than process_upload.upload_processor.process etc...
  • help appreciated, I am not imaginative
• Consider getting rid of from* and get* methods on these classes and make instantiation do that bit of work too (makes it prettier?)
• Perhaps a mixin class to standardize all these similar method names and reduce amount of code duplication.
• flakify
• flesh out the tests
  • as is we essentially just run the entire pipeline end-to-end so we excercise the code, but we should really make it more atomic
• fix how gallery works before this is merged.

[DinoBektesevic]

Ok, fixed the last comments and added tests for Models.

The remainder of issues here were punted other issues in which we will tackle them. Mainly all is well except: we don't have good error handling, data update/replace/remove logic and MOA-II WCS is a placeholder pending error handling, or getting Astrometry.net solver working.

Looking for any last comments?

[mrawls]

Understood. Let's get it merged so we can find fun new ways to break things 😄