emolter
commented
1 year ago Thanks so much, Mike! Let's talk about this at our meeting next week. I was planning to do the majority of this work myself, and I already have the bare-bones implemented. This detailed description of the format gives me a ton of help understanding what else is needed, not only with writing the code but also for documentation.
I know you're busy, but if you'd like to contribute something that isn't Python, you might consider taking the lead on the documentation of this file format as the "industry standard" for solar system data with a backplane.
Topic: Format of navigated FITS image files. I am new to Pylanetary. I assume the navigation step is like what I do in my IDL Simnav code: add metadata and backplanes. There may also be some manipulation of the image data itself, such as rotating north up or resampling to a finer equivalent plate scale.
Motivation: I have tons of processed image data and I would love for the files to be compatible with Pylanetary. I also think my format has some nice features that help the usability of the data. I would be happy if the Pylanetary format had a superset of my IDL Simnav keywords, and I would still be happy if the keywords sometimes had different names. This is not just for HST/WFC3 data obviously. As long as there was some kind of keyword like PYLANVER that gave the version of the Pylanetary format, I could include translator code to handle any differences (until I migrate 100% to Pylanetary).
Backplanes: I copied HST image data. Different instruments and levels of data processing/products from the HST pipeline have different FITS structures, but the one I adapted for NAV files resembles HST/WFC3 "FLT" files: it has extension 0 (primary HDU) with no data and only metadata, followed by as many extensions as needed, with minimal metadata in those subsequent extensions. For example, a navigated Uranus image looks like this:
Metadata: My principle has been to keep all the existing telescope/pipeline metadata, and add some keywords related to navigation. These keywords then can help with further science analysis, reprojection of the image data (cylindrical or polar maps), calibration to I/F reflectance units, etc.
Here are some of the most important keywords in my opion, but this is a draft list and hopefully there is a chance later on to refine the list. These are only Simnav-added keywords. Many telescope keywords (e.g., DATE-OBS and TIME-OBS) are equally important but assumed to not be Pylanetary's responsibility:
These always come in handy, and it's nice to be able to make tables of various things (usually longitude) without having to keep querying Horizons.
These are the main navigation keywords. Most of the first ones are related to the planet center relative to the image, or relative to the RA/DEC it's supposed to be at (which can also be found in telescope header keywords). The section mentions "human operator" because in my experience, automated methods are good but end up being biased by artifacts in individual frames. In Simnav, the human operator sets the final planet center, but the uncertainty estimate in the navigation is the rms offset between the human operator solution and the solutions from two automated algorithms.
In some cases, we made NAV files that are cropped with respect to the original image files, to remove blank space around the planet. This is fine, but it can cause issues trying to reconstruct what happened if you only have the final image. So the CROP_X0 and CROP_Y0 keywords are used to relate properties in the final image to properties in the input image to Simnav. With these keywords (not present in earlier versions of Simnav), recalibrated data can be renavigated without additional human interaction.
I like NaN for a bad data flag, but this keyword always annoys FITS format checkers. So maybe the value should be 'NaN' (ascii). In any case, NaN lets you take averages and such only over good data using Numpy or IDL. Another bad data flag could be used, and listed here, and then codes could use that instead of NaN to do smarter averaging.
The other keywords are for magnification. I find it useful for some smaller targets to enlarge the image. These keywords track that process and ensure that the plate scale in the final product is known to the user (in case they try to reconstruct it from the telescope pipeline metadata and find inconsistencies).
These photometry keywords are not well organized in the header. They apply to the SCI (science data) extension, which for drizzled WFC3 data has units of e-/sec. the PHOTFLAM is from the STScI pipeline, and multiplying this factor by the image array data in e-/sec converts the image to FLAM units of erg/s/cm^2/Angstrom.
For planetary work in the UV/vis/near-IR regime, we want reflectivity in I/F units, which we can get by PHOTIF*SCI. Similarly the SIG_PHOT gives the uncertainty, which includes uncertainty in the solar spectrum.
I made a Google Drive folder PYLAN_FITS_format with example Uranus NAV and REG format FITS files. REG is a cylindrical projection map format, but we can discuss that in a separate issue. Also included is a Python script from Troy that converts NAV to REG, although it is not fully debugged, so the REG file here was generated in IDL.