search

gammapy / gamma-cat

An open data collection and source catalog for gamma-ray astronomy
https://gamma-cat.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 17 forks source link

Better gamma-cat table output format #101

Open cdeil opened 7 years ago

cdeil commented 7 years ago

I'm currently fixing a few issues with the script the makes the gamma-cat FITS table, and the class in Gammapy that reads it.

There is a major issue / question how to support the different spectral models that we have.

These are the columns we have now: https://gist.github.com/cdeil/71d7c18a9ad3f6d71e9882284132ace9

This works for PowerLaw and ExponentialCutoffPowerLaw, but it doesn't work for PowerLaw2 where the norm is given as an integral flux with units cm-2 s-1 instead of cm-2 s-1 TeV-1. I plan to fix this tomorrow and introduce new or better columns for that.

@woodmd - You mentioned that you have a format in Fermipy that you use to serialise lists of spectral models into a Table. Do you have an example file? What columns do you use to support PowerLaw (differential), PowerLaw2 (integral) and ECPL ?

woodmd commented 7 years ago

I've uploaded a sample file that illustrates the current serialization format used by fermipy: https://www.dropbox.com/s/oh3lz5o3g1rj8k4/roi.fits?dl=0 Currently there's two parts to the model serialization. There are a set of vector columns in the CATALOG table that are provided as a convenience for the user called param_names, param_values, and param_errors. These are effectively variable length arrays with one entry per spectral parameter (with up to 10 parameters allowed by the length of the vector). Unfilled values are designated by setting param_names to an empty string. It's perhaps not the most elegant solution but it seemed like the easiest way to accomodate a variety of models with different numbers of parameters and parameter names.

In addition to these we have SOURCES and PARAMS tables which serialize the full contents of the XML model file. These tables have one row per source component and one row per parameter respectively. The mapping of parameters to source and parameter group (eg. spectral or spatial) is done with the source_name and group columns.

Let me know if you have any other questions about these formats. The convention for serializing XML models to FITS tables is probably another area where we might want to develop a specification in the gamma-astro-formats repo.

cdeil commented 7 years ago

This is the HDUs and columns of @woodmd's example file: https://gist.github.com/cdeil/d80329334e0085f8959c37b34181e718

@woodmd - In general, I do think it would be very valuable to write down a spec for the XML and this compact model serialisation format (see https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/issues/41).

Concretely for gamma-cat, I'm not sure it's appropriate because it's a "user-facing" catalog and probably this general format with the vector columns is too abstract for end-users.

Just in case we do want to start using it for gamma-cat or Gammapy: do you have code in Fermipy already to read / write models in this format?

cdeil commented 7 years ago

For now, I've introduced completely separate columns for the spectral models that we support: https://github.com/gammapy/gamma-cat/commit/2c0c279e04d55be4fbe6a9222998504188cb542c It makes the code that reads and writes the catalog (and instantiates model objects from the parameters) dead simple (at the cost of lots of nan values in the table). I'm not sure if this is the long-term solution, because the number of columns grows with the number of spectral models we support. Keeping this issue open for now ...