search

gammapy / gamma-cat

An open data collection and source catalog for gamma-ray astronomy
https://gamma-cat.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 17 forks source link

Add data from giuliani/astrisim/gsed #32

Open cdeil opened 7 years ago

cdeil commented 7 years ago

At the CTA consortium meeting last week I talked with Andrea Giuliani and he now mentioned http://www.iasf-milano.inaf.it/~giuliani/astrisim/gsed/ to me via email.

From a brief look, it seems very similar to what we want to do with gamma-cat, only smaller in scope. I didn't look yet what data they have that we haven't collected yet. We should collaborate. Discussing via email for now ...

cdeil commented 7 years ago

Andrea Guiliani (@cento14) and I discussed via email. The plan is that he or I move the astrisim/gsed info we don't have yet over to gammacat this week.

@cento14 - Could you please add your name to https://github.com/cento14 under "settings" so that it's easier for me and others to mention you on issues or find you on Github?

cdeil commented 7 years ago

@cento14 - Could you please have a look here? https://github.com/gammapy/gamma-cat/tree/master/other_cats/astrisim_gsed

I hacked together a script to make list the papers where we have already collected info for each source in your data collection: https://github.com/gammapy/gamma-cat/blob/master/other_cats/astrisim_gsed/astrisim_gsed.py

The result is here: https://github.com/gammapy/gamma-cat/blob/master/other_cats/astrisim_gsed/missing_info.csv

The next step would be to extend the script here to create ECSV and / or YAML files in gamma-cat/input with the info from your FITS tables: https://github.com/gammapy/gamma-cat/blob/master/other_cats/astrisim_gsed/astrisim_gsed.py#L57

Do you have time to do this in the coming days?

If no, I could also write the script and then you run it and commit the files, so that it's clear that it's coming from your data collection.

cento14 commented 7 years ago

@cdeil

ok, if you can write the script, it would be great !

cdeil commented 7 years ago

ok, if you can write the script, it would be great !

Will do tomorrow.

cdeil commented 7 years ago

I'm finally doing this now, sorry for the delay.

I have a few questions:

Looks like you didn't collect spectral model parameter info? I mean something like: https://github.com/gammapy/gamma-cat/blob/master/input/papers/2006/2006A%2526A...456..245A/tev-000065.yaml#L18

For the SED format, what you call FLUX_ERROR_MIN and FLUX_ERROR_MAX is what we call dnde_errn and dnde_errp, right? Example: https://gist.github.com/cdeil/6d49a6e4c32dcff6d7745b2fb226b0c5 See SED spec: http://gamma-astro-data-formats.readthedocs.io/en/latest/results/flux_points/index.html

For now I'll proceed under that assumption using c88ae46 . As far as I can see, you didn't record the info which SED is from which publication? I'll try to do the matching by hand and add it correctly to gamma-cat.

cdeil commented 7 years ago

Apparently the flux in your files is e2dnde in erg cm^-2 s^-1?

I've updated the script to convert to dnde in cm^-2 s^-1 TeV^-1: https://github.com/gammapy/gamma-cat/blob/master/other_cats/astrisim_gsed/astrisim_gsed.py#L82 to be comparable to what we have in gamma-cat for input SEDs.

The script writes the SEDs in ECSV format, and all the SEDs can be seen here: https://github.com/gammapy/gamma-cat/compare/master...cdeil:astrisim-gsed

For the first one I checked, you have similar, but not identical values to what we already have in gamma-cat:

First and last point in astrisim gsed:

e_ref dnde dnde_errn dnde_errp
0.386508 3.14143e-11 8.0112e-12 9.01682e-12
...
15.3386 6.42139e-15 1.43208e-15 1.50278e-15

First and last point in gamma-cat

e_ref dnde dnde_errp dnde_errn
 0.39    3.14e-11    9.00e-12    9.16e-12
...
15.43    6.37e-15    3.00e-15    2.97e-15

The gamma-cat values match the ones from here: https://www.mpi-hd.mpg.de/hfm/HESS/pages/publications/auxiliary/kookaburra_auxinfo.html

@cento14 - Question: How did you collect your SED data? Did you data pick values from plots? Did you add systematic errors?

To make your SEDs usable for gamma-cat, the most valuable thing would be if you added a paper_id pointing to the ADS identifier, and maybe a comment or URL where you got the data from.

Only then would it be really valuable for gamma-cat, i.e. limited effort to copy the information over.

For now I'll wait for a reply and continue with manual data entry in gamma-cat, not using astrisim gsed.

cdeil commented 7 years ago

I decided to continue with this, trying to ingest the SEDs into gamma-cat.

First step is to see which SED files are useful and comparate to what we already have.

I'm doing this here and putting "TODO" for points where I have questions or where data should be added: https://github.com/gammapy/gamma-cat/blob/master/other_cats/astrisim_gsed/README.md

cento14 commented 7 years ago

hi, here are the answers to your last massages,

Fluxes are given in terms of f * E^2 with units [ erg / cm2 s ]

FLUX_ERROR_MAX and FLUX_ERROR_MIN give the width of the asymmetric 1-sigma error bars (positive and negative respectively),
hence the measured flux is : FLUX +FLUX_ERROR_MAX -FLUX_ERROR_MIN

For most of the sources we picked the spectral point from plots found in literature.

For the models, we stored the numerical values of the functions describing the model, namely two columns, energy and flux, with the same units of the data points, but without errors.

For both data and models, the keyword 'PAPER' in the extension header gives the reference where data were taken.

cento14 commented 7 years ago

Here are some answers to questions in the : https://github.com/gammapy/gamma-cat/blob/master/other_cats/astrisim_gsed/README.md

It seems that the gsed.tar file on our web side was not the last version. Some fits files were old and in some cases duplicated. A part from few cases the files have the name of the TeVCat id (for ex. TeVJ1418-609.fits,... )

We have now cleaned everything. The new version fixes most of your questions and it is on line on our web site. If you prefer I can put the updated fits files on the repository with git. for example in : gamma-cat/other_cats/astrisim_gsed/data/

1418-609

(This is now TeVJ1418-609.fits)

We have taken the values from the plot in the paper.

1418-609_HESS_2006 has HESS data, while 1418-609_MODEL_HESS2006 is the model based on HESS data. As such they are both taken from the same paper (Aharonian+2006, that appears as PAPER in the DATA extension and in the MODEL extension)

1420-607

(now TeVJ1420-607.fits )

this file has been changed in the meantime. Now there is no 1420-607_DATI_FERMI-HESS but

1804-217

this file has been changed in the meantime. Now there is no DATI_CANGAROO but

CANGAROO2008 (4rows, CANGAROO data taken from Higashi et al. 2008, ApJ 683, 957, see PAPER keyword of this extension)

MODEL_CANGAROO2008 (300 rows, model taken from Higashi et al. 2008, ApJ 683, 957, see PAPER keyword of this extension)

cdeil commented 7 years ago

@cento14 - Thank you for all the info!!! I will fetch a new version tomorrow or Friday latest and add your SED data to gamma-cat. (unless you have time to make a pull request this week).

For me, the most important info is for every table:

  1. is it measured spectral points? I'm not interested in other data such as "sampled spectrum curves". We put "data_type: sed" in the header for our SEDs. Maybe you could do the same?
  2. which paper? -> have "PAPER" key in every header. Is this done?
  3. which origin? -> sounds like you always digitized Figures from PDF? Maybe could put a comment which Figure / points exactly, but probably obvious given "PAPER".
cdeil commented 7 years ago

@cento14 - I'm trying to update to a new version of your files now.

One thing I noticed is that data/CTB37A.fits from http://www.iasf-milano.inaf.it/~giuliani/astrisim/gsed/gsed.tar seems to be corrupt. I get an error if I try to run ftlist or ftverify on it. Can you check?

cdeil commented 7 years ago

Updates SED data from astrisim is here: https://github.com/gammapy/gamma-cat/compare/master...cdeil:astrisim-gsed

Just poking around a bit, there's still some stuff.

E.g. this one and many others don't have a "PAPER" header key, i.e. it's non-trivial to find out the origin of the data: https://github.com/cdeil/gamma-cat/blob/fbc227ae29364d5ad4033dfbda37cbe19a44c3c4/other_cats/astrisim_gsed/data/RXJ1713_fermi2011.ecsv

And there's still this one that has CANGAROO in the filename, but then points to a HESS paper!? https://github.com/cdeil/gamma-cat/blob/fbc227ae29364d5ad4033dfbda37cbe19a44c3c4/other_cats/astrisim_gsed/data/TeVJ1804-217_MODEL_CANGAROO2008.ecsv

@cento14 - Can you improve / fix?

cento14 commented 7 years ago

@cdeil

is it measured spectral points? I'm not interested in other data such as "sampled spectrum curves".

yes, all the extensions with name not beginning with MODEL or MAP contain measured spectral points.

which paper? -> have "PAPER" key in every header. Is this done?

Most of the files (49 of 55, those with names TeV... ) have the "PAPER" key in every header. The others not yet

data/CTB37A.fits

indeed the version in the tar seems corrupted. We don't understand why, the original fits file is ok. We re-created the tar file. You can also download olny this file from here : http://www.iasf-milano.inaf.it/~giuliani/astrisim/gsed/data/CTB37A.fits

TeV J1804-217 And there's still this one that has CANGAROO in the filename, but then points to a HESS paper!?

The CANGAROO extensions point to Higashi et al. 2008, ApJ 683, 957 that is a CANGAROO paper. The HESS extensions point to Aharonian et al. 2006, ApJ 636, 777 that is a HESS paper.