Invited contribution in the "Universe" journal on data formats for gamma-ray astronomy

[cosimoNigro]

Dear All,

@micheledoro contacted me as special editor of this issue of the "Universe" journal providing a retrospective on gamma-ray astronomy.

Michele would like to add a contribution in this issue describing the evolution of the way of using data in gamma-ray astronomy (especially at VHE) and the progress towards the standardisation of the data format.

I would like to ask if the contributors to this forum would be interested in writing this paper. First of all, I don't have a clear retrospective on the entire evolution of the data format and there are more senior contributors that can for sure provide it. Secondly, we can add another reference for the GADF, this time on a journal (so far we have only proceedings - I think). Although the deadline for submission is expired (if you check the page of the special issue), we would have an extension granted until the end of June (~3 months from now).

I thought about expanding what me and @TarekHC already presented at the ADASS XXX, with a more historical perspective. We signed that proceeding "on behalf of the Data Formats for Gamma-ray Astronomy forum contributors". Maybe this time, since this is a journal paper, we can add all the GADF contributors to the author list and give some official recognition to all people devoting time to this.

Let me know if you are interested, I think that in order to make it we need to put together a team of at least 3-4 people actively writing. If we do not manage to gather them I would just decline Michele's invitation.

Best, Cosimo

[kosack]

I'd be happy to help out. The original version of the format came from the internal note I wrote for CTA back in 2011, so I can perhaps help with some historical context. I don't have an enormous amount of time, unfortunately due to many other projects right now, however!

[lmohrmann]

While I'd love to see a paper that provides recognition of the work done on the data format, I cannot at this point commit to contribute to writing in a significant manner. I'd be happy to provide comments / suggestions once a draft exists, though.

Do I understand correctly that the historical perspective would go beyond the GADF data format? It might make sense to reach out to some (more senior) members of the relevant experiments in that case...?

[TarekHC]

Hi Cosimo,

This sounds like a good idea! I will be able to contribute if you need me (review, write specific sections, participate in discussions), although as others here I doubt I will have the time to really lead any effort.

Although I must say we don't really represent the main roles in the "data formats for gamma-ray astronomy" field. I would guess having MeV->GeV data format experts would probably make more sense, no? LAT data products have been used way more extensively than the one defined here.

But if the article wants to focus on the VHE regime, I would guess having @kosack and @cboisson helping out with the historical view and the ADASS proceeding we can put together something acceptable without too much effort...

[micheledoro]

Hi guys, let me add a few cents. Here is the project page: https://www.mdpi.com/journal/universe/special_issues/gamma-ray_astronomy. Our idea is to gather personal perspective on what happened in a certain field in the past 30 years, what went right, what went wrong, what challenges were overcome, ...

We have contribution about science and technology related to IACTs. They will not be full review, but as a I said, more of personal view.

I was thinking that it would be interesting to add a contribution that included also 'the way we think and treat our data'. I mean: now we are talking about common format, we are using shared scripts and methods, we are building combined likelihoods in a multi-w and multi-m context. 3 decades ago it was not so. It would be interesting to reason about what changed, what are we gaining, what did we lose etc. I contacted @cosimoNigro and I believe that this could be a good project for you experts on this.

[cosimoNigro]

Hello,

with some delay, and with my apologies, find at this link a draft for the review on the "Evolution of data formats in VHE gamma-ray astronomy". https://www.overleaf.com/1422738895ycdqhrmqfyfw I have expanded and re-formulated the content of the proceeding me and @TarekHC presented at the last ADASS. The general idea would be to give in section 2 a bit of a historical perspective, describing how was the data reduction done in this generation of IACTs (I tried to summarise it from the few proceedings I have found) and how VHE gamma-ray astronomers came to develop the GADF format. @kosack it would be nice if you could fill-in with some more technical details explaining how the data specifications were designed, I left an empty subsection for you. Section 3 instead reviews the project already using the GADF: the H.E.S.S. DL3 DR1; the joint-crab; the analysis of the DR1 with ctools and the validations of ctools and gammapy on the same data release. I think they make a clear point in showing the success of the standardisation effort. I left an empty section, the 4th, to describe the possible evolution of the format (extension to other telescopes etc...).

As for @TarekHC and @lmohrmann concerns, the Universe issue that will include this review is focused on ground-based gamma-ray astronomy. So I think that @kosack, @cboisson, @bkhelifi, @registerrier can provide all the historical background we need.

Anyone is welcome to comment, edit and propose changes in the overleaf and to add their name and affiliation in the author list. Let us continue the discussion there.

[cosimoNigro]

Hello, thanks to @TarekHC and @LauraOlivera we completed the issue which is now ready for submission. We would appreciate any comment on this latest version. You can address them to the three of us in private - for minor comments you can just use the overleaf function. Since we are quite close to the deadline for submission we would like to receive your suggestions within early next week (let us say by Wednesday 8 Sep).

Thanks again to those who reviewed the previous versions. Note that in the overleaf preview figure 1 has empty labels: it seems to be a visualisation problem, the figure is correctly displayed in the PDF version.

Thanks.

[cosimoNigro]

Apologies for the delay in reporting it, but the paper was actually published https://www.mdpi.com/2218-1997/7/10/374

Could it we add it as a reference in the README.md?

let me know, so we can close this issue.