Generalize to wide-field ground array instruments

[LauraOlivera]

Hello gamma-astro-data-format enthusiasts :grinning:,

The current status of the GADF is very IACT-centric, from the names of the sections (IACT events, IACT IRFs...) to the category of mandatory for some fields and the overall philosophy centered around pointing positions.

This doesn't have to be the case, and the format can be expanded to include wide-field ground-based arrays, such as HAWC (www.hawc-observatory.org/) , LHAASO and the proposed SWGO (www.swgo.org) with a few changes, some bigger than others.

IACT events

The proposed changes to the events section are quite simple. The definition of the GTIs is completely compatible with ground arrays, and the only change in the EVENTS table would be to remove RA_PNT and DEC_PNT from mandatory headers to make them optional headers. Some other optional fields that are commonly used by this type of instruments, like number of tanks triggered, or event position in the array, could be added. But since they are optional, it is less important. The POINTING table is not currently used, according to the text, and it could just be optional as well.

IACT IRFs

Usually, in HAWC, IRFs are produced in sky coordinates (R.A, Dec), so already projected into the sky. This is not the only option, as one can also have IRFs that depend in local coordinates (namely zenith angle range) that then get projected into the sky in a further step. In any case, besides the decoupling from the concept of pointing and spatial offsets, much of what is described in this section describes accurately the IRFs of a wide-field instrument. My suggestion would be the addition to one or two more types of IRFs besides Full-enclosure and Point-like, to include wide-field observations. In this case the spatial axes would be, instead of offsets THETA, either sky coordinates RA, DEC or zenith angle bins ZENITH_LO, ZENITH_HI.

IACT data storage

Only the title is exclusive to IACTs in this section :smile:

These are of course just suggestions, but I think the format would be greatly enriched by them, since at little cost, they allow for a whole class of instruments to be included.

What do you think?

Thanks!

[lmohrmann]

I think this is a great idea :+1:

[TarekHC]

I fully agree with @lmohrmann, this would be an excellent addition to the format! I'm 100% in.

We write "IACT" just everywhere, but that can be easily removed.

Some comments below to help with the implementation and trigger some discussion:

the only change in the EVENTS table would be to remove RA_PNT and DEC_PNT from mandatory headers to make them optional headers

I agree: This is a simple change that could be implemented. Feel free to prepare a pull request with these changes so we can directly comment on them.

Even if some people may think these changes may be out of scope, water detectors would behave similarly as an IACT during drift observations (meaning evolving RA_PNT and DEC_PNT vs time). So DL3 format should also be able to accommodate such an observation scheme.

Although note if we are not fixing RA_PNT/DEC_PNT we need some other way to fix the pointing. Perhaps defining an ALT_PNT and AZ_PNT?

in HAWC, IRFs are produced in sky coordinates (R.A, Dec), so already projected into the sky

I'm curious: how does this make sense? Wouldn't HAWC IRFs be very stable in zenith/azimuth coordinates? Interesting!

My suggestion would be the addition to one or two more types of IRFs besides Full-enclosure and Point-like

Note these variables mainly refer not to the way IRFs are stored, but the way they were calculated: How are IRFs calculated within HAWC? Simulating point sources over different positions in the sky and then optimizing cuts for a point-like source? Or simulating a diffuse/uniform distribution of gammas over the whole sky and not applying any theta cut? I believe creating a third option equivalent to these two would need to be justified from the MC/IRF generation point of view (and not with the format they are stored).

In this case the spatial axes would be, instead of offsets THETA, either sky coordinates RA, DEC or zenith angle bins ZENITH_LO, ZENITH_HI

Question: If we are able to fix the pointing with zenith/azimuth, then wouldn't the current format be enough? If the pointing direction is the zenith (and is fixed, and science tools are able to accommodate to that) THETA would be the zenith distance, so in principle this would be equivalent to bins in zenith. The only difference would be that IACTs cover a significantly smaller range in THETA. But I don't see this requiring a new format, no?

Of course, if you need any other direction, or you really need IRFs with a new format, this would not be a problem at all: they can be easily added (as we are fully aware we will need to add more formats in the future!). But I'm afraid most of these changes will need to be implemented in the science tools, and that is the point that will require more work.

[jknodlseder]

I fully agree that the format should also include non-IACT data.

In the same direction, we should probably check to which extent the Fermi-LAT format is already compatible with the GADF (at least for the IRFs this should be the case since the IRF format came from Fermi-LAT)

Le 1 déc. 2020 à 18:22, LauraOlivera notifications@github.com a écrit :

Hello gamma-astro-data-format enthusiasts 😀,

The current status of the GADF is very IACT-centric, from the names of the sections (IACT events, IACT IRFs...) to the category of mandatory for some fields and the overall philosophy centered around pointing positions.

This doesn't have to be the case, and the format can be expanded to include wide-field ground-based arrays, such as HAWC (www.hawc-observatory.org/ http://www.hawc-observatory.org/) , LHAASO and the proposed SWGO (www.swgo.org http://www.swgo.org/) with a few changes, some bigger than others.

IACT events

The proposed changes to the events section are quite simple. The definition of the GTIs is completely compatible with ground arrays, and the only change in the EVENTS table would be to remove RA_PNT and DEC_PNT from mandatory headers to make them optional headers. Some other optional fields that are commonly used by this type of instruments, like number of tanks triggered, or event position in the array, could be added. But since they are optional, it is less important. The POINTING table is not currently used, according to the text, and it could just be optional as well.

IACT IRFs

Usually, in HAWC, IRFs are produced in sky coordinates (R.A, Dec), so already projected into the sky. This is not the only option, as one can also have IRFs that depend in local coordinates (namely zenith angle range) that then get projected into the sky in a further step. In any case, besides the decoupling from the concept of pointing and spatial offsets, much of what is described in this section describes accurately the IRFs of a wide-field instrument. My suggestion would be the addition to one or two more types of IRFs besides Full-enclosure and Point-like, to include wide-field observations. In this case the spatial axes would be, instead of offsets THETA, either sky coordinates RA, DEC or zenith angle bins ZENITH_LO, ZENITH_HI.

IACT data storage

Only the title is exclusive to IACTs in this section 😄

These are of course just suggestions, but I think the format would be greatly enriched by them, since at little cost, they allow for a whole class of instruments to be included.

What do you think?

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/issues/167, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW2QVYVGZODCJR52QDF333SSUQ4JANCNFSM4UJJ5ZKA.

[LauraOlivera]

Thank you all for your responses and suggestions!

As @TarekHC mentions,

Although note if we are not fixing RA_PNT/DEC_PNT we need some other way to fix the pointing. Perhaps defining an ALT_PNT and AZ_PNT?

it would also be ok to use ALT_PNT, AZ_PNT, which indeed would make the IRF format completely compatible, since THETA would be then bins in zenith. The key issue is what types of things the pointing information is used for, I would distinguish between the concept of pointing as the definition of the central axis of the telescope, in which, indeed, a fixed pointing at zenith would be enough, or, for example, pointing understood as information about the events inside a run, i.e. if at any point you select observations by their pointing position. This second option would not make sense for wide-field data, and also wouldn't make sense for local coordinates pointing without timing information. I don't know if this is ever used like that at the moment, but I think it should be distinguished carefully. Something to note is that, while a typical IACT run is in the order of tens of minutes, a standard "observation" for a ground array, that is, a period in which the detector remains stable is usually much longer.

For the IRFs I had misunderstood what the different types were. My bad!

Note these variables mainly refer not to the way IRFs are stored, but the way they were calculated: How are IRFs calculated within HAWC? Simulating point sources over different positions in the sky and then optimizing cuts for a point-like source? Or simulating a diffuse/uniform distribution of gammas over the whole sky and not applying any theta cut?

In HAWC we typically simulate a point source as it moves across the sky during one transit, for different declination bins. So the end product is already convoluted with a source transit and given in sky coordinates as traditionally, the standard data product in HAWC are maps in sky coordinates. This is however a choice, and having IRFs in local coordinates is completely possible and has some other benefits, and thus is likely to be used in the future. So I think the current description is actually fine.

So in summary,

Question: If we are able to fix the pointing with zenith/azimuth, then wouldn't the current format be enough? If the pointing direction is the zenith (and is fixed, and science tools are able to accommodate to that) THETA would be the zenith distance, so in principle this would be equivalent to bins in zenith. The only difference would be that IACTs cover a significantly smaller range in THETA. But I don't see this requiring a new format, no?

Yes, I think so. Once the pointing handling is sorted out, that would pretty much be it, since the current description of the IRFs is indeed already inclusive of the formats used for ground arrays. That and every section name, of course :laughing:

I will prepare a PR about the pointing then, thank you all again for your very useful input!

[lmohrmann]

@LauraOlivera Just noticed that in the observation index table (https://gamma-astro-data-formats.readthedocs.io/en/latest/data_storage/obs_index/index.html), no changes have been made in #168. Shouldn't we also include OBS_MODE here, as in the EVENTS header?

[maxnoe]

@lmohrmann yes, indeed, we should make the same changes there.

[LauraOlivera]

that is a good point. I will open a new PR hopefully later today with those changes

[maxnoe]

Reopening, since the observation index table still contains the required columns mentioned above and does not contain the obs mode.

@LauraOlivera are you still up for opening the corresponding PR?

[maxnoe]

Closing, since #187 has been merged.