Closed LauraOlivera closed 3 years ago
Keep in mind that "pointing" is the position of the center of the field of view of the instrument. For IACTs, that can move in time, both in Alt/Az and RA/Dec, depending on the mode of operation. The standard "tracking" mode of operations has each pointing fixed in RA/Dec, which means that Alt/Az is moving in time to account for the rotation of the earth. However, one might also have observations that are about a fixed Alt/Az (standard for water cherenkov telescopes, and sometimes called "drift" mode for IACTs). In drift mode, RA/Dec change in time, again due to the rotation of the earth.
Note that in general there is also a third case: "slew" mode, where neither the Alt/Az nor Ra/Dec coordinate of the center of the FOV is fixed in time. THis is not used often, but could always occur, particularly for fast observations of transients like GRB, where the telescope may still be moving to the target RA/Dec while data are being recorded. There are also cases in other instruments (XMM, etc) where data recorded during slewing between targets is later used to make a sky map.
This is why including pointing info as header information is not possible in a general sense, and a separate pointing table that includes time is always a better solution (or including the pointing along with each event, which wastes some space since it changes slowly in general, but makes it easier to work with).
If you want to have headers that describe the pointing, at least these three cases may need to be taken into account. Mode=["tracking", "drift", "slew"], and at least for tracking and drift, one can also include fixed coordinates in ICRS and Horizontal coordinates respectively. FOr "slew", a reference to a separate table, or per-event pointing must be used.
There is also an implicit assumption here that the FOV is circular, so it can be defined only by the pointing position. FOr non-circular FOVs, you would need more information, at least a rotation angle (imagine a rectangular FOV that can be in any orientation)
I would also think about what is the purpose of having this info in the header? The answer is I think only to be able to select observations at the "run" level that overlap some region on the sky, to speed up data processing. After all, the science tools do not really care about the center of the FOV, only on the event coordinates. So the general problem is: without having to loop over all events for all observations, how to tell quickly when an observation is the the region of the Sky that I want?
That may not even be a major consideration for large FOV instruments: for a 2PI steradian FOV, you would save at most 50% of analysis time for a single source if you could throw away data where it was looking in the opposite direction from your source. But generally wide FOV instruments just make all sky maps rather than analyzing a single ROI on the sky. so this is not a problem that needs to be solved.
Therefore I'd say make all of these headers optional - for wide FOV, they don't need them at all, and for small FOV, it depends on the mode.
Currently GammaLib is using the "RA_PNT" and "DEC_PNT" keywords to extract the pointing direction, I assume that the same happens in gammapy. The pointing direction is needed since the IRF is given in instrument coordinates, and to transform between sky and instrument coordinates you need to know the pointing direction (and in principle also a orientation, which is however not relevant for axisymmetric response functions).
So these keywords definitely have a use, whether they should be mandatory is another question.
Le 18 déc. 2020 à 12:24, Karl Kosack notifications@github.com a écrit :
I would also think about what is the purpose of having this info in the header? The answer is I think only to be able to select observations at the "run" level that overlap some region on the sky, to speed up data processing. After all, the science tools do not really care about the center of the FOV, only on the event coordinates. So the general problem is: without having to loop over all events for all observations, how to tell quickly when an observation is the the region of the Sky that I want?
That may not even be a major consideration for large FOV instruments: for a 2PI steradian FOV, you would save at most 50% of analysis time for a single source if you could throw away data where it was looking in the opposite direction from your source. But generally wide FOV instruments just make all sky maps rather than analyzing a single ROI on the sky. so this is not a problem that needs to be solved.
Therefore I'd say make all of these headers optional - for wide FOV, they don't need them at all, and for small FOV, it depends on the mode.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/pull/168#issuecomment-748033547, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW2QV6K72WYDS3JPGDP6OLSVM3XZANCNFSM4U77XZBQ.
We also have the pointing table: https://gamma-astro-data-formats.readthedocs.io/en/latest/events/pointing.html
Indeed, but is there some software around that is actually using it and has any instrument ever filled this table?
Le 18 déc. 2020 à 14:27, Maximilian Nöthe notifications@github.com a écrit :
We also have the pointing table: https://gamma-astro-data-formats.readthedocs.io/en/latest/events/pointing.html https://gamma-astro-data-formats.readthedocs.io/en/latest/events/pointing.html — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/pull/168#issuecomment-748083892, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW2QV67RVMR4RRP6JLSXKLSVNKERANCNFSM4U77XZBQ.
I have no strong preference, as @kosack says, for wide FOV instruments you pretty much would never select observations based on some defined pointing. Mainly because usually an "observation" or minimum data unit is quite long, so the entire available FOV has been covered. The idea with azimuth and altitude pointing was to have a fixed value (zenith), so it doesn't really add any information, it was just some sort of compromise.
Making both optional would definitely be simpler, especially given that a separate pointing table is defined. But maybe that is problematic somewhere else? I can implement/change whatever we decide to do in the end, just let me know what you think is best. Thanks!
HI all, just wanted to revive this. I feel like there was no strong opposition to make the pointing information optional to allow wide-field instrument event lists to conform to the standard but there are several options how to achieve this. The options seem to be
Note that for wide-field observatories the pointing information is essentially useless and never accessed. The "pointing" direction is just always zenith. So the issue is to allow for this while respecting the fact that this information is important and used for IACTs.
I would say that the third option is my preferred one since it means that existing files produced for IACTs wouldn't need to change, and I just don't need to add an OBS_MODE/TRACKING for my HAWC events. But perhaps there's nuance that I am missing. Option 1 is the simplest but it could lead to a situation where a technically compliant IACT event list without any pointing information in the header leads to errors in data reduction since that information is actually required for the IRF projection onto the sky.
What are your thoughts?
Thanks!
Hi all, The option "Keep one set as mandatory but specify which one (so RA/DEC or ALT/AZ) via the existing keyword OBS_MODE" might satisfy both pointed and non-pointed instrument. My preferred option also, a mandatory position (ALPHA_PNT/BETA_PNT) whom OBS_MODE precises the referential.
@bkhelifi If we go that road, I would rather use LON_PNT
, LAT_PNT
instead of more use of less than obvious greek letters. We have too many of those already.
Hi all,
I wanted to check if some of these questions have been answered by people in the past (surely smarter than me). So I did a quick search on what are other OBS_MODE
FITS uses, to see if there is any standard. Even if sometimes the keyword is used to describe data-taking modes (e.g. imaging/spectroscopy) it seems the HEASARC standard is:
KEYWORD: OBS_MODE
REFERENCE: HEASARC
HDU: any
DATATYPE: string
COMMENT: instrumental mode of the observation
DEFINITION: The value field shall contain a character string which gives
the observing mode of the observation. This is used in cases where the
instrument or detector can be configured to operate in different modes
which significantly affect the resulting data. Examples: 'SLEW',
'RASTER', or 'POINTING'
I found the reference extremely informative, and I believe we can fully accommodate to that OBS_MODE
standard.
The standard, multi-mission character string values of the OBS_MODE keyword are as follows:
* OBS_MODE='POINTING'
for data when a three-axes stabilized satellite is pointing in a particular direction of the sky, and a 'mean' pointing position can be given.
* OBS_MODE='RASTER'
for data collected by a three-axes stabilized satellite which is making a 'raster scan' over a region of the sky.
* OBS_MODE='SLEW'
for data obtained whilst a (typically three-axes stabilized) is moving between one observation and another.
* OBS_MODE='SCAN'
for data collected by a scanning (spinning) satellite.
None of these modes is exactly what we need, so I would perhaps define a "DRIFT" mode (would that be enough?).
Then, regarding the keywords to use to fix the "pointing" direction (ideally both for IACT's drift mode and HAWC):
@bkhelifi If we go that road, I would rather use LON_PNT, LAT_PNT instead of more use of less than obvious greek letters. We have too many of those already.
Even with the risk of showing how little I know... LON
and LAT
are GCS, so I personally strongly disagree to use those to specify the pointing direction (to the sky!). In addition, that would only cover water Cherenkov detectors, and not IACT drift observations (unless I'm missing something!).
Keep one set as mandatory but specify which one (so RA/DEC or ALT/AZ) via the existing keyword OBS_MODE or defining a new one (TRACKING = {ICRS,HORIZONTAL}?). RA/DEC could be the default, in the absence of the extra keyword.
Just a short comment: the reference coordinate frame is already set within RADECSYS
, which is already implemented and seems to be the standard (see this, or this).
So, the way I would implement this with the smallest possible changes to the current standards is:
RA_PNT
and DEC_PNT
mandatory for OBS_MODE = POINTING
AZ_PNT
and ALT_PNT
mandatory for OBS_MODE = DRIFT
All these keywords are already within the DL3 specifications, so we would only need to improve the description. In fact, there is already a section explicitly saying the OBS_MODE
keyword "in the future will be a key piece of information in the DL3 data model, defining the observation mode (e.g. pointed, divergent, slewing, …) and being required to analyse the data correctly.".
Would this work? Or perhaps I'm missing something?
Even with the risk of showing how little I know... LON and LAT are GCS,
No, longitude and latitude are generic terms for the two coordinates of spherical system.
No, longitude and latitude are generic terms for the two coordinates of spherical system.
I disagree. For data format standards, you cannot use terms that are so heavily associated with specific coordinate systems (I mean... google "lon lat" and tell me what you find). Coordinate systems are already hard enough to understand... We should not make it even harder.
But again, why not just use AZ_PNT
and ALT_PNT
which are already there? What am I missing?
@TarekHC That was my initial proposal, but then @bkhelifi said he would rather have a single set of required keywords that could hold both. My comment was on only iff we use that approach, then I wouldn't use ALPHA/BETA as these don't have any meaning.
We already use lon/lat
for field of view coordinates and e.g. astropy uses it for the generic variables lat/lon.
google "lon lat" and tell me what you fin
Sorry, but that is not really an argument. Because for "normal people" the GCRs is the only spehrical coordinate system they will ever use, that's not the case for us.
There is a reason why the fields in FITS are called OBSGEO-*
Ok, let's be constructive: I believe small changes following already available specifications is the way to go (we probably all agree on that).
Now the question is: do we leave RA/DEC pointing in the Mandatory header keywords (as they will be indeed be required for all POINTING-mode observations) or put them as optional (as they won't be always needed). My preference is to put both RA/DEC and AZ/ALT pointing keywords as required, and specify when each is indeed required. That way it is clear that the EVENTS table will always need pointing keywords. Perhaps we could add a small "IF" section, meaning one "point" describing that adding one pointing keyword is mandatory, but depends on the specific OBS_MODE.
So, very specifically, the required changes we would need are:
OBS_MODE
description, give the references I gave above, and perhaps discuss about an additional "DRIFT" mode in which local zenith/azimuth coordinates remain constant.RA_PNT
and DEC_PNT
in the Mandatory header keywords. Perhaps reorganize the information a bit as discussed above.AZ_PNT
and ALT_PNT
into the Mandatory keywords, explicitly saying they are only required for DRIFT-mode observationsIf there is agreement with this proposal, please add a thumbs up/+1/or your favorite emoji. If there is not, please suggest a change until we reach consensus. If there is consensus and no one raises any concern, then we can proceed with implementing the changes.
So, in the case of drift mode, what contain the mandatory fields, ra_pnt and dec_pnt? An average value (over time)? This behaviour should be documented somewhere
I understood that in DRIFT mode they are simply not mandatory anymore, and hence not filled (and that is what I implemented in my latest commit)
Why not use the centre of the start when in drift mode?
@cboisson I guess you mean the RA/DEC of zenith at the start of a run. For wide-field observatories (which this keyword is intended for), the duration of a run or minimal observation unit is much longer than for IACTs, on the order of one day. So that information is really not useful or informative, and I would say including it would just be confusing.
Hi @bkhelifi and @cboisson!
So, in the case of drift mode, what contain the mandatory fields, ra_pnt and dec_pnt? An average value (over time)? This behaviour should be documented somewhere
The idea of the drift mode is to fix the zenith/azimuth of the observation. I did not find any OGIP standard for such an observing mode (although it probably exists! I'm sure...). But if the idea is to fix the zenith/azimuth, the mandatory keywords should indeed refer to those parameters explicitly. I believe that would be way more straightforward both for understanding the content of the FITS files, and also for the science tools to handle the data.
Why not use the centre of the start when in drift mode?
What do you mean with "centre of the start"?
Whatever the choice, it should be explicitely describe somewhere such that any user can understand the keyword. My personal choice goes towards an average value over time, such that these parameters can be used as observation selection. Depending of the experiment, the duration of an observation is different. So it might be hard to guess which part of the sky is seen with ra/dec only at the beginning of an observation.
Hi @bkhelifi and @cboisson!
So, in the case of drift mode, what contain the mandatory fields, ra_pnt and dec_pnt? An average value (over time)? This behaviour should be documented somewhere
The idea of the drift mode is to fix the zenith/azimuth of the observation. I did not find any OGIP standard for such an observing mode (although it probably exists! I'm sure...). But if the idea is to fix the zenith/azimuth, the mandatory keywords should indeed refer to those parameters explicitly. I believe that would be way more straightforward both for understanding the content of the FITS files, and also for the science tools to handle the data.
Why not use the centre of the start when in drift mode?
What do you mean with "centre of the start"?
We agree Tarek... The point is that ra_pnt and dec_pnt will be still in the header, even in the drift mode. they are mandatory. So, what will contain these mandatory values? Nan? value at the start of an observation? mean value over the observation length?
My personal choice goes towards an average value over time, such that these parameters can be used as observation selection.
EDIT: Ok, I see we converge on this! (I believe that, for the science tools, alt/az_pnt keywords are required). Any other information included could indeed be useful, but could also help to misinterpret the data.
At least within gammapy, you have HDU tables devoted to list some of the parameters within runs (never remember their name!). Gammapy could easily calculate ra/dec of each run, and add them to those tables to provide the use case you are proposing.
We agree Tarek... The point is that ra_pnt and dec_pnt will be still in the header, even in the drift mode. they are mandatory. So, what will contain these mandatory values? Nan? value at the start of an observation? mean value over the observation length?
What we are proposing is not to include them for the special case of DRIFT mode.
Can you define a "drift mode" ? what will be stored to know where we are on sky , how it drifts etc? I certainly have misunderstood what you ar speaking aboit.
Can you define a "drift mode" ?
Sure. The drift mode will be an observation in which the telescope pointing is not changing in local coordinates (zenith/azimuth) while the sky is "drifting". This is important because:
As DL3 data already contains enough metadata to determine the location of the instrument, and the exact arrival time of each event, the ra/dec of each event is uniquely defined as long as you know the azimuth/zenith of the pointing.
Maybe we can do a more verbose version of what @LauraOlivera has already drafted?
The current definition is:
"Additionally to the OGIP-defined values (POINTING
, RASTER
, SLEW
and SCAN
), we propose an additional option DRIFT
to accomodate ground-based instruments, in which local zenith/azimuth coordinates remain constant. In this case, the header keywords RA_PNT
and DEC_PNT
are no longer mandatory, and instead ALT_PNT
and AZ_PNT
are required."
I can expand it if you think something is missing.
My personal choice goes towards an average value over time, such that these parameters can be used as observation selection.
For wide-field observatories a typical run contains the entire field of view (i.e. ~1 day of observation), which is a very large portion of the sky. This means that data cannot be selected based on the pointing position like it is done for IACTs. I believe then, not having RA_PNT/DEC_PNT is the most clear solution for this case, as any other option means choosing some position "artificially" and would not even be useful to select data based on it. For other cases it can of course be present and defined in a different way, but I am not sure what would be the best option.
I support Bruno´s position, some information is better than nothing. An average of the position of the "center of the FoV", usually the zenith for static detectors, gives some idea and in many cases a rather accurate one.
El vie, 11 jun 2021 a las 11:48, LauraOlivera @.***>) escribió:
The current definition is:
"Additionally to the OGIP-defined values (POINTING, RASTER, SLEW and SCAN), we propose an additional option DRIFT to accomodate ground-based instruments, in which local zenith/azimuth coordinates remain constant. In this case, the header keywords RA_PNT and DEC_PNT are no longer mandatory, and instead ALT_PNT and AZ_PNT are required."
I can expand it if you think something is missing.
My personal choice goes towards an average value over time, such that these parameters can be used as observation selection.
For wide-field observatories a typical run contains the entire field of view (i.e. ~1 day of observation), which is a very large portion of the sky. This means that data cannot be selected based on the pointing position like it is done for IACTs. I believe then, not having RA_PNT/DEC_PNT is the most clear solution for this case, as any other option means choosing some position "artificially" and would not even be useful to select data based on it. For other cases it can of course be present and defined in a different way, but I am not sure what would be the best option.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/pull/168#issuecomment-859456852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC52BBMJI5PXLHLHA5IVQVTTSHLV3ANCNFSM4U77XZBQ .
-- Jose Luis Contreras Physicist e-mail @.*** Dpto EMFTE Universidad Complutense de Madrid Avda Complutense s/n Tel +34 91 3944551 E-28040 Madrid (Spain ) Fax +34 91 3945193
I personally feel adding ra_pnt in drift mode could lead to misinterpretations of the data... But I agree that ra/dec would allow any user with a quick look to know where the data was roughly pointing. Again, let's try to be constructive and reach a consensus:
- Pointing Right Ascension (see :ref:
coords-radec
). Not required ifOBS_MODE=DRIFT
Perhaps we could simply turn this into something like:
coords-radec
). Not mandatory if OBS_MODE=DRIFT
, but average values could still be provided.Very minor comment: please say "optional" instead of "not mandatory", so we only have mandatory and optional headers, not mandatory, optional and not mandatory ...
Very minor comment: please say "optional" instead of "not mandatory", so we only have mandatory and optional headers, not mandatory, optional and not mandatory ...
I actually feel "not mandatory" is suitable here, because the keyword will be in the mandatory section.
Again, this is a very minor correction... So I don't mind any of the two options. @LauraOlivera feel free to include the alternative of your choice, and let's see if everyone agrees.
I am afraid it will lead to confusion wether "not mandatory" keywords are different from "optional" keywords and how.
Though it is late... @tarek thanks for the explanation, it corresponds to what I had in mine. This is a common radio astronomy mode. And sorry for the noise, I realized I sent a truncated sentence, it should have read : start time, end time and position. So I backup @bkhelifi and @contrera
Not mandatory if
OBS_MODE=DRIFT
, but average values could optionally be provided.
Let's give a week or two for others to comment on this changes. If no additional red flags are raised, I vote for merging this PR.
Thank you @LauraOlivera !
I am fine with this, but I want to repeat my point about trying to avoid confusion with the three terms
I would rather only have
but I see that the conditional mandatory keyword is necessary
Thanks for your comment @maxnoe. I thought about it, and concluded that because the keywords are in the mandatory section, using "not mandatory" was the clearest formulation because it clearly refers back to the section they are in. Otherwise it could lead to confusion. I also added "can optionally be provided" to make even more clear that we mean that the information in that keyword would then be optional. Given that the structure of the page is clearly divided into Mandatory/Optional, I don't think this leads to confusion with thinking there is an additional third option. But if this is a showstopper I can of course change it...
I am thinking about people implementing readers and writers of the format here.
Right now, we have to clear types of keywords: mandatory and optional.
This introduces a third type: "Not Mandatory IF SOME CONDITION". I see that this is necessary, but we should make sure that this is well-defined and clear.
Also, in this special case, this CONDITION
depends on an optional keyword. I think this is not good.
So at least, we should now also promote the OBS_MODE
keyword to mandatory or at least define a default.
Also, you might want to edit / expand the notes section on OBS_MODE: https://gamma-astro-data-formats.readthedocs.io/en/latest/events/events.html#obs-mode
which for example doesn't fit the expectations @TarekHC had about obs_mode.
So I think, we need better define OBS_MODE
before being able to use it in conditions like this.
If you look at the files edited, I have already expanded the section on OBS_MODE, I copy below for convenience:
The keyword ``OBS_MODE`` specifies the mode of the observation.
This is used in cases where the instrument or detector can be configured to
operate in different modes which significantly affect the resulting data, or to accomodate
different types of insrument. More details can be found
`here <https://heasarc.gsfc.nasa.gov/docs/heasarc/ofwg/docs/general/ogip_94_001/ogip_94_001.html>`__
Additionally to the OGIP-defined values (``POINTING``, ``RASTER``, ``SLEW`` and ``SCAN``), we
propose an additional option ``DRIFT`` to accomodate ground-based instruments, in
which local zenith/azimuth coordinates remain constant. In this case, the header keywords
``RA_PNT`` and ``DEC_PNT`` are no longer mandatory, and instead ``ALT_PNT`` and ``AZ_PNT``
are required.
This keyword is optional, which means that in its absence, the default becomes to consider
``RA_PNT`` and ``DEC_PNT`` as mandatory.
It is likely that ``OBS_MODE`` in the future will be a key piece of information
in the DL3 data model, defining the observation mode (e.g. pointed, divergent,
slewing, ...) and being required to analyse the data correctly.
So I defined a default behavior and expanded the definition. I can also promote the keyword to mandatory, which is already sort of foreseen in the description.
@LauraOlivera Yes, sorry, missed that.
I have one small wording comment for that and would go for making OBS_MODE mandatory. Any other opinions on this?
I agree with making OBS_MODE mandatory. It is too important to have it undefined.
El mar, 15 jun 2021 a las 13:38, Maximilian Nöthe @.***>) escribió:
@LauraOlivera https://github.com/LauraOlivera Yes, sorry, missed that.
I have one small wording comment for that and would go for making OBS_MODE mandatory. Any other opinions on this?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/pull/168#issuecomment-861424617, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC52BBJMKUQD7IRH63HFOPLTS43THANCNFSM4U77XZBQ .
-- Jose Luis Contreras Physicist e-mail @.*** Dpto EMFTE Universidad Complutense de Madrid Avda Complutense s/n Tel +34 91 3944551 E-28040 Madrid (Spain ) Fax +34 91 3945193
I think we have enough consensus for a merge, right?
@LauraOlivera Thanks again for contributing this and to everyone participating in the discussion to get this right!
Hi all,
As we discussed in the issue #167, I've made some small changes to make the format inclusive to wide-field instruments, such as HAWC.
RA_PNT
,DEC_PNT
in the event lists are now optional, and instead,AZ_PNT
andALT_PNT
are mandatory, which keeps the pointing information in the mandatory headers, while allowing wide-field instruments to set these to some fixed value (zenith) and remain compatible.For this last point, however, it is important to distinguish what type of purpose would this pointing information be used for. I would distinguish between the concept of pointing as the definition of the central axis of the telescope, in which, indeed, a fixed pointing at zenith would be accurate for ground arrays, or, for example, pointing understood as information about the events inside a run, i.e. if at any point you select observations by their pointing position. This second option would not make sense for wide-field data, and also wouldn't make sense for local coordinates pointing without timing information. I don't know if this is ever used like that at the moment, but I think it should be distinguished carefully. Something to note is that, while a typical IACT run is in the order of tens of minutes, a standard "observation" for a ground array, that is, a period in which the detector remains stable is usually much longer.
That is why I didn't touch the
POINTING
section because I don't know exactly what it would be used for.Please let me know what your thoughts are and if you think I should change anything!