Resolve coordinate system issues

[Zarquan]

Following on from discussions at the October 2019 IVOA interop meeting, several people raised comments and issues about coordinate system transforms in ADQL.

This is a placholder issue that represents an ongoing discussion on the DAL mailing list. The discussion will continue as normal on the mailing list, and this GitHub issue will be closed when the discussion reaches a conclusion.

[Zarquan]

Email from Gilles Landais (CDS)

Coordinate System had been deprecated from the ADQL region. This capability was may be not very used, and was may be ambiguous. However to change the coordinate system was used in VizieR. Moreover, there is not today a pivot coordinate system used for all tables in all data-centers. I am agree that coordinate System could be an option, but in my opinion, to remove it without any alternatives will impoverish ADQL. If coordinate system in option is rejected, we could also add a function such as the function proposed by Markus in Gavo to make possible the change of coordinate System:

gavo_transform(from_sys TEXT, to_sys TEXT, geo GEOMETRY) -> GEOMETRY

So, I ask IVOA people to consider this capability as a ADQL standard (as a function - may be optional).

[gmantele]

Two conditions are required for a (TAP) service to support the coordinate system argument of geometric function:

1. To have a database providing coordinates in different coordinate systems
2. To support conversion of coordinates from one coordinate system to another

I think the 1st condition does not represent most of the (TAP) services ; it is generally the case with large databases like the one of VizieR, GAVO and CADC.

The 2nd condition is generally quite difficult to implement. Besides, in the very few services where it is already supported, it is partially implemented ; it does not cover all combinations of coordinate systems.

Consequently, I think it makes sense to make the coordinate system argument optional, as it is proposed in the current PR-ADQL-2.1 document.

---

However, I agree with Gilles: ADQL is an astronomical query language and in that sense there should be a way to deal with the different coordinate systems existing in astronomy. It is then important for services providing coordinates in different coo. sys., and it can not be seriously asked to them to choose only one coordinate system for all their coordinates.

So, from what Gilles proposed, I see two solutions:

1. ADQL-2.1 proposes optional function(s) for such conversions. Something like:

   COO_CONVERT(POINT pos,
              <string_literal> from_coo_sys,
              <string_literal> to_coo_sys)    -> POINT

   (I am aware that this is neither a correct BNF nor a correct UDF definition ; I just wanted to be brief and highlight the fact that the 2 last arguments should be string literals ; I am not very proud of the function name, but that is just a detail that can be discussed later)

2. ADQL-2.1 does not offer any (optional or not) function, but refers to the PEN-"Catalogue of ADQL User Defined Functions" document, which would have an ivo_ UDF for coordinate conversion, such as the one proposed by GAVO (i.e. gavo_transform), already listed there.

   _. @msdemlei , @jontxu : would that be possible to move gavo_transform into ivo_transform (or ivo_coo_convert or whatever else would be a better name)? What would be the condition(s) required to make that possible? A second implementation like the one of GAVO?_

I personally like the 2nd option, but I am not strongly opposed to the 1st one.

---

Any comment on these propositions? Any other proposition? What would be your preference?

[gmantele]

I forgot to take a look on the email thread. @msdemlei already replied to that in October:

If coordinate system in option is rejected, we could also add a function such as the function proposed by Markus in Gavo to make possible the change of coordinate System:

gavo_transform(from_sys TEXT, to_sys TEXT, geo GEOMETRY) -> GEOMETRY

So, I ask IVOA people to consider this capability as a ADQL standard (as a function - may be optional).

The current proposal is to have a list of common UDFs in a separate list rather than in the ADQL spec itself, and I guess the ADQL editor(s) will be grateful if we try to keep it this way.

A proposal for that separate list is in http://ivoa.net/documents/udf-catalogue/, and I guess it should go to RFC after the next TCG.

Which means this is the perfect moment to add a proposal -- if VizieR and DaCHS agree on this, I'd say that's a good base for making this and "agreed" UDF and call it ivo_transform.

As usual, the devil is in the details, namely what people can put into the system strings. And how they can figure out what systems a given service supports -- I suppose VizieR will want to support legacy equatorial systems at various equinoxes, for instance, which DaCHS at least so far doesn't.

So, here's a proposal for what the UDF catalog might say:

ivo_transform(from_sys TEXT, to_sys TEXT, geo GEOMETRY) -> GEOMETRY

The function transforms ADQL geometries between various reference systems. geo can be a POINT, a CIRCLE, or a POLYGON, and the function will return a geometry of the same type. From_sys and to_sys must be literal strings (i.e., they cannot be computed through expressions or be taken from database columns).

Note that the epoch is not changed by this function (i.e., no proper motions are applied).

Where services support reference frames from the IVOA vocabulary of reference frames (http://www.ivoa.net/rdf/refframe), terms from that list should be used in from_sys and to_sys. The description of the TAPRegExt feature declaring support for ivo_transform should enumerate the frames supported by a service. Using an unknown frame should be an error, and the accompanying error message should again enumerate the strings understood by a service.

Pointing to the refframe vocabulary certainly isn't good enough for frames that need an equinox. We could leave that to the services, but then it's probably better if we can give some sensible guidelines. I could see either

Referencence frames requiring an equinox append it after a single blank. Use B to indicate a Besselian, J to indicate a Julian epoch, e.g., "FK4 B1950" or "FK5 J1975.0"

(which would provide maximum flexibility) or

For legacy reference frames, it is recommended to just use the equinox with B indicating a Besselian, J to indicating a Julian epoch.

This latter thing would help with, for instance I/122/bd a.k.a. Bonner Durchmusterung -- though, of course, you could name its (popular, in its day) frame in the first scheme by saying EQUATORIAL B1855.0 just as well.

[msdemlei]

On Wed, Jan 15, 2020 at 02:14:57AM -0800, Grégory Mantelet wrote:

2. ADQL-2.1 does not offer any (optional or not) function, but refers to the PEN-"Catalogue of ADQL User Defined Functions" document, which would have an ivo_ UDF for coordinate conversion, such as the one proposed by GAVO (i.e. gavo_transform), already listed there.

   _. @msdemlei , @jontxu : would that be possible to move gavo_transform into ivo_transform (or ivo_coo_convert or whatever else would be a better name)? What would be the condition(s) required to make that possible? A second implementation like the one of GAVO?_

Moving the function is of course possible, and indeed I personally would hope that that is usually the way ivo_ functions are being added.

There is no formal condition defined (yet). The intention definitely is that several data centers agree on a certain signature before going for ivo_, with the TCG having the last word whether using it is sanctioned during the endorsement process.

I personally like the 2nd option, but I am not strongly opposed to the 1st one.

I agree here. Less extra syntax is generally a good thing.

Incidentally, as regards the implementation: Realistically, what is possible are simple rotations, which are also easy to implement (at least on top of pgsphere, but I guess any other spherical geometry system will have these, too). These cover transformations between different equinoxes on the same reference system and thus also, to sufficient precision, between FK5-based coordinates and ICRS. Modulo slightly different conventions, it's also good for going back and forth between Galactic, FK5/ICRS, and modern Ecliptical.

When going between FK4 and FK5 (not to mention even older reference systems), the transformations become really complex if you're aiming for arcsec precision; for one, FK4 and older rotate quite significantly against FK5 and ICRS, so you'd have to look at epochs and fix proper motions in non-trivial ways, and then there's FK4 with extra ("relativistic") corrections around the pole, and there's FK4 without, and it goes on like this.

So, we'll have to be honest about what kind of precision you can expect when using ivo_transform (for a sobering review of the state of affairs in the wider world of astronomical software, see http://www.astropy.org/coordinates-benchmark/summary.html).

Also, we have to make clear that "the horizon system" (i.e., azimuth and elevation) is not in scope for the function even though AZ_EL is probably going to remain in http://ivoa.net/rdf/refframe (which I suggest we should draw the system names from). The horizon system is different for each observer (defined by geolocation, height, temperature, pressure...), the transformation is complex (check out SOFA or the Astronomical Almanac supplement if you don't take my word for it), and it may require extra data beyond the spherical position for precise work (e.g., foreshortening) -- none of this should be required for ADQL.

[though I'm not saying that a UDF going to horizon system positions doesn't have a certain appeal to me: stellarium in ADQL would be fun]

[gmantele]

I asked to Gilles Landais about having functions like gavo_transform and gavo_to_jd in VizieR, and he agreed. He will not implement that in the coming weeks, but he will certainly do that in the future.

Having such agreement from another TAP Service and considering this function is already implemented in DACHS, would it be enough to rename gavo_transform (and gavo_to_jd) into ivo_transform (and ivo_to_jd)?

If yes, some description of allowed values (or syntax) and examples for the system coordinate arguments would be important to ensure that every implementations behave a minima the same. @msdemlei , do you think you can already give some in this discussion thread (while waiting for an update of the PEN about the UDFs Catalogue)?

[msdemlei]

On Wed, Apr 29, 2020 at 02:49:40AM -0700, Grégory Mantelet wrote:

I asked to Gilles Landais about having functions like gavo_transform and gavo_to_jd in VizieR, and he agreed. He will not implement that in the coming weeks, but he will certainly do that in the future.

I believe gavo_to_jd is, in principle, unproblematic, but for gavo_transform my impression is that we should think about it a bit more, and I'll certainly want to see a second implementation before moving on.

Having such agreement from another TAP Service and considering this function is already implemented in DACHS, would it be enough to rename gavo_transform (and gavo_to_jd) into ivo_transform (and ivo_to_jd)?

As you say, that would need to happen in the UDF catalogue, which still is in RFC for its first version. And there are no reviews in yet (Grégory: perhaps you could Bambi-eye your colleagues with seats in the TCG to break the ice?).

As long as that's the state of affairs, I'd like to avoid changing the UDF catalogue, as that would further confuse affairs.

ivo_transform will be an excellent opportunity to try out UDF catalogue updating, though. Promise: As the UDF cat moves to EN, I'll start this in a PEN for 1.01.

If yes, some description of allowed values (or syntax) and examples for the system coordinate arguments would be important to ensure that every implementations behave a minima the same. @msdemlei , do you think you can already give some in this discussion thread

I think we are agreed that astrometry operations, among them these transforms, will be handled through geometry-valued UDFs, and that the UDF catalogue is the place to specify them. In my book, that would be sufficient justification for closing this issue, or to leave it until the UDF cat becomes EN and close it then.

[gmantele]

I think we are agreed that astrometry operations, among them these transforms, will be handled through geometry-valued UDFs, and that the UDF catalogue is the place to specify them. In my book, that would be sufficient justification for closing this issue, or to leave it until the UDF cat becomes EN and close it then.

I agree. I would tend to leave this issue opened until the UDF cat becomes EN...so that we do not forget about it.

~~Anyway, is it still possible to refer to this Note (even if still PEN) in the ADQL document? My idea was to drop somewhere in there a comment indicating to whoever is still interested in sys. coord. conversion/transform where they can find such information. But if not possible, I think it could still wait for the next version of ADQL....people would end to know anyway about this UDF cat. without this comment.~~

[gmantele]

Sorry, forget about the last part of my last comment: it was not supposed to be in this issue but in #16

[msdemlei]

On Wed, Apr 29, 2020 at 09:01:39AM -0700, Grégory Mantelet wrote:

Anyway, is it still possible to refer to this Note (even if still PEN) in the ADQL document?

You can cite whatever you want, and there's plenty of precedent for referencing in-progress documents even in RECs, although there's language against such practice in the drafts.

However, given that ADQL 2.1 isn't even in PR yet, I'm rather confident that the UDF cat will be EN before ADQL 2.1 becomes REC. Hence, I'd just say something like "User defined functions with the ivo_ prefix are defined in a catalgoue of user defined functions \citep{TODO} maintained by the IVOA using rules given in that document" or so -- so, the citation would, for now, go nowhere. That way, once people do the pre-flight check (http://ivoa.net/documents/Notes/IVOATex/20180814/NOTE-ivoatexDoc-1.2-20180814.html#tth_sEc3.11), they'll notice there's something they need to fix. If we put in a reference to the PEN now, I'm rather certain we'll forget to update the reference when there's an EN.

[molinaro-m]

I agree leaving this open till UDF cat goes EN or at least ADQL goes RFC. (note: ADQL-2.1 is already Proposed REC, but with a really old, Jan 2018, document)