General spatial reference and position types and functions

[c42f]

Continuing some discussion from #11, it would be great for Geodesy.jl to define a common language for spatial reference systems and transformations. This common interface could be implemented by several transformation backends such as Proj4, and hopefully also allow for interoperability between the backends.

Here's a big brain dump of tasks and design decisions I think are relevant - I'd be very happy to discuss to refine these into a proper roadmap for the package.

1. Define functions and abstract type(s?) for talking about spatial reference systems. A survey of OGC standards and libraries proj.4, OGR and other similar libraries would be useful here to understand which concepts should be captured. Things to consider
   • Functions and types for communicating information about spatial reference systems http://www.opengeospatial.org/standards/sfa http://www.gdal.org/classOGRSpatialReference.html ...
   • Some way to deal with libraries of predefined SRS (authority:srid form). (Bonus: interop with http://spatialreference.org/ and/or http://epsg.io/ ?)
2. Define a low level transformation interface which may be implemented by Proj4.jl, GeographicLib.jl (if it existed), julia native GeoTransforms.jl (if it existed), etc. Some things to consider -
   • Keep the point types simple; pass SRS information independently of the points
   • Per point or multi-point transforms? Per point is preferred due to simplicity, but proj.4 has the ability to transform multiple points at a time; need a little benchmarking to determine whether this batching provides a speed boost. IIRC GeographicLib has an entirely per-point API.
   • Flexibility about the contents of a point. In principle these are 2D (realistically only lat,lon) 3D, or 4D (including time for high precision surveying in, eg ITRF) in order of increasing fidelity
   • Interop between different transformation backends via a shared central SRS (proj.4 uses WGS84 ECEF. What about points containing time?)
   • Differential information is sometimes desired. Traditionally this would be convergence and scale for map projections, but true differentials are also interesting in some circumstances. In proj.4 this is only available via the semi-internal projects.h, but GeographicLib exposes it as a first class part of the API.
3. Define a high level interface making use of more powerful information in the point types.
   • Geospatial points may contain SRS information in their type, auxilary fields, or some combination of these (see also some discussion starting at https://github.com/JuliaGeo/Geodesy.jl/pull/11#issuecomment-181008533). We should be able to be quite flexible about this with appropriate generic functions
   • It's easy to put basic information like the ellipsoid in the point types, a la #11. This seems to make good sense.
   • In #11 there's several types defining different flavours of SRS: LLA for geodetic, ECEF for Cartesian earth-fixed, and ENU for local geodetic frames, each of which should be parameterized on an underlying Datum. Presumably once we've defined an appropriate set of functions there's no need for these to be blessed types, but merely one possible implementation.
   • For some use cases, there are SRSs which are parameterized on continuous variables with many different values (eg, ENU). It would be nice to give the user a clean way to avoid the resulting type explosion. (Or maybe they just use the low level interface?)

Phew, there's probably a lot more to add, but I've run out of steam for now!

@awbsmith

[yeesian]

This looks good, thanks! I'll linkdump some related discussions:

• https://github.com/FugroRoames/Proj4.jl/pull/7#issuecomment-143563526
• https://github.com/JuliaGeometry/meta/wiki/First-Meetup-Minutes

Personally, having gone down this path with GeoInterface (and from stalling on the discussions above), I don't think we're ready to begin talking about a high-level interface yet. But from mucking around, it's become quite clear that

• we'll need to be able to represent objects/attributes in spatial reference systems (e.g. through a Projection, or Geodesic object, the form/function of which is not entirely clearcut to me). An understanding of the OpenGIS specification (if any), and survey of the APIs across libproj, geographiclib and OSR will help here.
• there are multiple forms point data might come in (e.g. as individual coordinate arguments to the function, as a tuple/vector/vec2/C-pointer/etc). And no one likes to "copy" data if they can help it. An interface will need to address the ability to deal with the "combinatorial explosion" of different argument types, or at least to "bless" some over others.

For that reason, I think it'll help to separate the representation of the spatial reference system from the point-data for now, i.e.

Manually keep track of the coordinate system, and explicitly pass it to conversion functions as needed.

and to make progress on the first 2 bullet points in Chris' list.

[c42f]

Thanks for the links. Regarding the meetup minutes, I agree that having point types as tuples of parameterized coordinates is pretty much a nonstarter. IMO the position along a given axis of an SRS doesn't have a useful existence independent of the other coordinates. Personally I feel that the way out of the short-vector type mess is more of a traits based approach where it's the generic functions which matter, and we can afford to be rather permissive with the actual concrete types. Only prototyping and experience will tell whether that's actually doable.

Having said that, I'm hopeful that something like FixedSizeArrays will eventually be an integral part of the standard library, and people will converge on using it by default in many circumstances. Unfortunately FixedSizeArrays still has design problems which are very hard to fix without a few changes to Base and the language itself.

[awbsmith]

Hi all,

I've gone ahead and forked the Geodesy "xy" (datum) branch while trying to incorporating the discussion in this thread and the pull request. I've tried to make the ReadMe be informative regarding the organization. Any feedback as to whether its what you're both expecting would be good.

https://github.com/awbsmith/Geodesy.jl

[c42f]

Cheers, thanks Andrew. I'm not going to have time to have a good look at the implementation before I get back to work on March 9th, but I'm excited by the traits based design for point types which we've been putting together.

@yeesian and others - for some context, @awbsmith has been working on the high level interface for point types - how to add coordinate system information to points (including user-defined point types), and let these nicely transform between coordinate systems.

We've also been discussing the semantics of the Datum concept and how it relates to the transformation system. My current opinion is that point types should be associated with a Datum which is simply a tag type (much as the LLA in the original PR which started all this). This seems to neatly parallel the original notion of Datum in surveying, as a physical reference point from which a coordinate system could be extended via a triangulation network (the relationship being type_tag ~ physical_object). The type system would neatly prevent mixing datums by accident, with a datum shift requiring the user to think a bit about what they're doing (or load a package like Proj4 which has some selected datum shifts built in). I remain a bit confused about modern worldwide datums, in particular the exact technical details of regressing these based on satellite and celestial measurements. Some more reading is probably required there to make sure the system represents the concept of Datum in a fully consistent way.

The nature of a lower level interface is still a big TODO, along with the interface to Proj4 or other transformation libraries.

[yeesian]

@awbsmith has been working on the high level interface for point types - how to add coordinate system information to points (including user-defined point types), and let these nicely transform between coordinate systems.

I won't have time to look at this for quite some time, but am excited about the progress you are both making, so don't take my silence as disapproval. I don't think my understanding of SRS and Datums is anywhere near complete; so a good set of specs and extensive documentation will really help. Since everything's still exploratory, I won't worry about the dependency on Proj4, or lack of a lower-level interface (since there aren't really any competing backends/wrappers) just yet.

[yeesian]

Andrew and Chris, the following might be relevant: https://github.com/JuliaGeometry/GeometryTypes.jl/issues/46

I just got reminded of it by @SimonDanisch

[awbsmith]

@yeesian I've been in the same boat learning about geospatial systems and datums as I go. I've been finding the terminology to be a mess of different groups using different and conflicting names for things.

I've been trying / meaning to follow the terminology used by the EPSG authority (the GIS guys at work tell me its terminology is "official") . Searching for code 4326 at http://www.epsg-registry.org/ gives an example of the data structuring and terminology I'm trying to replicate.

That said there's no guarantee other software packages / groups are using the same terminology, e.g. Oracle's Spatial package uses spatial reference system where the EPSG group uses coordinate reference system, and then goes on to to use coordinate reference system and coordinate system interchangeably, whereas in EPSG terminology they're distinct things.

On the documentation side, we have a plan to improve it by incrementally getting new people to read it and create a list of questions / unexplained stuff. Then we'll update the readme and get the next person to read it...

[c42f]

We've had a couple of iterations of design for Geodesy.jl in FugroRoames/Geodesy.jl trying to make it into something that'll work for what we're doing internally. We now have a PR for what seems like a reasonable but fairly minimal direction forward at https://github.com/FugroRoames/Geodesy.jl/pull/3

We need to figure out how FugroRoames/Geodesy.jl will relate to JuliaGeo/Geodesy.jl. The JuliaGeo version seems dormant at the moment, so either

• A bunch of us will need to join JuliaGeo if we want to drive this forward (myself, @awbsmith and @andyferris I guess?)
• We make FugroRoames/Geodesy.jl into the primary version
• We rename our version of the package to something else

On balance I'd prefer the first of these since it seems ideal for everyone to collaborate on the same version if possible.

[visr]

I invited you guys to join JuliaGeo, I agree it would be best to work on one version.

[andyferris]

Thanks @visr!

[yeesian]

+1 from me. I agree it will be best to work on one version too. This work has been ongoing for a while in the open and is definitely not a coup d'etat. I have the following requests:

• Given that FugroRoames/Geodesy.jl has a dependence on FugroRoames/Proj4.jl, is it okay to move the latter to JuliaGeo too?
• During the overhaul, to give the current types/methods in JuliaGeo/Geodesy.jl one round of deprecations/warnings for how they should be re-written to work with the new types/methods.

cc @garborg

[garborg]

I think this is great. I've been wanting to get back to this but unable to, and am glad to see the work you've put into redesigning the package.

Re: deprecation period, that sounds right, but I want to make avoid taking wind out of the current devs's sails, so whatever they think is the easiest way to achieve it for a while without cluttering up the code base they're actively working on.

[andyferris]

@yeesian Yes, we are currently working on Proj4.jl so that it is based on the new Geodesy.jl, so for me it's OK at looking at transferring it over when it becomes functional again.

@garborg Thanks. Yes I think I will keep deprecation warnings to the minimum (error message regarding removal of Bounds (this looked more like generic geometry than geodesy per se), and a warning for lack of default datum) - otherwise many of the underlying things should work as-is for the moment.

When we say "one round" of deprecation warnings, does that mean a single patch-level tag?

[garborg]

I'd say let Base.@deprecate do its deprecation warnings and rerouting for a minor-level tag or a month or so, which ever is greater, then clear out anything that's annoying dev-wise.

[c42f]

@visr thanks a lot! Sorry I haven't been very reponsive on this issue - just really busy here. On the backward compatibility front, I'm not sure we can meaningfully deprecate things since the design we're iterating toward seems quite different from what's here now. IMHO the right thing is a clean break (it's currently version 0.0.1 after all!) and people can Pkg.pin() to v0.0.1 if necessary.

Having said that, things are still quite in a state of flux - I hope we'll have a presentable design in a few days. Even so, it's not ideal to have the PR over at FugroRoames - @andyferris can you push your WIP changes to a JuliaGeo/Geodesy branch and create a pull request here? We should be looking to centralize the discussion.

[c42f]

BTW guys, I've (foolishly?) decided to do a talk at juliacon 2016 which in part will involve whatever we end up doing with Geodesy.jl and Proj4.jl.

[yeesian]

On the backward compatibility front, I'm not sure we can meaningfully deprecate things since the design we're iterating toward seems quite different from what's here now. IMHO the right thing is a clean break (it's currently version 0.0.1 after all!) and people can Pkg.pin() to v0.0.1 if necessary.

Sure, that's reasonable. I'm currently busy with preparation for my general exam, and will let you all make the call on the best way forward.

BTW guys, I've (foolishly?) decided to do a talk at juliacon 2016 which in part will involve whatever we end up doing with Geodesy.jl and Proj4.jl.

That's really great! I was about to ask if any of you might be interested in doing so. I'll be back in Cambridge on 23June, and will be happy to meet up.

[c42f]

I'll be back in Cambridge on 23June, and will be happy to meet up.

Great, I'm definitely keen to meet up!

[andyferris]

@yeesian Cool, I'll be coming to Juliacon also and it would be great to discuss.

[c42f]

Given that FugroRoames/Geodesy.jl has a dependence on FugroRoames/Proj4.jl, is it okay to move the latter to JuliaGeo too?

I forgot to reply to this, but yes it does make sense to move Proj4 under the JuliaGeo umbrella. We're not planning to make it a dependency for Geodesy, even though it is in some of our prototyping.