add optional coordinate epoch to nodes

[andrewharvey]

Nodes contain lat/lon in WGS84, however as pointed out https://twitter.com/nyalldawson/status/1393050234041757698 and https://geosupportsystem.se/2021/05/19/det-ror-pa-sig/ without the coordinate epoch those coordinates have an implict inaccuracy.

This is especially a problem for parts of the world where plate movement is significant, combined with higher accuracy GPS and/or availability of high resolution and high positional accuracy aerial imagery we have reached a point where OSM is and can be accurate enough to start seeing the problems of no coordinate epoch.

Each node has a timestamp, but this is the edit date, and is not necessarily the coordinate epoch.

I'm proposing we add a new epoch attribute along side existing lat, lon, timestamp, where the value shall be a date YYYY-MM-DD. The field shall be optional, and only set automatically by editors when using an imagery layer which has a known epoch, or when using a GPS trace or live GPS coordinates.

There is no need for the API to transform data into a common epoch, that can be left to downstream clients who choose to do so (optional, but improves precision).

In my opinion OSM absolutely needs to eventually support a coordinate epoch field, I understand this is a significant undertaking to modify the data format and software to support this.

[tomhughes]

Surely the general errors and inconsistencies in imagery alignment far exceed any variation related to plate movement?

This just seems to be adding vast complication in aid of a level of accuracy that is far beyond anything OSM normal achieves or can hope to achieve.

[mmd-osm]

Related discussion on the wiki: https://wiki.openstreetmap.org/wiki/API_v0.7#Node_coordinate_epoch

By the way, supporting epoch seems to be a special case of https://wiki.openstreetmap.org/wiki/API_v0.7#Better_handling_of_sourcing_.28.27source.27_tag.29

[andrewharvey]

The diary at https://www.openstreetmap.org/user/StephaneP/diary/390290 has some good detail.

Surely the general errors and inconsistencies in imagery alignment far exceed any variation related to plate movement?

In ACT, AU, the government publish aerial imagery mappers use to trace in editors. Sampling a random area it's within about 0.6m. When they changed datum the imagery is published in from 1994 to 2020 the imagery noticeably shifted 1.8m. So someone tracing features against the imagery with the old datum, now has to manually move all their nodes 1.8m to align with the new imagery.

Traditionally the OSM way would be to offset the new imagery in the editor to align with the old, but it's workaround more suited when you have poor referencing in the imagery, and even then without a coordinate epoch attribute on the node, we don't actually know where all the imagery is aligned to.

In ACT we know exactly which epoch the imagery is aligned.

[andrewharvey]

It's certainly true that for the vast majority of sources (imagery and consumer GPS) there's no use storing the epoch since the error is already larger than that from the unknown epoch, but for a small minority there is value in storing it. Imagery and consumer GPS is only going to get more accurate from here, not less so over time it will become more relevant.

[tomhughes]

Right but I don't see how it helps unless data is normalised - so I download an area and I get a whole load of objects that were added at different times from different sources.

Now even if all those objects have a date attached how does it practically help me edit?

Not to mention that two different imagery sources might be different at the same date - in fact newer dated imagery from a different source might be less accurate.

So without knowing the source as well the date tells me nothing useful.

You whole scheme is predicated on the fact that you happen to have very good imagery so you can assume it's perfectly aligned and shift data to normalise position based on date, but it seems to me that is very specific to your use case, yet you want to change the whole world on the assumption that is either true everywhere already or will be in due course.

It may be true that, in general and on average, things will get better over time but there's no guarantee that different providers will get better at the same rate, or will be starting from the same place so once you're comparing across different providers a date alone is not useful.

[andrewharvey]

Right but I don't see how it helps unless data is normalised - so I download an area and I get a whole load of objects that were added at different times from different sources. Now even if all those objects have a date attached how does it practically help me edit?

Because the editor software can take those object coordinates, and transform them all into a common epoch. If using aerial imagery they could be transformed into the same epoch as the imagery so everything aligns, or without imagery then it doesn't really matter which epoch so long as it's some epoch.

Not to mention that two different imagery sources might be different at the same date - in fact newer dated imagery from a different source might be less accurate.

Yes but all of this is only relavent for imagery sources that are accurate enough for it to make a difference and actually publish the epoch of the imagery. It's not the date the imagery was captured, it's the epoch of the imagery georeferencing. Because it's likely during the orthorectification process they transform to some fixed epoch.

So without knowing the source as well the date tells me nothing useful.

Yeah, the epoch date needs to be published by the imagery provider (if not then simply not set the epoch value), or come from a higher accuracy GPS.

You whole scheme is predicated on the fact that you happen to have very good imagery so you can assume it's perfectly aligned and shift data to normalise position based on date, but it seems to me that is very specific to your use case, yet you want to change the whole world on the assumption that is either true everywhere already or will be in due course.

Good imagery (which we do have in some regions already) and/or a GPS better than standard phone/consumer grade hardware.

It's not really changing the whole world, the epoch attribute would be optional and opt in, it's only set when either using a more accurate GPS or some compatible imagery layers, most of the uploaded nodes wouldn't use an epoch just yet.

[mmd-osm]

It's not really changing the whole world, the epoch attribute would be optional and opt in, it's only set when either using a more accurate GPS or some compatible imagery layers, most of the uploaded nodes wouldn't use an epoch just yet.

There's really no way to add an additional attribute without bumping the API version to 0.7 along the way. You would have to change pretty much every single component used in the process, including but not limited to the OSM website, CGImap, planet generation, osmdbt for minutely diff generation, and last but not least, every single editing application. Otherwise, there's no way to even fetch the additional attribute in a consistent way. Also, non-supporting editing apps might silently discard your new shiny attribute when re-uploading changed nodes.

[Firefishy]

FYI: gdal and proj now have epoch support. I've used epoch to adjusted aerial imagery for OpenStreetMap in the past. 3cm accuracy is realistically possible today on a £150 "GPS".

[andrewharvey]

There's really no way to add an additional attribute without bumping the API version to 0.7 along the way. You would have to change pretty much every single component used in the process, including but not limited to the OSM website, CGImap, planet generation, osmdbt for minutely diff generation, and last but not least, every single editing application. Otherwise, there's no way to even fetch the additional attribute in a consistent way. Also, non-supporting editing apps might silently discard your new shiny attribute when re-uploading changed nodes.

Running on the assumption that we do want to add epoch support, and we can't expect all editing software to be compatible from launch. Perhaps if the API receives a new node version from an editor, which used to have an epoch, and it's calling 0.7, then on the backend the API will retain the previous epoch. Or we could decide to only retain it if the coordinates didn't change.

That way editing software with no support for epoch will keep working, but software which does support it can still add/remove epoch as usual via an 0.8 API.

If you don't bump the API version, then I'm not sure how you'd distinguish between someone removing the epoch, or the editing software just not aware of it.

[gravitystorm]

There's definitely a problem with the slow drifting of tectonic plates, and more problems involving rectification of imagery, decade-old GPS traces etc. I'm happy to live somewhere with less drift than most places, but even here the positions of mapped features (and GPS traces) are slowly moving away from where they were when they were mapped a few years ago.

So OSM needs some way of dealing with this, whether storing epochs and letting editors + consumers deal with it (could get messy), or dealing with it centrally by nudging everything around every now and then (also could get messy).

I think it's clear that whatever approach is used will have big ramifications for a lot of parts of the ecosystem, so it's probably best to prototype this first, to make sure it works without unintended side effects. I should also point out that anything that depends on changing the API is unlikely to happen quickly - I had no end of pushback a few years ago on this topic, and I'm not surprised to see more pushback here too.

This specific proposal is that the API should allow an optional epoch attribute, but otherwise do nothing with it. Editors and consumers would be free to ignore or update it as they see fit. So my counter-proposal would be to use an epoch tag on the nodes, instead of an epoch attribute. This way you can prototype different approaches, test it out in various editors, and it's much more likely to make it through all of the standard processing chains as @mmd-osm describes. When there's widespread adoption (of whichever approach) and agreement across the community, it can, if necessary, be standardised as an attribute in a future API version.

Is there anything I'm missing that would prevent you moving forward using a tag instead of an attribute? Or any other drawbacks that I'm not considering?

[pnorman]

I think prototypes need to be done before we consider a specific solution like adding an epoch tag. You could do some investigations on the dev api with a tag to model how the information could be propagated through editors, rendering tools, etc. From that prototyping you could see if an optional epoch property solves the problems. It might need to be required, or it might not solve the problems at all.

[mmd-osm]

I'm wondering if anyone has started any prototype work on the subject. Are there any other new findings that are worth sharing?

I tend to close this issue, since it's not really actionable in its current form. It anyway needs much wider osm ecosystem involvement to move forward.