lat/lon = 214.7483647, xml-reader has problem with redacted nodes

MaZderMind commented 11 years ago

Hi

Brian DeRocher pointed me at an issue where strange lines occured on his history renderings.I followed the path of the bad data and found, that when reading the current history_2013-02-05_1701.osh.pbf you'll get millions of nodes with the invalid coordinate lon=214.7483647 lat=214.7483647

./osmium_debug history_2013-02-05_1701.osh.pbf | grep 214.748 -C 5 | less

gives us

node:
  id=18553402
  version=3
  uid=44659
  user=|Phil Shipley|
  changeset=525049
  timestamp=2008-09-04T21:14:53Z
  visible=no
  endtime=
  tags: (count=0)
  lon=214.7483647
  lat=214.7483647
node:
  id=18553402
  version=4
  uid=44659
  user=|Phil Shipley|
  changeset=525049
  timestamp=2008-09-04T21:14:55Z
  visible=no
  endtime=
  tags: (count=0)
  lon=214.7483647
  lat=214.7483647

The problem does not occur when reading the respective osh.bz2 file. Both files are from http://osm.personalwerk.de/full-experimental/ (the osh.bz2 beeing a copy of planet.osm.prg).

Taking a look at those specific node-versions shows, that at those node-versions don't have a location, because the redaction bot fiddled out, that the locations originate from a decliners account, while the tags are from an acceptor. So the full-history-dump contains nodes with tags but without (valid) coordinates.

Those nodes are represented in xml as

<node id="18553402" version="4" changeset="525049" user="Phil Shipley" uid="44659" visible="false" timestamp="2008-09-04T21:14:55Z">

Which irritates the xml reader. As a small testcase I downloaded the respective way-history-xml from the api and tried to process it with osmium:

wget http://www.openstreetmap.org/api/0.6/node/18553402/history -O 18553402-history.osh

osmium_debug 18553402-history.osh

meta:
  generator=OpenStreetMap server
before_nodes
node:
  id=18553402
  version=3
  uid=44659
  user=|Phil Shipley|
  changeset=525049
  timestamp=2008-09-04T21:14:53Z
  visible=no
  endtime=
  tags: (count=2)
    k=|created_by| v=|r_coastlines|
    k=|source| v=|PGS|
  lon=214.7483647
  lat=214.7483647
node:
  id=18553402
  version=4
  uid=44659
  user=|Phil Shipley|
  changeset=525049
  timestamp=2008-09-04T21:14:55Z
  visible=no
  endtime=
  tags: (count=2)
    k=|created_by| v=|r_coastlines|
    k=|source| v=|PGS|
  lon=214.7483647
  lat=214.7483647
node:
  id=18553402
  version=7
  uid=722137
  user=|OSMF Redaction Account|
  changeset=12298830
  timestamp=2012-07-18T21:14:53Z
  visible=yes
  endtime=
  tags: (count=0)
  lon=-79.0652769
  lat=43.0506978
after_nodes
final

I'd be happily providing a patch, when I just hat an idea on how a library like osmium should work with such crippled nodes.

Just dropping is sure not an option, because there may be applications out there that want to deal on their own with this kind of nodes.
Adding another "Hi-I'm-a-node-but-I-do-not-carry-a-coordinate"-Flag would probably be most streight forward, but there are numerous issues arising with that
- Is there such a flag in the pbf format?
- How is a software like the history-spitter supposed to handle nodes without the knowledge if it is in- our outside of the bbox/polygon?
- Are there nodes with no coords and no tags? Can they be dropped?

Those issues should probably be discussed over at dev@

woodpeck commented 11 years ago

They have been (mentioned on dev), http://lists.openstreetmap.org/pipermail/dev/2012-October/026049.html

MaZderMind commented 11 years ago

Oh, thank you. I was not aware of the node->position().defined() method which enables my client code to ignore those nodes (they can't be rendered on a map anyway). As osmium offers a way to handle this gracefully, it's probably not an issue with osmium, so I'll close this ticket.

openbrian commented 11 years ago

Thanks for the help ;)

joto / osmium

lat/lon = 214.7483647, xml-reader has problem with redacted nodes #64