Open zishanbilal opened 7 years ago
Hi @zishanbilal. It looks like OSM data contains ways referencing nodes that are not included in the OSM data. We should probably modify R5 to ignore missing nodes but the simplest solution is to provide input data with referential integrity. Providing the completeWays=yes parameter to your Osmosis bounding-box operation will ensure that all referenced nodes are included in the clipped OSM file.
Actions to be taken on the R5 side:
This should probably also close #240
Thanks for the hint @buma
No problem. I just remembered that there is some similar issue between issues.
I appreciate and even inspired by you quick response with solution. Thanks @abyrd
As an addendum to this issue, though maybe a new but highly related issue, I found that if I filter certain classes of ways from OSM using OSMOSIS, the completeWays / completeRelations don't solve the integrity problem.
R5 ended up throwing a NullPointerException on this line:
Because there is a reference to a way in a turn restriction relation that was filtered out from an OSMOSIS command that looked like this:
osmosis --read-pbf file=california-latest.osm.pbf --bounding-box top=37.7256 left=-123.4232 bottom=36.9369 right=-121.6254 --tf reject-ways highway=service completeWays=yes completeRelations=yes --write-pbf file=sf-bay.osm.pbf
I've solved the issue by no excluding any way types, but this could come up for others in the future.
Thanks!
Hi @colinsheppard, thanks for the report. Indeed we should also be checking for missing relation references. However I think this can again be remedied by using a different Osmosis command. The options completeWays=yes completeRelations=yes
do not apply to the tag filter task, they must be applied to the bounding box task. I believe if you move them before --tf reject-ways highway=service
you will get all referenced objects in your output PBF.
Thanks Andrew. I actually ran the command correctly but copy/pasted incorrectly above (b/c I pasted the command in piecemeal). When I ran the filter / clipping this way, I still got the NP due to the turn restriction relation problem:
osmosis --read-pbf file=california-latest.osm.pbf --bounding-box top=37.7256 left=-123.4232 bottom=36.9369 right=-121.6254 completeWays=yes completeRelations=yes --tf reject-ways highway=service --write-pbf file=sf-bay.osm.pbf
Note that by running R5 in debug mode, I confirmed this issue, namely that it was looking for an edge that didn't exists because the OSM data (which I looked through as XML) had a turn restriction relation referring to a way that wasn't otherwise specified.
I see. Yes, we need to make R5 robust when faced with incorrect OSM data which can occur even when it's not caused by geographic clipping.
@colinsheppard I pushed a fix for this which should hit master relatively soon. For the time being I'm guessing that the bad data might be because one of the referenced ways in the relation is tagged highway=service
and is being rejected further down the pipe.
The turn restriction problem should have been fixed in a205af8c1131c0814fe154b15dc6583b09f8890d yet it still happens. It appears that as @mattwigway said above, some ways are being filtered out further down the pipe after the relations are loaded and this is breaking the referential integrity. @demory has recently run into this on some data for the NY region.
One thing is certain: this code is in severe need of modularization. I'm looking at an if
clause that's 195 lines long.
Hmm, thanks to a205af8c1131c0814fe154b15dc6583b09f8890d we're validating that every single member of the restriction is available in the OSM data before grabbing it. Yet fetching the "to" member is yielding a null pointer. I think the only way this can happen is if someone specified a node as the "to" member. It would then be validated as being present, but would be looked up as a way (since all from and to members of turn restrictions are supposed to be ways). I'll add a check for this.
R5 fails to load clipped osm using OSMOSIS to clip a sub-region from a larger region. Where as when with i run it with OpenTripPlanner it work perfectly fine and generates Graph.obj. I run through the code to debug the scenario, below are my findings.
While loading osm
StreetLayer.loadFromOsm(osm, true, false)
when it start making edges for the way that has no intersection nodeosm.intersectionNodes.contains(way.nodes[n])
and it still go to make edgeStreetLayer.makeEdge(way, beginIdx, n, entry.getKey())
with beginIdx = 0 and n = last-index-of-nodes, it fails to find Node fromosm.nodes.get(osmNodeId)
, insideStreetLayer.getVertexIndexForOsmNode(long osmNodeId)
forbegainOsmNodeId
and throws null pointer exception as follows:To produce the scenario try building with the osm file in here that is titled Berkeley-no-service https://www.dropbox.com/sh/htfwroom0kt0khm/AADDYbB-y68O7q47j051LrkEa?dl=0