Strange stray lines in import

[Komzpa]

Imported this way:

kom@nucat:~$ osmconvert BY.osm.pbf --out-osm --drop-author --drop-version  > by.osm
kom@nucat:~$ osm2pgrouting -f by.osm -d kom -W 111 -U kom

Dump copy: https://www.dropbox.com/s/w5q39je7wps6sk7/BY.osm.pbf?dl=0

osm2pgsql does not create these stray lines.

[cayetanobv]

Hi @Komzpa . What version are you using?

[Komzpa]

Hi, it was latest shipped with Ubuntu Bionic.

[Komzpa]

183 is another instance of this

[cayetanobv]

Hi @Komzpa ,

Sorry for delay in my response but last week was hard... I've imported Belarus complete dataset from Geofabrik and I don't see these strange lines in output (rendered by QGIS):

A detail of Vitesbk:

Commands used:

$ wget http://download.geofabrik.de/europe/belarus-latest.osm.pbf
$ osmconvert belarus-latest.osm.pbf --drop-author --drop-version --out-osm -o=belarus_reduc.osm
$ osm2pgrouting -c /usr/local/share/osm2pgrouting/mapconfig.xml -f /tmp/belarus_reduc.osm -d osm2pgrouting_tests -h localhost -U postgres -W postgres --schema belarus  --chunk=100000 --clean

Could you try with these data and last osm2pgrouting version?

Thanks,

[Komzpa]

Hi @cayetanobv, This ticket and #183 report the same issue separately. I shared reproducing dump. I don't need the geofabrik dump, I want to use the same dataset across all my tools. OSRM and osm2pgsql handle the dump just fine.

My guess is that osm2pgrouting can't handle nodes absent but referenced in the dump and substitutes it with dirty memory. Can you check it?

[dkastl]

I also remember, that the "Metro Extracts" caused issues, which could be the same problem: https://mapzen.com/documentation/metro-extracts/

Regarding osm2pgsql I assume, that it does not try to "correct" data as osm2pgRouting does. But like @Komzpa says, it could be a data issue with geometries that lie at the border of the extracted area.

[cvvergara]

@Komzpa

My guess is that osm2pgrouting can't handle nodes absent but referenced in the dump and substitutes it with dirty memory. Can you check it?

Your guess is almost correct.

For routing we need a cost, the cost is based on the latitude and longitude of the geometry, and speed (given in the configuration file). The geometry that osm2pgrouting is the geometry from where the length, cost, reverse_cost, cost_s, reverse_cost_s are calculated from.

here If the node is not on the node section, then it does not have the latitude and longitude, therefore its ignored (we can not calculate cost information needed for routing) so that node is not added.

osm2pgrouting focus is to build a working routing topology (not a geometry topology), based on the information that comes directly from OSM (.osm) with the following structure.

• nodes
• ways
• relations

and within each section the information is ordered by osm_id

Current Simplified Problem:

Get the edge information needed for routing

node data

id	lat	lon
12345	0	1
23456	1	1

Way "1" is a "one way" road and the nodes are 12345, 1, 20, 300, 444, 55, 6666, 23456

Based on the provided information The length of traveling from 12345 to 23456 is 1.4142

The resulting row on the edge table + the geometry of the edge from where it was calculated:

id	source	target	cost	reverse_cost	geom
1	12345	23456	1.4142	-1	LINESTRING(1 0, 1 1)
2	...

What if ...

Resulting edge table is:

id	source	target	cost	reverse_cost	geom
1	12345	23456	1.4142	-1	NULL
2	...

Benefits:

• The edge table is ready for routing and you don't see the strange lines in the graph

Prediction:

• Open New Issue: osm2pgrouting does not calculate all the edges geometry
• You can't visually detect that the graph is not the graph you need for your routing application
• Open New Issue: with some routing queries my results are in the middle of nowhere

Another What if ...

Resulting edge table is:

id	source	target	cost	reverse_cost	geom
1	12345	23456	NULL	-1	see "what if" above
2	...

Benefits

• NULL because the "complete" cost could not be calculated
• Filter the QGIS layer using WHERE cost IS NOT NULL

Prediction:

• Open Issue: The edge table is not ready for routing

When importing from osm2pgrouting I get

SELECT * FROM pgr_dijkstra(
    'SELECT gid AS id, source, target, cost FROM ways', 
    1639, 1195, directed := false);
ERROR:  Unexpected Null value in column cost
CONTEXT:  SQL function "pgr_dijkstra" statement 1

Yet Another What if ...

Resulting edge does not contain the offending edge:

id	source	target	cost	reverse_cost	geom
2	...

Benefits

• You don't see the weird straight lines

Prediction: Open Issue: I see that way "1" is in the OSM file but the edge table does not have it. Because of my application I processed the osm file removing the nodes that are irrelevant for the cost calculation.

My conclusion

We have plans to rewrite osm2pgrouting to be able to process unordered files, read this but for the moment make sure that the processed file has:

• nodes
• ways
• relations

and within each section the information is ordered by osm_id making sure that all the nodes of the ways that are going to be used for routing are in the file and make a suitable configuration file that filter out the ways that are not going to be used for routing.

If strange straight line remain

• if the line is going to be used for routing double check that the all the nodes needed are in the osm file
• if the line is not going to be used for routing:
  • option one: double check your configuration file
  • option two: delete the row from the database

[dkastl]

This ticket and #183 report the same issue separately. I shared reproducing dump. I don't need the geofabrik dump, I want to use the same dataset across all my tools. OSRM and osm2pgsql handle the dump just fine.

@Komzpa , I understand that you don't want to use the Geofabric dump. However, with their extracts I (and probably also others) have never had an issue, but with other OSM extracts we had some.

Vicky tried to understand the matter and what possible solutions exist. However, for us osm2pgrouting at the moment has not as much priority as working on version 3 of pgRouting. And since there is a way to get it work with osm2pgrouting by using Geofabric data, we would be happy to accept any pull request to fix this issue, but for us this does not have that high priority, because there is a high chance that this is a data problem and not a problem of osm2pgrouting.

Would it be better, if osm2pgrouting validates OSM extracts before running the process and refuses to do so, when data is not as expected?

[dkastl]

Maybe pgRouting could be run with an optional flag to also take into account the "geometry" of the road links ... just someone hat to implement this -> #GSOC ;-)

[cvvergara]

Graph problems are not geometry problems pgRouting functions needs: id, source, target, cost, and reverse cost pgRouting functions don't need: geom

ALTER TABLE edge_table DROP COLUMN geom and pgRouting will still work fine.

What could be done is have an extra flag in osm2pgrouting where the user choose: --no-geom don't get the geometries from where the "cost" comes from --no-bad-input don't include the rows where my data has incomplete information --stop-on-bad-input Stop everything when my data has incomplete information

And yes, whatever we decide, do it on next year GSoC program

[Komzpa]

Hello,

If nodes are skipped, why do the lines go to seemingly random locations inside the country? Just skipping them as if they never existed will make the image same as osm2pgsql does (which is a good solution).

[cvvergara]

It is are not skipping nodes that dont exist on the data. please make sure that for the lines in question, all the node information is there, that you didnt skip them

[cvvergara]

Note that the geometries returned are from where the cost is calculated from

[maciejb]

In case this is helpful to anyone looking here for same reason as me, I was running into this issue when when querying the Overpass API with the recursion method shown in most of the examples: <; Digging deeper, I changed it to the "Completed ways and relations" method: <; >; as per https://wiki.openstreetmap.org/wiki/Overpass_API/Advanced_examples#Completed_ways_and_relations and the issue is resolved.

[EPajares]

I had the same issue with a file that I produced with osmosis using the version 2.3.3 of osm2pgrouting. With the compiled version 2.2.0 of osm2pgrouting it worked as it should. So maybe this could be useful for somebody. Maybe this is helpful for somebody.

[cyang-kth]

I had the same issue

I think one case is when it tries to split a long edge into shorter ones according to the intersections.

The original OSM link is shown below. https://www.openstreetmap.org/way/29033314#map=17/61.72805/17.11454