How can i navigate on off-line OSM?

[mkvasnicka]

Hi!

I need to find distances between all gas stations in the Czech Republic. Moreover, I need to do it offline. I downloaded OSM of the Czech Republic from https://download.geofabrik.de/europe/czech-republic.html in https://download.geofabrik.de/europe/czech-republic-latest-free.shp.zip file. When I unzip it, I can work with the roads via sf package.

Is it possible to use such a map in dodgr? I tried something like this:

library(sf)
library(dodgr)
library(dplyr)

OSM_FOLDER <- file.path("map_shapes", "OpenStreetMaps")

roads <- st_read(OSM_FOLDER, "gis_osm_roads_free_1", stringsAsFactors = FALSE)
car_roads <- c("motorway", "trunk", "primary", "secondary", "tertiary",
               "unclassified", "residential", "living_street",
               "motorway_link", "trunk_link", "primary_link", "secondary_link",
               "service")
roads <- roads %>%
    filter(fclass %in% car_roads) %>%
    select(-name, -ref, -layer, -bridge, -tunnel) %>%
    rename(highway = fclass)

graph <- weight_streetnet(roads, wt_profile = "motorcar")

xy <- data.frame(x = c(49.194798, 49.478480), y = c(16.583504, 17.822017), name = c("home", "kelc"))
xym <- as.matrix(xy[, 1:2])
res <- dodgr_dists(graph, from = xym, to = xym[2:1, ], wt_profile = "motorcar", parallel = FALSE)

However, I was unsuccessful: both building the graph and navigation took very long and soon consumed all my memory (8 GB) and the the path was not found (it is there in the real world).

Could you hint me how this could be done?

Best wishes, Michal

[mpadge]

The simple answer is likely to be that you just need a more powerful computer! I will have a bit of a deeper look when I find time, but you could also try downloading the .pbf instead of the .shp data, pre-processing it with osmium to extract only the desired roads, exporting in OSM/XML format, then reading with osmdata.

The big difference between GDAL (the tool used by sf::st_read()) and osmdata is that the latter retains the unique OSM ID values for all points, lines, polygons, everything, whereas GDAL/sf strips these off. This could cause problems here, because processing with dodgr is more efficient when OSM data contain these data - that is, when it is extracted with osmdata rather than st_read. But that's likely a long shot, and I suspect the short answer will remain accurate here.

Either way, you're creating a huge network with weight_streetnet, so it's no surprise that this takes a really long time. You could also try reducing the types of ways included - gas stations are unlikely to be accessible via highways with values of "tertiary", "unclassified", "residential", or "living_street". Excluding these should greatly reduce your data size. You'll nevertheless still have a huge graph, and so routing will still likely take absolutely forever.

If you really do need to do this, I'd suggest implementing your own hierarchical model, with the national-scale data containing only "motorway" and "primary" ways, and finer scales connecting each station to it's nearest node on the national-scale data.

[mkvasnicka]

Many thanks for your fast reply.

Is there any chance that if I followed the path you sketch, I would be able to construct and navigate on a network of the size of the Czech Republic on a reasonable computer (i5, 8 GB of memory) reasonably fast (doing everything say in hours)? I need to find about 100,000 routes.

If not so, I need to use another approach. It seems that Graphhopper could be twisted to do it and it's incredibly fast. However, I strongly prefer staying in R if possible.

Many thanks for your answer.

Best wishes, Michal

[mpadge]

The chances should be reasonably good if you simply remove all of the non-major roads. 100,000 routes is definitely no problem at all. 8GB of memory may be an issue. I tried osmdata, but the overpass server won't even deliver anything as large as just the primary highways, so that's out of the question here.

[mpadge]

So the osmium approach seems to work well here, but this has highlighted a number of glitches in my osmdata code. I'll keep this open until it works. Regardless, you must first install the library following the instructions here. You can then do everything within R with the following code, starting with extracting individual files for the different kinds of highways. This test case uses just three values, but you can easily expand to more.

system ("osmium tags-filter czech-republic-latest.osm.pbf w/highway=primary -o cz-primary-highways.osm.bz2"), secondary
system ("osmium tags-filter czech-republic-latest.osm.pbf w/highway=trunk -o cz-trunk-highways.osm.bz2")
system ("osmium tags-filter czech-republic-latest.osm.pbf w/highway=motorway -o cz-motorway-highways.osm.bz2")
system ("osmium tags-filter czech-republic-latest.osm.pbf w/highway=secondary -o cz-secondary-highways.osm.bz2")

Then unzip the .bz2 archives:

library (R.utils)
bunzip2 ("cz-primary-highways.osm.bz2", "cz-primary-highways.osm", remove = TRUE)
bunzip2 ("cz-trunk-highways.osm.bz2", "cz-trunk-highways.osm", remove = TRUE)
bunzip2 ("cz-motorway-highways.osm.bz2", "cz-motorway-highways.osm", remove = TRUE)
bunzip2 ("cz-secondary-highways.osm.bz2", "cz-secondary-highways.osm", remove = TRUE)

And finally read in using osmdata and join all types together with the osmdata::c method. This does not currently work, but should be fixed very soon:

library (osmdata)
primary <- osmdata_sf (doc = "cz-primary-highways.osm", quiet = FALSE)
trunk <- osmdata_sf (doc = "cz-trunk-highways.osm", quiet = FALSE)
motorway <- osmdata_sf (doc = "cz-motorway-highways.osm", quiet = FALSE)
secondary <- osmdata_sf (doc = "cz-secondary-highways.osm", quiet = FALSE)
cz <- c (primary, trunk, motorway, secondary)

[mpadge]

Okay, that commit fixed it, and cz then ends up with only 59,387 linestrings. You then need to merge all polygonal ways into the "osm_lines" item with:

cz <- osm_poly2line (cz)

That increases it to 59,606 lines, which is still totally manageable, and you shouldn't need to go any lower down in the OSM hierarchy for gas stations. (The hierarchy is given here.) The dodgr street network is then:

net <- weight_streetnet (cz$osm_lines, wt_profile = "motorcar")

and that has 845,000 edges, which should be totally manageable. It won't be fast, but should work. If you want it faster, you could remove the "secondary" ways from the above, and simply replace those with straight-line distances from gas stations to the nearest points on the primary network. That is as easy as this example:

cz <- c (primary, trunk, motorway)
cz <- osm_poly2line (cz)
net <- weight_streetnet (cz$osm_lines, wt_profile = "motorcar")
bbox <- as.numeric (strsplit (cz$bbox, " ") [[1]])
pts <- cbind (bbox [1] + runif (10) * (bbox [3] - bbox [1]),
              bbox [2] + runif (10) * (bbox [4] - bbox [2]))
colnames (pts) <- c ("x", "y")
v <- dodgr_vertices (net)
ids <- match_pts_to_graph (v, pts)
d <- diag (geodist::geodist (pts, v [ids, ]))

The vector d then gives the distances (in metres) from the points to the nearest network point, so all network distances from each item can then simply be adjusted by these distances. (The example gives rather large distances because the points are just made up, and can end up a long way from the network.) Also note that geodist gives distances in metres, while dodgr gives distances in km.

Closing for now because that should work. Feel free to re-open if you encounter any issues.

Ping @Robinlovelace: This has provided the first opportunity to work with a pretty large geofarbik pbf file, and apart from a couple of minor bugs it uncovered, all worked really well. So we can now definitely work directly with pbf files via osmium. Great progress thanks to @mkvasnicka!

[Robinlovelace]

This is great progress @mpadge - good timing as we may have a use case here this afternoon: https://github.com/pedalea/pctSantiago