Closed missinglink closed 4 years ago
I've scheduled automated jobs to cut Valhalla extracts once a month for whole-country
and multi-region
projects (the big ones), once I've published them to http://geocode.earth/data we can merge this and update the pelias/docker
projects accordingly (to download the data and skip the prepare polylines
step).
By default, the algorithm to generate polyline data from a PBF file is the
pbf streets
command from my https://github.com/missinglink/pbf repo.This algorithm is memory-bound and so doesn't scale well past a city or small country on modest hardware. I made some attempts in the past to spit out a very ugly warning message, but this warning is easy to miss.
Recently https://github.com/pelias/polylines/issues/245 highlighted that even 125GB RAM is insufficient for the
pbf streets
command to process the whole planet.As such I'm going to turn this warning into a fatal error.
The current upper limit for a PBF file is 1GB, which I'm open to changing if it's shown that the
pbf streets
is able to handle larger PBF files on modest hardware (8-16GB RAM).Users who wish to work on larger extracts such as
North America
orPlanet
have three options:precision=6
is the default).The
planet-latest-valhalla.polylines.0sv.gz
file on the Geocode Earth data download page is almost a year old, I'll go ahead and update that file now so that anyone affected by this has fresh data to work with.If you elect to download the data from Geocode Earth then you can simply skip the
pelias prepare polylines
step. If you're usingpelias prepare all
then you'll need to replace thatall
step with multipleprepare
steps which cover all other prepare operations other thanprepare polylines
.