pelias / polylines

Pelias import pipeline for polyline (road network) data.
MIT License
17 stars 25 forks source link

change missinglink/pbf warning to fatal error #248

Closed missinglink closed 4 years ago

missinglink commented 4 years ago

By default, the algorithm to generate polyline data from a PBF file is the pbf streets command from my https://github.com/missinglink/pbf repo.

This algorithm is memory-bound and so doesn't scale well past a city or small country on modest hardware. I made some attempts in the past to spit out a very ugly warning message, but this warning is easy to miss.

Recently https://github.com/pelias/polylines/issues/245 highlighted that even 125GB RAM is insufficient for the pbf streets command to process the whole planet.

As such I'm going to turn this warning into a fatal error.

The current upper limit for a PBF file is 1GB, which I'm open to changing if it's shown that the pbf streets is able to handle larger PBF files on modest hardware (8-16GB RAM).

Users who wish to work on larger extracts such as North America or Planet have three options:

The planet-latest-valhalla.polylines.0sv.gz file on the Geocode Earth data download page is almost a year old, I'll go ahead and update that file now so that anyone affected by this has fresh data to work with.

If you elect to download the data from Geocode Earth then you can simply skip the pelias prepare polylines step. If you're using pelias prepare all then you'll need to replace that all step with multiple prepare steps which cover all other prepare operations other than prepare polylines.

missinglink commented 4 years ago

I've scheduled automated jobs to cut Valhalla extracts once a month for whole-country and multi-region projects (the big ones), once I've published them to http://geocode.earth/data we can merge this and update the pelias/docker projects accordingly (to download the data and skip the prepare polylines step).