MKuranowski / pyroutelib3

Simple routing over OpenStreetMap data
https://pyroutelib3.readthedocs.io/
GNU General Public License v3.0
79 stars 24 forks source link

Pyroutelib3 taking all my RAM #26

Closed RedBlaze42 closed 3 years ago

RedBlaze42 commented 3 years ago

I have downloaded the France osm.pbf file to calculate large itinerary offline but when I import the file in pyroutelib3, it eats up all my ram and taking forever to import Am I seeing too big ? Or am I doing something wrong with my pbf file import ?

My code is simply the one from the README:

from pyroutelib3 import Router
router = Router("car", "europe-latest.osm.pbf", localfileType="pbf")
MKuranowski commented 3 years ago

At first I suspected an issue with reading PBF files, since a single blob has to be loaded at once into memory. However the max size of a blob is 2 GiB, so that's not the issue.

On the other hand, you're saying France, but you're loading Europe. The latter is around 6.3 times bigger then the former (comparing OSM PBF file sizes).

Huge RAM usage is expected when loading every single street on a continent (or even a medium-sized country).

When doing inter-city routing you probably don't need every little highway=service/highway=living_street or even highway=residential. You should probably create a custom type for inter-city car travel, maybe looking like this:

{
    "name": "motorcar",
    "weights": {
        "motorway": 5, "trunk": 4, "primary": 2, "secondary": 1.5,
        "tertiary": 1, "unclassified": 1,
    },
    "access": ["access", "vehicle", "motor_vehicle", "motorcar"],
}
RedBlaze42 commented 3 years ago

Thank you for the quick awnser, very interesting. So I filtered out my file with bounds to only have France and Belgium, down to 7Go Then I filtered again to keep the roads like you said with this command: osmium tags-filter map_bounded.osm.pbf w/highway=motorway w/highway=trunk w/highway=primary w/highway=secondary w/highway=secondary w/highway=unclassified -o map_bounded_filtered.osm.pbf My file is now 320M !

But when I import it with the custom type you gave me, it takes ages and use 11Go of ram. Is that normal or I am doing something wrong like last time ?

Here is the link to my file if you want to test it: link