Downloading and extracting planet fails - Seems like out of memory?

[ppapasav]

I've been trying to extract the entire planet OSRM - Server details

Dedicated High speed server with 256GB ram running VMWare - Here's what the server looks like

240GB Ram 500GB SSD 10G San 32 CPUs (Cores) 50GB Swap file (We increased it to this as it failed earlier) Latest Ubuntu Runner on Docker

Should we increase the swap file more? looking at HTOP it was near max memory allocation when it stopped

Would it be easier to download individual continents and merge them? Never did a merge before

It ran about 18 hours and stopped here:

[info] Parsed 0 location-dependent features with 0 GeoJSON polygons [info] Using script /data/car.lua [info] Input file: planet-latest.osm.pbf [info] Profile: car.lua [info] Threads: 46 [info] Parsing in progress.. [info] input file generated by planet-dump-ng 1.2.0 [info] timestamp: 2021-08-15T23:59:59Z [info] Using profile api version 4 [info] Found 3 turn restriction tags: [info] motorcar [info] motor_vehicle [info] vehicle [info] Parse relations ... [info] Parse ways and nodes ... [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Using profile api version 4 [info] Parsing finished after 2402.69 seconds [info] Raw input contains 2848195231 nodes, 793758380 ways, and 793210 relations, 1537333 restrictions [info] Collecting way information on 1537333 restrictions...ok, after 27.2052s [info] Collecting way information on 64 maneuver overrides...ok, after 2.74886s [info] Sorting used nodes ... ok, after 13.5848s [info] Erasing duplicate nodes ... ok, after 4.58028s [info] Sorting all nodes ... ok, after 36.7345s [info] Building node id map ... ok, after 25.5128s [info] Confirming/Writing used nodes ... ok, after 121.402s [info] Writing barrier nodes ... ok, after 0s [info] Writing traffic light nodes ... ok, after 0s [info] Processed 1269730159 nodes [info] Sorting edges by start ... ok, after 87.5055s [info] Setting start coords ... ok, after 147.634s [info] Sorting edges by target ... ok, after 47.3467s [info] Computing edge weights ... ok, after 396.532s [info] Sorting edges by renumbered start ... ok, after 67.0465s [info] Writing used edges ... ok, after 125.35s -- Processed 1330919485 edges [info] Writing way meta-data ... ok, after 0.981208s -- Metadata contains << 139130478 entries. [info] Collecting node information on 64 maneuver overrides...ok, after 0.002542s [info] Collecting node information on 1537333 restrictions...ok, after 7.31182s [info] writing street name index ... ok, after 0.847032s [info] extraction finished after 3561.52s [info] Generating edge-expanded graph representation [info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100% [info] Node compression ratio: 0.169992 [info] Edge compression ratio: 0.207294 [info] graph compression removed 129090303 annotations of 139130478 in 1188 seconds [info] Find segregated edges in node-based graph ... [info] ok, after 19113.2s [info] Segregated edges count = 4376656 [info] Writing nodes for nodes-based and edges-based graphs ... [warn] Clipped 68 segment weights to 4194302 [info] Geometry successfully removed: compressed edges: 551187400 compressed geometries: 2660029526 longest chain length: 8297 cmpr ratio: 0.207211 avg chain length: 4.826 [info] Removing invalid restrictions...removed 15441 invalid restrictions, after 22.7281s [info] Constructing restriction graph on 1488010 restrictions...ok, after 2.08322s [info] Generating edge expanded nodes ... [info] . 10%

[mjjbell]

Yes, you'll need at least 500GB to process planet-latest Depending on your use-case, it may be easier to process smaller OSM files and interact with them independently. There's no concept of "merging" OSRM files from different areas.

[merschformann]

Since I just ran a successful planet build, I thought I share some insights. (only ran extract & contract though)

OSRM: v5.26.0 Hardware: EC2 instance (r5d.16xlarge) Planet file: planet-211018.osm.pbf Resulting file size on disk (generated files + osm.pbf): 317 GB

RAM consumption over time:

Steps shown in above graph:

sudo docker run --rm -t -v "/mnt/data1/planet:/data" osrm/osrm-backend osrm-extract -p /opt/car.lua /data/planet-211018.osm.pbf
sudo docker run --rm -t -v "/mnt/data1/planet:/data" osrm/osrm-backend osrm-contract /data/planet-211018.osrm

The good news is that memory for the pre-built/running instance seems to stay well below the 200 GB (specifically the 256 GB mark).

[merschformann]

I collected more details and shared them here, in case someone is interested. :relaxed:

[Gamma-Software]

Hey @merschformann, Any chance I can download your extracted planet.osrm ?

[merschformann]

Hey @Gamma-Software ! I don't have the files of that pass anymore, but I could upload the ones of a more recent one. They are quite big though. What do you need exactly?

[github-actions[bot]]

This issue seems to be stale. It will be closed in 30 days if no further activity occurs.