search

GIScience / osm-transform

Custom data filtering and enrichment for OpenStreetMap PBF files for a better openrouteservice graph building
3 stars 0 forks source link

Measure performance of the pre-processor #17

Closed TheGreatRefrigerator closed 6 months ago

TheGreatRefrigerator commented 9 months ago

Measure the performance of the current pre-processor https://gitlab.gistools.geog.uni-heidelberg.de/giscience/openrouteservice-infrastructure/ors-preprocessor

TheGreatRefrigerator commented 9 months ago

Hetzner server benchmark for germany: 5:06 min

Tue Jan  9 16:00:49 CET 2024
Calculating required memory...
[======================================================================] 100%
Max IDs: Node 11410764431 Way 1230426855 Relation 16812086
Allocating memory: 1360.27 Mb nodes, 146.68 Mb ways, 2.00 Mb relations

Processing first pass: validate ways & relations...
[======================================================================] 100%
valid nodes: 99094604 (392677719), valid ways: 18115289 (64026978), valid relations: 526597 (802154)
Processed in 95.737 s

Processing second pass: rebuild data...
Progress: 457506851 / 457506851 (100.00 %)
Processed in 195.255 s

Original:           4230275980 b
Reduced:            1309712247 b
Reduction:          2920563733 b (= 69.04 %)
Elevation:                0.05 % failed (45157)

./ors-preprocessor -m /opt/osm-files/bench-de.osm.pbf  1181.11s user 32.84s system 396% cpu 5:06.42 total
Tue Jan  9 16:05:55 CET 2024
TheGreatRefrigerator commented 9 months ago 
Hetzner server benchmark for planet: 1:51:15 h
Thu Jan 11 12:30:11 CET 2024
Calculating required memory...
[======================================================================] 100%
Max IDs: Node 11393203395 Way 1228537538 Relation 16782673
Allocating memory: 1358.18 Mb nodes, 146.45 Mb ways, 2.00 Mb relations

Processing first pass: validate ways & relations...
[======================================================================] 100%
valid nodes: 2530828769 (8773657450), valid ways: 270108468 (984319217), valid relations: 4815655 (11588208)
Processed in 1029.191 s

Processing second pass: rebuild data...
Progress: 9769564875 / 9769564875 (100.00 %)
Processed in 5460.121 s

Original:          77238064072 b
Reduced:           23793999356 b
Reduction:         53444064716 b (= 69.19 %)
Elevation:                2.50 % failed (63162856)

./ors-preprocessor -m /opt/osm-files/planet-latest.osm.pbf  16030.66s user 1799.43s system 267% cpu 1:51:15.74 total
Thu Jan 11 14:21:27 CET 2024