search

clarisma / gol-tool

Command-line utility for creating and managing Geographic Object Libraries
https://docs.geodesk.com/gol
GNU Affero General Public License v3.0
12 stars 3 forks source link

Collect performance statistics for `build` #50

Open clarisma opened 2 years ago

clarisma commented 2 years ago

This will help establish a baseline for future enhancements and tuning.

clarisma commented 2 years ago

10-core 32 GB: Building germany.gol from ../../mapdata/de-2021-01-29.osm.pbf using default settings... Analyzed ../../mapdata/de-2021-01-29.osm.pbf in 12s 9ms Sorted 334,239,530 nodes / 54,516,652 ways / 671,291 relations in 44s 636ms Validated 1019 tiles in 10s 330ms Compiled 1019 tiles in 1m 1s Linked 1019 tiles in 725ms Built germany.gol in 2m 12s

clarisma commented 2 years ago

10/31/22: 10-core 32 GB: Building w4.gol from ../mapdata/planet-2022-10-17.osm.pbf using default settings... 08:05:28 Minimum instance count is now 3 Analyzed ../mapdata/planet-2022-10-17.osm.pbf in 3m 11s 08:18:54 Rejected way/143887005 because of invalid node count (1)ions 08:24:27 Rejected way/1096316478 because of invalid node count (1)ons 08:24:29 Rejected way/1102695909 because of invalid node count (1)ons Sorted 7,977,707,354 nodes / 894,229,423 ways / 10,313,812 relations in 16m 43s 08:31:49 Could not calculate quad for relation/14515115 [processing 6/53/24] Validated 29433 tiles in 7m 17s Compiled 29433 tiles in 14m 28s 08:46:33 relation/14245409 has not been exported by tile 000002, can't resolve at 000000/00000124 Linked 29433 tiles in 18s 952ms Built w4.gol in 42m 4s

Planet: 70.6 GB (Recent no-meta): 51.9 GB GOL (29K tiles) 82.5 GB GOL (16K tiles) 81.9 GB Tileset (29K tiles) 59.4 GB Tileset (16K tiles) 58.9 GB

1106774150 features 202235176 nodes

Compare to w2.gol (planet form July 2020):

Planet no-meta: 44.5 GB GOL 65.0 GB Tileset (65K tiles) 46.9 GB

849433318 features 159712527 nodes

30% increase in features 27% increase in nodes

17% increase in planet size 27% increase in GOL size

Possible that new planet contains lots of trivial tagged nodes?

clarisma commented 2 years ago

10-core 32 GB: Building world.gol from ../../mapdata/planet-2022-10-17.osm.pbf using default settings... 13:46:30 Minimum instance count is now 3 Analyzed ../../mapdata/planet-2022-10-17.osm.pbf in 3m 18s 13:57:49 Rejected way/143887005 because of invalid node count (1)
14:00:38 Rejected way/1096316478 because of invalid node count (1)
14:00:39 Rejected way/1102695909 because of invalid node count (1)
Sorted 7,977,707,354 nodes / 894,229,423 ways / 10,313,812 relations in 11m 35s 14:06:52 Could not calculate quad for relation/14515115 [processing 6/53/24] Validated 29433 tiles in 6m 14s Compiled 29433 tiles in 14m 18s 14:21:26 relation/14245409 has not been exported by tile 000002, can't resolve at 000000/00000124 Linked 29433 tiles in 21s 814ms Built world.gol in 35m 58s (nothing running in bground)

clarisma commented 2 years ago

11/2/22: 2-core 8 GB

Building germany.gol from ..\mapdata\de-2021-01-29.osm.pbf using default settings... Analyzed ..\mapdata\de-2021-01-29.osm.pbf in 53s 43ms Sorted 334,239,530 nodes / 54,516,652 ways / 671,291 relations in 9m 39s Validated 1019 tiles in 55s 980ms Compiled 1019 tiles in 5m 0s Linked 1019 tiles in 3s 526ms Built germany.gol in 16m 34s

Source file size: 3,525,823,648 (no metadata) GOL size: 4,876,587,008 (38% increase) Number of tiles: 1,019 Tileset size: 3,382,918,579 (4% decrease)

clarisma commented 2 years ago

11/3/22: 2-core 8 GB (after Mappedfile change):

Building germany2.gol from ..\mapdata\de-2021-01-29.osm.pbf using default settings... Analyzed ..\mapdata\de-2021-01-29.osm.pbf in 53s 535ms Sorted 334,239,530 nodes / 54,516,652 ways / 671,291 relations in 11m 36s Validated 1019 tiles in 57s 852ms Compiled 1019 tiles in 5m 9s Linked 1019 tiles in 2s 225ms Built germany2.gol in 18m 46s

clarisma commented 2 years ago

11/3/22: 2-core 8 GB (after enhanced Mappedfile change):

Building germany3.gol from ..\mapdata\de-2021-01-29.osm.pbf using default settings... Analyzed ..\mapdata\de-2021-01-29.osm.pbf in 54s 566ms Sorted 334,239,530 nodes / 54,516,652 ways / 671,291 relations in 9m 7ss Validated 1019 tiles in 49s 832ms Compiled 1019 tiles in 5m 9s Linked 1019 tiles in 3s 123ms Built germany3.gol in 16m 9s

Fastest so far

clarisma commented 2 years ago

RC4 on 10-core 32 GB: Building world.gol from ../../mapdata/planet-2022-10-17.osm.pbf using default settings... 14:17:19 Minimum instance count is now 3 Analyzed ../../mapdata/planet-2022-10-17.osm.pbf in 3m 21s 14:29:19 Rejected way/143887005 because of invalid node count (1)
14:33:50 Rejected way/1096316478 because of invalid node count (1)
14:33:51 Rejected way/1102695909 because of invalid node count (1)
Sorted 7,977,707,354 nodes / 894,229,423 ways / 10,313,812 relations in 14m 4s 14:41:01 Could not calculate quad for relation/14515115 [processing 6/53/24] Validated 29433 tiles in 7m 6s Compiled 29433 tiles in 14m 49s 14:56:09 relation/14245409 has not been exported by tile 000002, can't resolve at 000000/00000124 Linked 29433 tiles in 30s 438ms Built world.gol in 40m 1s

clarisma commented 2 years ago

RC4 (Jul 2020 planet). 10-core 32 GB Building world-old.gol from ../../mapdata/planet.osm.pbf using default settings... 16:53:03 Minimum instance count is now 3 Analyzed ../../mapdata/planet.osm.pbf in 2m 25s Sorted 6,176,252,058 nodes / 681,804,747 ways / 7,976,914 relations in 10m 53s Validated 22801 tiles in 5m 24s Compiled 22801 tiles in 11m 37s Linked 22801 tiles in 20s 359ms Built world-old.gol in 30m 49s

clarisma commented 1 year ago

0.1.4 (Feb 2023 planet). 10-core 32 GB $ gol build world ../mapdata/planet-2023-02-19.osm.pbf -k --tag-duplicate-nodes --tag-orphan-nodes --tile-zoom-levels=0,2,4,6,8,10,12 Building world.gol from ../mapdata/planet-2023-02-19.osm.pbf using default settings... Analyzed ../mapdata/planet-2023-02-19.osm.pbf in 3m 18s Sorted 8,220,206,424 nodes / 921,092,026 ways / 10,673,673 relations in 9m 37s Validated 44641 tiles in 7m 17s Compiled 44641 tiles in 13m 26s Linked 44641 tiles in 24s 800ms Built world.gol in 34m 10s

clarisma commented 1 year ago

(Useless data, system had multiple application running)

0.1.8 (Oct 2023 planet with metadata). 10-core 32 GB (Windows 10)

Building w2.gol from E:\geodesk\mapdata\planet-2023-10-07.osm.pbf using default settings... Analyzed E:\geodesk\mapdata\planet-2023-10-07.osm.pbf in 3m 53s 15:28:09 Rejected way/197630066 because of invalid node count (1) 15:29:57 Rejected way/456150641 because of invalid node count (1) 15:30:03 Rejected way/474519355 because of invalid node count (1) 15:30:16 Rejected way/548372416 because of invalid node count (1) 15:33:20 Rejected way/1148888353 because of invalid node count (1) 15:33:20 Rejected way/1148888360 because of invalid node count (1) Sorted 8,658,483,163 nodes / 971,692,928 ways / 11,340,554 relations in 18m 31s 15:39:36 Could not calculate quad for relation/16134991 [processing 9/274/182] Validated 31721 tiles in 14m 50s Compiled 31721 tiles in 22m 29s 16:11:58 relation/15025423 has not been exported by tile 000002, can't resolve at 000000/0000034C Linked 31721 tiles in 16s 657ms Built w2.gol in 1h 0m

clarisma commented 9 months ago

0.1.8 1/31/2024 10-core 32 GB (Windows 10) Building de3.gol from e:\geodesk\mapdata\de-2024-01-31.osm.pbf using default settings... Analyzed e:\geodesk\mapdata\de-2024-01-31.osm.pbf in 14s 681ms Sorted 394,994,322 nodes / 64,369,732 ways / 807,622 relations in 49s 922ms Validated 1267 tiles in 12s 273ms Compiled 1267 tiles in 1m 20s Linked 1267 tiles in 514ms Built de3.gol in 2m 43s