Closed knowname closed 6 months ago
Just checking -- is planet-osmium-catted.osm.pbf
a PBF that was created by running osmium renumber
?
This requirement is noted in the docs here: https://github.com/systemed/tilemaker/blob/95644d22637afb0f357d73e913dc7942641f4ce2/docs/RUNNING.md?plain=1#L50-L51 -- but I think we might actually be able to make tilemaker detect and enforce it at runtime if it sees non-sequential numbering when it's reading the PBF
Just checking -- is
planet-osmium-catted.osm.pbf
a PBF that was created by runningosmium renumber
?
Yes, sorry for the confusion. The error persists, even if the file is renumbered. I just double checked.
tilemaker --config config-debug.json --process process-debug.lua --output planet.mbtiles --store planet.tmp --compact planet-renumbered.osm.pbf
...
Store size 529G | 1/6 Block 3001/124010 terminate called after throwing an instance of 'std::out_of_range'
what(): Could not find node with id 8860665610
Command terminated by signal 6
Information on the renumbered PBF file:
osmium fileinfo -e planet-renumbered.osm.pbf
File:
Name: planet-renumbered.osm.pbf
Format: PBF
Compression: none
Size: 80383299050
Header:
Bounding boxes:
(-180,-90,180,90)
With history: no
Options:
generator=osmium/1.11.1
osmosis_replication_timestamp=2024-01-08T00:59:59Z
pbf_dense_nodes=true
pbf_optional_feature_0=Sort.Type_then_ID
sorting=Type_then_ID
timestamp=2024-01-08T00:59:59Z
[======================================================================] 100%
Data:
Bounding box: (-180,-90,180,90)
Timestamps:
First: 2005-05-21T21:03:22Z
Last: 2024-01-08T00:59:59Z
Objects ordered (by type and id): yes
Multiple versions of same object: no
CRC32: not calculated (use --crc/-c to enable)
Number of changesets: 0
Number of nodes: 8860666091
Number of ways: 992064323
Number of relations: 11728203
Smallest changeset ID: 0
Smallest node ID: 1
Smallest way ID: 1
Smallest relation ID: 1
Largest changeset ID: 0
Largest node ID: 8860666091
Largest way ID: 992064323
Largest relation ID: 11728203
All objects have following metadata attributes: version+timestamp+changeset
Some objects have following metadata attributes: all
Number of buffers: 14000446 (avg 704 objects per buffer)
Sum of buffer sizes: 879570319624 (879.57 GB)
Sum of buffer capacities: 917956198400 (917.956 GB, 96% full)
Information on the (original) planet file I am using is:
osmium fileinfo -e planet-latest.osm.pbf
File:
Name: planet-latest.osm.pbf
Format: PBF
Compression: none
Size: 77937988660
Header:
Bounding boxes:
(-180,-90,180,90)
With history: no
Options:
generator=planet-dump-ng 1.2.4
osmosis_replication_timestamp=2024-01-08T00:59:59Z
pbf_dense_nodes=true
pbf_optional_feature_0=Has_Metadata
pbf_optional_feature_1=Sort.Type_then_ID
sorting=Type_then_ID
timestamp=2024-01-08T00:59:59Z
[======================================================================] 100%
Data:
Bounding box: (-180,-90,180,90)
Timestamps:
First: 2005-05-21T21:03:22Z
Last: 2024-01-08T00:59:59Z
Objects ordered (by type and id): yes
Multiple versions of same object: no
CRC32: not calculated (use --crc/-c to enable)
Number of changesets: 0
Number of nodes: 8860666091
Number of ways: 992064323
Number of relations: 11728203
Smallest changeset ID: 0
Smallest node ID: 1
Smallest way ID: 37
Smallest relation ID: 11
Largest changeset ID: 0
Largest node ID: 11498744706
Largest way ID: 1237933295
Largest relation ID: 16998994
All objects have following metadata attributes: version+timestamp+changeset
Some objects have following metadata attributes: all
Number of buffers: 13472424 (avg 732 objects per buffer)
Sum of buffer sizes: 879570319624 (879.57 GB)
Sum of buffer capacities: 883921846272 (883.921 GB, 100% full)
Thanks for the detailed debug data! I think there are two issues here.
1) tilemaker could offer a more helpful error when invoked with --compact
on a non-renumbered file. I think this must be the case of the first error, since a node with ID 11,498,743,532 ought not be present in a renumbered file -- the max node ID should be 8,860,666,091
2) The combination of --compact
and --shard-stores
is broken. (--store
implies --shard-stores
unless overridden by --fast
)
There's actually a comment in the code that explains why the two flags don't work together... but that's not very helpful. :) Instead, the code should not use --shard-stores
when --compact
is passed. I've opened a PR to improve this.
When running tilemaker v3.0.0 (release binary on Ubuntu 22.04) with the planet on debug profile / config, I get the following error:
osmium getid
confirmed that my input file does contain NodeID 11498743532:If I drop
--compact
, it runs through without a problem. This makes me think, it might have something to do with https://github.com/systemed/tilemaker/blob/3af62f6b33d6cf0372f215b6728003824c1ac027/src/node_stores.cpp#L93-L97.