protomaps / OSMExpress

Fast database file format for OpenStreetMap
BSD 2-Clause "Simplified" License
229 stars 19 forks source link

Converting full planet pbf file fail #24

Closed vkrause closed 3 years ago

vkrause commented 3 years ago

Using the pre-compiled 0.2.0 Linux binary I get the following assert when trying to load a full planet pbf file:

$ osmx expand planet-latest.osm.pbf planet.osmx
Start convert
generator planet-dump-ng 1.1.8
osmosis_replication_timestamp 2020-07-13T00:00:00Z
pbf_dense_nodes true
pbf_optional_feature_0 Has_Metadata
pbf_optional_feature_1 Sort.Type_then_ID
timestamp 2020-07-13T00:00:00Z
Box: (-180,-90,180,90)
Timestamp: 2020-07-13T00:00:00Z
Sequence#: 
Start insert
[======================================================================] 100% 
Start External sort cell_node
[======================================================================] 100% 
Finished External sort cell_node in 3596.03 seconds.
Start External sort node_way
MDB_KEYEXIST: Key/data pair already exists, file /home/ubuntu/OSMExpress/src/storage.cpp, line 157.
Aborted (core dumped)

Using a smaller extract worked fine.

bdon commented 3 years ago

I wasn't able to reproduce this using the Linux 0.2.0 binary on Ubuntu 18.04.4 LTS, with the same weekly planet dump (2020-07-13) Can you ensure you had enough disk space? For an entire planet you will need around 1TB to be safe, including tempfiles that are only being used during expand

vkrause commented 3 years ago

I tried with a self-compiled version meanwhile, with extra debug output around that assert in the hope of creating a smaller test case. That interestingly didn't trigger the problem at all, and should only differ from 0.2.0 in a few docs commits otherwise. Disk space should not be an issue, this is all running on a new 2TB SSD.

I'll try again with a self-compiled version and unmodified sources, if that also works this would seem to be rather a problem with the pre-compiled binary (not necessarily in general, but regarding compatibility with my system probably), rather than a problem in the code or the input data.

bdon commented 3 years ago

Compiling the binary and the test I ran above were on AWS x86 instances - I haven't used it outside of that specific environment, so it's possible there is some issue with the pre-compiled binary

vkrause commented 3 years ago

Tried locally compiled unmodified 0.2.0 code now, works perfectly fine as well, so it's just a weird compatibility issue of the binary. Sorry for the noise, and thanks for the great tool, performs beautifully after the initial import :)