search

systemed / tilemaker

Make OpenStreetMap vector tiles without the stack
https://tilemaker.org/
Other
1.46k stars 230 forks source link

Assertion failed: (has_wire_type(pbf_wire_type::varint) && "not a varint"), function get_sint64, file pbf_reader.hpp, line 570 #696

Closed saevioapps closed 6 months ago

saevioapps commented 6 months ago

I am trying to generate tiles from multiple input files, and i get this error when reading the second input file file:

tilemaker --input mapdata/africa-output.osm.pbf \ --input mapdata/asia-output.osm.pbf \ --input mapdata/australia-oceania-output.osm.pbf \ --input mapdata/central-america-output.osm.pbf \ --input mapdata/europe-output.osm.pbf \ --input mapdata/north-america-output.osm.pbf \ --input mapdata/south-america-output.osm.pbf \ --output tilemaker-output/tiles/ \ --config tilemaker-config/config.json \ --process tilemaker-config/process.lua \

Layer countries (z0-14) Layer place (z0-14) Layer boundary (z0-14) Layer poi (z12-14) Layer poi_detail (z14-14) -> poi Layer housenumber (z14-14) Layer waterway (z7-14) Layer waterway_detail (z12-14) -> waterway Layer transportation (z4-12) Layer transportation_name (z8-14) Layer transportation_name_mid (z12-14) -> transportation_name Layer transportation_name_detail (z14-14) -> transportation_name Layer building (z13-14) Layer water (z6-14) Layer ocean (z0-14) -> water Layer water_name (z14-14) Layer water_name_detail (z14-14) -> water_name Layer aeroway (z11-14) Layer aerodrome_label (z10-14) Layer park (z6-14) Layer landuse (z4-14) Layer urban_areas (z4-8) -> landuse Layer landcover (z0-14) Layer ice_shelf (z0-9) -> landcover Layer glacier (z2-9) -> landcover Layer mountain_peak (z11-14) Bounding box -27.262, -60.3167, 66.7228, 37.7782 Reading shapefile countries Reading shapefile ocean Reading shapefile urban_areas Unable to open mapdata/ne_10m_urban_areas/ne_10m_urban_areas.shp or mapdata/ne_10m_urban_areas/ne_10m_urban_areas.SHP in rb mode. Reading shapefile ice_shelf Unable to open landcover/ne_10m_antarctic_ice_shelves_polys/ne_10m_antarctic_ice_shelves_polys.shp or landcover/ne_10m_antarctic_ice_shelves_polys/ne_10m_antarctic_ice_shelves_polys.SHP in rb mode. Reading shapefile glacier Unable to open landcover/ne_10m_glaciated_areas/ne_10m_glaciated_areas.shp or landcover/ne_10m_glaciated_areas/ne_10m_glaciated_areas.SHP in rb mode. Generated points: 0, lines: 0, polygons: 6943 Reading .pbf mapdata/africa-output.osm.pbf (Scanning for ways used in relations: 80%) (226 ms) Block 7944/7945 (3242 ms) SortedNodeStore: 155743 groups, 2137014 chunks, 63549997 nodes, 412803356 bytes (5.3% wasted) Block 138/139 (9158 ms) SortedWayStore: 18635 groups, 286325 chunks, 951609 ways, 60053795 nodes, 174128662 bytes only 5 relation blocks; subdividing for better parallelism Block 39/40 (131454 ms) Reading .pbf mapdata/asia-output.osm.pbf Assertion failed: (has_wire_type(pbf_wire_type::varint) && "not a varint"), function get_sint64, file pbf_reader.hpp, line 570. zsh: abort tilemaker --input mapdata/africa-output.osm.pbf --input --input --input

Could this be a RAM issue(i am on a low RAM machine now)? If i run with only one input file it works.

systemed commented 6 months ago

Yep, confirmed. Not a RAM issue, it happens even on a big machine. Stacktrace is:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f2c569ac859 in __GI_abort () at abort.c:79
#2  0x00007f2c569ac729 in __assert_fail_base (fmt=0x7f2c56b42588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55f62332bd60 "has_wire_type(pbf_wire_type::varint) && \"not a varint\"", 
    file=0x55f62332afa0 "./include/protozero/pbf_reader.hpp", line=570, function=<optimised out>) at assert.c:92
#3  0x00007f2c569bdfd6 in __GI___assert_fail (assertion=0x55f62332bd60 "has_wire_type(pbf_wire_type::varint) && \"not a varint\"", file=0x55f62332afa0 "./include/protozero/pbf_reader.hpp", line=570, 
    function=0x55f62332bd98 "int64_t protozero::pbf_reader::get_sint64()") at assert.c:101
#4  0x000055f6231fac67 in PbfReader::PbfReader::readHeaderBBox(protozero::data_view) ()
#5  0x000055f6231fbbca in PbfReader::PbfReader::readHeaderBlock(protozero::data_view) ()
#6  0x000055f6231fc93e in PbfReader::PbfReader::readHeaderFromFile(std::istream&) ()
#7  0x000055f6231f3791 in PbfProcessor::ReadPbfFile(unsigned int, bool, SignificantTags const&, SignificantTags const&, unsigned int, std::function<std::shared_ptr<std::istream> ()> const&, std::function<std::shared_ptr<OsmLuaProcessing> ()> const&, NodeStore const&, WayStore const&) ()
#8  0x000055f62302a662 in main ()
cldellow commented 6 months ago

Hi @saevioapps, I think this was a regression in my v3 work.

Can you try https://github.com/systemed/tilemaker/pull/701 and see if it works for you? I expect that it should work, but it won't be using many of the recent improvements, so it will use more memory and be slower than if you had the same PBFs in a single file.