[BUG] Planetiler reaches a max filesize error, no information online about the cause.

[HomingHamster]

Describe the bug A clear and concise description of what the bug is.

To Reproduce java -Xmx20g -jar planetiler.jar --area=planet --bounds=planet ----osm-path=/mnt/ele/nominatim-data/planet-latest.osm.pbf --download --download-threads=10 --download-chunk-size-mb=1000 --fetch-wikidata --output=/mnt/ele/output.mbtiles --nodemap-type=array --storage=mmap

(the same behaviour is noted when asking planetiler to download its own planet.osm) (there is space on the hard drive, but i couldnt check 100% if a temp file on the boot disk was being used)

System specs are 64gb ram 6core 3.5ghz processor, 1tb boot drive with 200gb free, 5tb ntfs-3g disk mounted with user permissions enabled at /mnt/ele/ (4tb free) that I am trying to build on, fast working internet.

Expected behavior output.mbtiles to be generated successfully for the planet.

Screenshots felix@felix-B550-UD-AC-Y1:/mnt/ele/planetiler$ java -Xmx20g -jar planetiler.jar --area=planet --bounds=planet ----osm-path=/mnt/ele/nominatim-data/planet-latest.osm.pbf --download --download-threads=10 --download-chunk-size-mb=1000 --fetch-wikidata --output=/mnt/ele/output.mbtiles --nodemap-type=array --storage=mmap 0:00:00 DEB - argument: config=null (path to config file) 0:00:00 DEB - argument: download_dir=data/sources (download directory) 0:00:00 DEB - argument: area=planet (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet')) 0:00:00 INF - argument: stats=use in-memory stats 0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage) 0:00:00 DEB - argument: storage=mmap (default storage type for temporary data, one of [ram, mmap, direct]) 0:00:00 DEB - argument: threads=12 (num threads) 0:00:00 DEB - argument: write_threads=1 (number of threads to use when writing temp features) 0:00:00 DEB - argument: process_threads=11 (number of threads to use when processing input features) 0:00:00 DEB - argument: bounds=Env[-180.0 : 180.0, -85.0511287798066 : 85.0511287798066] (bounds) 0:00:00 DEB - argument: polygon=null (a .poly file that limits output to tiles intersecting the shape) 0:00:00 DEB - argument: minzoom=0 (minimum zoom level) 0:00:00 DEB - argument: maxzoom=14 (maximum zoom level up to 15) 0:00:00 DEB - argument: render_maxzoom=14 (maximum rendering zoom level up to 15) 0:00:00 DEB - argument: tmpdir=data/tmp (temp directory) 0:00:00 DEB - argument: feature_read_threads=1 (number of threads to use when reading features at tile write time) 0:00:00 DEB - argument: tile_write_threads=1 (number of threads used to write tiles - only supported by [files, csv, tsv, proto, pbf, json]) 0:00:00 DEB - argument: loginterval=10 seconds (time between logs) 0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings) 0:00:00 DEB - argument: append=false (append to the output file - only supported by [files, csv, tsv, proto, pbf, json]) 0:00:00 DEB - argument: compress_temp=false (compress temporary feature storage (uses more CPU, but less disk space)) 0:00:00 DEB - argument: mmap_temp=true (use memory-mapped IO for temp feature files) 0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks) 0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks) 0:00:00 DEB - argument: nodemap_type=array (type of node location map, one of [noop, sortedtable, sparsearray, array]) 0:00:00 DEB - argument: nodemap_storage=mmap (storage for node location map, one of [ram, mmap, direct]) 0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations) 0:00:00 DEB - argument: multipolygon_geometry_storage=mmap (storage for multipolygon geometries, one of [ram, mmap, direct]) 0:00:00 DEB - argument: multipolygon_geometry_madvise=true (use linux madvise(random) for temporary multipolygon geometry storage) 0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP) 0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP) 0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP) 0:00:00 DEB - argument: http_retry_wait=5 seconds (How long to wait before retrying HTTP request) 0:00:00 DEB - argument: download_chunk_size_mb=1000 (Size of file chunks to download in parallel in megabytes) 0:00:00 DEB - argument: download_threads=10 (Number of parallel threads to use when downloading each file) 0:00:00 DEB - argument: download_max_bandwidth= (Maximum bandwidth to consume when downloading files in units mb/s, mbps, kbps, etc.) 0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming) 0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level) 0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming) 0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level) 0:00:00 DEB - argument: osm_lazy_reads=true (Read OSM blocks from disk in worker threads) 0:00:00 DEB - argument: skip_filled_tiles=false (Skip writing tiles containing only polygon fills to the output) 0:00:00 DEB - argument: tile_warning_size_mb=1.0 (Maximum size in megabytes of a tile to emit a warning about) 0:00:00 DEB - argument: color=null (Color the terminal output) 0:00:00 DEB - argument: keep_unzipped=false (keep unzipped sources by default after reading) 0:00:00 DEB - argument: tile_compression=gzip (the tile compression, one of [gzip, none]) 0:00:00 DEB - argument: output_layerstats=false (output a tsv.gz file for each tile/layer size) 0:00:00 DEB - argument: debug_url=https://onthegomap.github.io/planetiler-demo/#{z}/{lat}/{lon} (debug url to use for displaying tiles with {z} {lat} {lon} placeholders) 0:00:00 DEB - argument: tile_weights=data/tile_weights.tsv.gz (tsv.gz file with columns z,x,y,loads to generate weighted average tile size stat) 0:00:00 DEB - argument: max_point_buffer=Infinity (Max tile pixels to include points outside tile bounds. Set to a lower value to reduce tile size for clients that handle label collisions across tiles (most web and native clients). NOTE: Do not reduce if you need to support raster tile rendering) 0:00:00 DEB - argument: log_jts_exceptions=false (Emit verbose details to debug JTS geometry errors) 0:00:00 DEB - argument: feature_source_id_multiplier=10 (Set vector tile feature IDs to (featureId * thisValue) + sourceId where sourceId is 1 for OSM nodes, 2 for ways, 3 for relations, and 0 for other sources. Set to false to disable.) 0:00:00 DEB - argument: only_download=false (download source data then exit) 0:00:00 DEB - argument: tests=null (run test cases in a yaml then quit) 0:00:00 DEB - argument: download=true (download sources) 0:00:00 DEB - argument: refresh_sources=false (download new version of source files if they have changed) 0:00:00 DEB - argument: download_osm_tile_weights=true (download OSM tile weights file) 0:00:00 DEB - argument: temp_nodes=data/tmp/node.db (temp node db location) 0:00:00 DEB - argument: temp_multipolygons=data/tmp/multipolygon.db (temp multipolygon db location) 0:00:00 DEB - argument: temp_features=data/tmp/feature.db (temp feature db location) 0:00:00 DEB - argument: osm_parse_node_bounds=false (parse bounds from OSM nodes instead of header) 0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit) 0:00:00 DEB - argument: fetch_wikidata=true (fetch wikidata translations then continue) 0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file) 0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path) 0:00:00 DEB - argument: refresh_lake_centerlines=false (Download new version of lake_centerlines if changed) 0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: lake_centerlines_url=https://dev.maptiler.download/geodata/omt/lake_centerline.shp.zip (lake_centerlines shapefile url) 0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path) 0:00:00 DEB - argument: refresh_water_polygons=false (Download new version of water_polygons if changed) 0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: water_polygons_url=https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip (water_polygons shapefile url) 0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path) 0:00:00 DEB - argument: refresh_natural_earth=false (Download new version of natural_earth if changed) 0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: natural_earth_url=https://dev.maptiler.download/geodata/omt/natural_earth_vector.sqlite.zip (natural_earth sqlite db url) 0:00:00 DEB - argument: natural_earth_keep_unzipped=false (keep unzipped natural_earth after reading) 0:00:00 DEB - argument: osm_path=/mnt/ele/nominatim-data/planet-latest.osm.pbf (osm OSM input file path) 0:00:00 DEB - argument: refresh_osm=false (Download new version of osm if changed) 0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: osm_url=aws:latest (osm OSM input file url) 0:00:00 DEB - argument: output=/mnt/ele/output.mbtiles (output tile archive path) 0:00:00 DEB - argument: version=false (show version then exit) 0:00:00 INF - Planetiler build git hash: a74f274c3162cd547c9b19a9a5e8705982a61755 0:00:00 INF - Planetiler build version: 0.8.2 0:00:00 INF - Planetiler build timestamp: 2024-07-02T11:38:28.480Z 0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names) 0:00:00 DEB - argument: languages=am,ar,az,be,bg,bn,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fa,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pa,pnb,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,ur,vi,zh,zh-Hant,zh-Hans (languages to use) 0:00:00 DEB - argument: only_layers= (Include only certain layers) 0:00:00 DEB - argument: exclude_layers= (Exclude certain layers) 0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries) 0:00:00 DEB - argument: boundary_osm_only=false (boundary layer: only use OSM, even at low zoom levels) 0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13) 0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13) 0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways) 0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield)) 0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways) 0:00:00 DEB - argument: transportation_name_minor_refs=false (transportation_name layer: include name and refs from minor road networks if not present on a way) 0:00:00 DEB - argument: help=false (show arguments then exit) 0:00:01 DEB - argument: layer_stats=/mnt/ele/output.mbtiles.layerstats.tsv.gz (layer stats output path) 0:00:01 INF - Building OpenMapTilesProfile profile into file:///mnt/ele/output.mbtiles in these phases: 0:00:01 INF - wikidata: Fetch translations from wikidata query service 0:00:01 INF - lake_centerlines: Process features in data/sources/lake_centerline.shp.zip 0:00:01 INF - water_polygons: Process features in data/sources/water-polygons-split-3857.zip 0:00:01 INF - natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip 0:00:01 INF - osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members) 0:00:01 INF - osm_pass2: Process OpenStreetMap nodes, ways, then relations 0:00:01 INF - sort: Sort rendered features by tile ID 0:00:01 INF - archive: Encode each tile and write to TileArchiveConfig[format=MBTILES, scheme=FILE, uri=file:///mnt/ele/output.mbtiles, options={}] 0:00:01 INF [wikidata] - 0:00:01 INF [wikidata] - Starting... 0:00:01 INF [wikidata] - Starting with 11 process threads 0:00:12 INF [wikidata] - loaded from 1006838 mappings from /mnt/ele/planetiler/data/sources/wikidata_names.json in 12s cpu:13s avg:1.1 0:00:13 INF [wikidata] - skipping 1006838 mappings we already have 0:00:28 INF [wikidata] - blocks: [ 63 6/s ] nodes: [ 13M 1.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.2 gc: 1% heap: 1.8G/21G direct: 32k postGC: 1.5G parse() -> pbf( 1%) -> (23/34) -> filter( 8% 9% 8% 7% 8% 9% 8% 9% 9% 8% 9%) -> (565/1M) -> fetch( 0%) 0:00:40 INF [wikidata] - blocks: [ 135 7/s ] nodes: [ 28M 1.5M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.8 gc: 0% heap: 2G/21G direct: 32k postGC: 1.6G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 5% 5% 5% 5% 5% 5% 5% 5% 5% 5%) -> (652/1M) -> fetch( 0%) 0:00:50 INF [wikidata] - blocks: [ 234 8/s ] nodes: [ 49M 1.8M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1 gc: 0% heap: 4G/21G direct: 32k postGC: 3.2G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 6% 5% 6% 6% 6% 6% 6% 6% 6% 6%) -> (552/1M) -> fetch( 0%) 0:01:01 INF [wikidata] - blocks: [ 353 11/s ] nodes: [ 74M 2.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.2 gc: 1% heap: 2.3G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 7% 7% 7% 7% 7% 6% 7% 7% 7% 8% 7%) -> (566/1M) -> fetch( 0%) 0:01:11 INF [wikidata] - blocks: [ 478 11/s ] nodes: [ 100M 2.4M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.2 gc: 0% heap: 3.8G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 8% 8% 7% 8% 8% 8% 7% 8% 7% 7% 7%) -> (440/1M) -> fetch( 0%) 0:01:22 INF [wikidata] - blocks: [ 599 11/s ] nodes: [ 126M 2.4M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.1 gc: 0% heap: 2.7G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 7% 6% 7% 7% 7% 7% 7% 8% 7% 7% 7%) -> (503/1M) -> fetch( 0%) 0:01:32 INF [wikidata] - blocks: [ 727 12/s ] nodes: [ 153M 2.5M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1 gc: 0% heap: 3.8G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 7% 7% 6% 7% 7% 7% 6% 6% 6% 7%) -> (610/1M) -> fetch( 0%) 0:01:43 INF [wikidata] - blocks: [ 835 10/s ] nodes: [ 177M 2.1M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.3 gc: 0% heap: 2.4G/21G direct: 32k postGC: 2.4G parse() -> pbf( 0%) -> (23/34) -> filter( 8% 8% 8% 8% 7% 8% 8% 8% 8% 8% 7%) -> (629/1M) -> fetch( 0%) 0:01:53 INF [wikidata] - blocks: [ 938 10/s ] nodes: [ 199M 2.2M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.2 gc: 0% heap: 3.2G/21G direct: 32k postGC: 3.1G parse() -> pbf( 0%) -> (23/34) -> filter( 7% 7% 7% 8% 6% 7% 7% 6% 7% 6% 7%) -> (498/1M) -> fetch( 0%) 0:02:04 INF [wikidata] - blocks: [ 1k 10/s ] nodes: [ 223M 2.1M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.2 gc: 0% heap: 1.9G/21G direct: 32k postGC: 1.5G parse() -> pbf( 0%) -> (23/34) -> filter( 7% 6% 6% 8% 7% 7% 8% 7% 6% 6% 8%) -> (446/1M) -> fetch( 0%) 0:02:14 INF [wikidata] - blocks: [ 1.1k 10/s ] nodes: [ 246M 2.2M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.2 gc: 0% heap: 3.4G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 8% 7% 8% 7% 8% 8% 8% 7% 8% 7% 8%) -> (602/1M) -> fetch( 0%) 0:02:25 INF [wikidata] - blocks: [ 1.2k 12/s ] nodes: [ 280M 3.2M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.3 gc: 0% heap: 2.3G/21G direct: 32k postGC: 2.3G parse() -> pbf( 0%) -> (23/34) -> filter( 7% 8% 7% 8% 9% 8% 8% 8% 9% 9% 8%) -> (561/1M) -> fetch( 0%) 0:02:36 INF [wikidata] - blocks: [ 1.4k 11/s ] nodes: [ 310M 2.8M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.4 gc: 0% heap: 2.7G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 8% 8% 7% 8% 8% 8% 7% 7% 8% 7% 8%) -> (638/1M) -> fetch( 0%) 0:02:46 INF [wikidata] - blocks: [ 1.5k 11/s ] nodes: [ 340M 2.8M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 1.2 gc: 0% heap: 2.9G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 8% 8% 8% 7% 8% 9% 9% 9% 7% 8% 9%) -> (599/1M) -> fetch( 0%) 0:02:57 INF [wikidata] - blocks: [ 1.6k 13/s ] nodes: [ 375M 3.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.9 gc: 0% heap: 5.5G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 7% 7% 7% 7% 8% 8% 9% 8% 7% 8% 7%) -> (585/1M) -> fetch( 0%) 0:03:07 INF [wikidata] - blocks: [ 1.8k 13/s ] nodes: [ 410M 3.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.9 gc: 0% heap: 3.9G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 9% 8% 8% 8% 8% 7% 8% 8% 8% 8% 7%) -> (445/1M) -> fetch( 0%) 0:03:18 INF [wikidata] - blocks: [ 1.9k 12/s ] nodes: [ 443M 3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.8 gc: 0% heap: 6G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 7% 8% 6% 7% 8% 7% 7% 7% 8% 6% 8%) -> (503/1M) -> fetch( 0%) 0:03:28 INF [wikidata] - blocks: [ 2k 11/s ] nodes: [ 469M 2.4M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 6G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 6% 6% 5% 6% 6% 6% 6% 6% 7% 6%) -> (314/1M) -> fetch( 0%) 0:03:39 INF [wikidata] - blocks: [ 2.1k 11/s ] nodes: [ 496M 2.5M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 1.4G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 7% 5% 6% 6% 6% 7% 6% 6% 6% 6%) -> (444/1M) -> fetch( 0%) 0:03:49 INF [wikidata] - blocks: [ 2.2k 11/s ] nodes: [ 523M 2.5M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 2G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (18/34) -> filter( 6% 6% 6% 6% 5% 6% 6% 6% 6% 6% 6%) -> (576/1M) -> fetch( 0%) 0:04:00 INF [wikidata] - blocks: [ 2.4k 11/s ] nodes: [ 550M 2.5M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.6 gc: 0% heap: 1.3G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 6% 5% 6% 6% 5% 4% 5% 6% 5% 5%) -> (523/1M) -> fetch( 0%) 0:04:11 INF [wikidata] - blocks: [ 2.5k 10/s ] nodes: [ 574M 2.2M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.6 gc: 0% heap: 5.7G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (21/34) -> filter( 6% 5% 5% 6% 6% 5% 5% 5% 5% 6% 6%) -> (409/1M) -> fetch( 0%) 0:04:22 INF [wikidata] - blocks: [ 2.6k 10/s ] nodes: [ 600M 2.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 5.6G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 6% 6% 6% 6% 7% 6% 5% 5% 6% 6%) -> (621/1M) -> fetch( 0%) 0:04:32 INF [wikidata] - blocks: [ 2.7k 11/s ] nodes: [ 623M 2.1M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 5.2G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 5% 5% 6% 6% 6% 6% 6% 5% 6% 6%) -> (547/1M) -> fetch( 0%) 0:04:43 INF [wikidata] - blocks: [ 2.8k 10/s ] nodes: [ 648M 2.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.6 gc: 0% heap: 4.5G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 5% 5% 6% 6% 6% 6% 5% 5% 5% 5% 5%) -> (409/1M) -> fetch( 0%) 0:04:53 INF [wikidata] - blocks: [ 2.9k 9/s ] nodes: [ 670M 2.1M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.6 gc: 0% heap: 3.3G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 5% 5% 5% 6% 5% 5% 7% 5% 4% 5% 6%) -> (583/1M) -> fetch( 0%) 0:05:04 INF [wikidata] - blocks: [ 3k 10/s ] nodes: [ 694M 2.2M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.6 gc: 0% heap: 2.5G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 5% 6% 5% 6% 6% 5% 6% 6% 5% 6% 6%) -> (426/1M) -> fetch( 0%) 0:05:15 INF [wikidata] - blocks: [ 3.2k 11/s ] nodes: [ 720M 2.4M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 2.2G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 5% 6% 6% 6% 6% 5% 5% 6% 6% 6% 6%) -> (590/1M) -> fetch( 0%) 0:05:26 INF [wikidata] - blocks: [ 3.3k 11/s ] nodes: [ 746M 2.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 2.2G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 7% 6% 7% 6% 6% 7% 7% 6% 6% 7%) -> (584/1M) -> fetch( 0%) 0:05:37 INF [wikidata] - blocks: [ 3.4k 10/s ] nodes: [ 771M 2.2M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 2.2G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (21/34) -> filter( 6% 6% 5% 5% 6% 6% 6% 6% 6% 6% 6%) -> (587/1M) -> fetch( 0%) 0:05:48 INF [wikidata] - blocks: [ 3.5k 11/s ] nodes: [ 799M 2.5M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.7 gc: 0% heap: 2.3G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 6% 6% 7% 6% 5% 6% 6% 5% 6% 6%) -> (656/1M) -> fetch( 0%) 0:05:58 INF [wikidata] - blocks: [ 3.6k 10/s ] nodes: [ 824M 2.3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 278M cpus: 0.6 gc: 0% heap: 2.5G/21G direct: 32k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter( 6% 6% 6% 6% 6% 5% 6% 5% 6% 6% 6%) -> (610/1M) -> fetch( 0%) ...snip... 0:43:13 INF [wikidata:fetch] - Fetched batch 123 (5000 qids) 1s cpu:2s avg:1.9 0:43:16 INF [wikidata:fetch] - Fetched batch 124 (5000 qids) 1s cpu:2s avg:1.9 0:43:21 INF [wikidata:fetch] - Fetched batch 125 (5000 qids) 3s cpu:5s avg:1.6 0:43:22 INF [wikidata] - blocks: [ 37k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 526M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 620k 1.4k/s ] 363M cpus: 1.6 gc: 0% heap: 3.8G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(14% 15% 14% 13% 13% 14% 13% 15% 14% 15% 13%) -> (156k/1M) -> fetch( 2%) 0:43:24 INF [wikidata:fetch] - Fetched batch 126 (5000 qids) 2s cpu:3s avg:1.6 0:43:32 INF [wikidata] - blocks: [ 38k 17/s ] nodes: [ 9.3B 0/s ] ways: [ 539M 1.2M/s ] rels: [ 0 0/s ] wiki: [ 630k 969/s ] 364M cpus: 1.5 gc: 0% heap: 2.5G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(14% 13% 14% 14% 14% 12% 12% 12% 12% 13% 14%) -> (152k/1M) -> fetch( 1%) 0:43:33 INF [wikidata:fetch] - Fetched batch 127 (5000 qids) 1s cpu:3s avg:2.2 0:43:35 INF [wikidata:fetch] - Fetched batch 128 (5000 qids) 1s cpu:2s avg:1.7 0:43:37 INF [wikidata:fetch] - Fetched batch 129 (5000 qids) 1s cpu:1s avg:1.1 0:43:42 INF [wikidata:fetch] - Fetched batch 130 (5000 qids) 1s cpu:2s avg:1.3 0:43:43 INF [wikidata] - blocks: [ 38k 16/s ] nodes: [ 9.3B 0/s ] ways: [ 551M 1.1M/s ] rels: [ 0 0/s ] wiki: [ 645k 1.4k/s ] 366M cpus: 1.5 gc: 0% heap: 5.3G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(14% 14% 13% 13% 14% 13% 14% 13% 13% 13% 14%) -> (142k/1M) -> fetch( 2%) 0:43:44 INF [wikidata:fetch] - Fetched batch 131 (5000 qids) 1s cpu:2s avg:1.4 0:43:51 INF [wikidata:fetch] - Fetched batch 132 (5000 qids) 2s cpu:2s avg:1.1 0:43:53 INF [wikidata:fetch] - Fetched batch 133 (5000 qids) 1s cpu:2s avg:1.7 0:43:53 INF [wikidata] - blocks: [ 38k 17/s ] nodes: [ 9.3B 0/s ] ways: [ 564M 1.1M/s ] rels: [ 0 0/s ] wiki: [ 660k 1.4k/s ] 368M cpus: 1.4 gc: 0% heap: 4G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 12% 13% 12% 11% 13% 14% 13% 12% 12% 11%) -> (131k/1M) -> fetch( 2%) 0:43:55 INF [wikidata:fetch] - Fetched batch 134 (5000 qids) 2s cpu:3s avg:1.9 0:43:57 INF [wikidata:fetch] - Fetched batch 135 (5000 qids) 1s cpu:2s avg:1.6 0:44:01 INF [wikidata:fetch] - Fetched batch 136 (5000 qids) 2s cpu:3s avg:1.5 0:44:03 INF [wikidata] - blocks: [ 38k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 577M 1.3M/s ] rels: [ 0 0/s ] wiki: [ 675k 1.4k/s ] 370M cpus: 1.6 gc: 0% heap: 3.8G/21G direct: 98k postGC: 1.5G parse() -> pbf( 0%) -> (23/34) -> filter(14% 13% 13% 15% 15% 14% 13% 13% 14% 15% 13%) -> (121k/1M) -> fetch( 2%) 0:44:04 INF [wikidata:fetch] - Fetched batch 137 (5000 qids) 1s cpu:2s avg:2 0:44:12 INF [wikidata:fetch] - Fetched batch 138 (5000 qids) 1s cpu:2s avg:1.8 0:44:14 INF [wikidata] - blocks: [ 38k 17/s ] nodes: [ 9.3B 0/s ] ways: [ 590M 1.2M/s ] rels: [ 0 0/s ] wiki: [ 690k 1.4k/s ] 372M cpus: 1.4 gc: 0% heap: 2.7G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 13% 12% 11% 12% 13% 12% 12% 13% 13% 13%) -> (109k/1M) -> fetch( 1%) 0:44:14 INF [wikidata:fetch] - Fetched batch 139 (5000 qids) 1s cpu:3s avg:2.6 0:44:17 INF [wikidata:fetch] - Fetched batch 140 (5000 qids) 1s cpu:2s avg:1.6 0:44:22 INF [wikidata:fetch] - Fetched batch 141 (5000 qids) 1s cpu:2s avg:1.9 0:44:24 INF [wikidata] - blocks: [ 39k 18/s ] nodes: [ 9.3B 0/s ] ways: [ 604M 1.3M/s ] rels: [ 0 0/s ] wiki: [ 700k 950/s ] 373M cpus: 1.6 gc: 0% heap: 2.6G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(14% 14% 14% 14% 14% 14% 14% 14% 13% 14% 14%) -> (105k/1M) -> fetch( 1%) 0:44:26 INF [wikidata:fetch] - Fetched batch 142 (5000 qids) 1s cpu:2s avg:1.5 0:44:31 INF [wikidata:fetch] - Fetched batch 143 (5000 qids) 2s cpu:3s avg:2 0:44:35 INF [wikidata] - blocks: [ 39k 17/s ] nodes: [ 9.3B 0/s ] ways: [ 617M 1.1M/s ] rels: [ 0 0/s ] wiki: [ 710k 963/s ] 374M cpus: 1.5 gc: 0% heap: 5.8G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 14% 13% 13% 14% 14% 14% 14% 14% 14% 13%) -> (100k/1M) -> fetch( 1%) 0:44:36 INF [wikidata:fetch] - Fetched batch 144 (5000 qids) 1s cpu:2s avg:1.5 0:44:42 INF [wikidata:fetch] - Fetched batch 145 (5000 qids) 1s cpu:2s avg:1.7 0:44:45 INF [wikidata] - blocks: [ 39k 18/s ] nodes: [ 9.3B 0/s ] ways: [ 630M 1.2M/s ] rels: [ 0 0/s ] wiki: [ 720k 955/s ] 376M cpus: 1.4 gc: 0% heap: 5.7G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 13% 12% 13% 13% 12% 12% 13% 11% 12% 13%) -> (95k/1M) -> fetch( 1%) 0:44:47 INF [wikidata:fetch] - Fetched batch 146 (5000 qids) 1s cpu:3s avg:2 0:44:54 INF [wikidata:fetch] - Fetched batch 147 (5000 qids) 1s cpu:2s avg:1.6 0:44:55 INF [wikidata] - blocks: [ 39k 18/s ] nodes: [ 9.3B 0/s ] ways: [ 644M 1.3M/s ] rels: [ 0 0/s ] wiki: [ 730k 956/s ] 377M cpus: 1.5 gc: 1% heap: 1.7G/21G direct: 98k postGC: 1.5G parse() -> pbf( 0%) -> (23/34) -> filter(13% 12% 12% 14% 14% 14% 13% 13% 14% 14% 13%) -> (90k/1M) -> fetch( 1%) 0:44:57 INF [wikidata:fetch] - Fetched batch 148 (5000 qids) 1s cpu:2s avg:1.5 0:44:59 INF [wikidata:fetch] - Fetched batch 149 (5000 qids) 1s cpu:2s avg:1.8 0:45:02 INF [wikidata:fetch] - Fetched batch 150 (5000 qids) 1s cpu:2s avg:1.5 0:45:04 INF [wikidata:fetch] - Fetched batch 151 (5000 qids) 1s cpu:2s avg:1.5 0:45:05 INF [wikidata] - blocks: [ 39k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 658M 1.3M/s ] rels: [ 0 0/s ] wiki: [ 750k 1.9k/s ] 379M cpus: 1.5 gc: 1% heap: 1.5G/21G direct: 98k postGC: 1.5G parse() -> pbf( 0%) -> (22/34) -> filter(14% 14% 14% 15% 13% 13% 14% 12% 12% 12% 13%) -> (72k/1M) -> fetch( 2%) 0:45:09 INF [wikidata:fetch] - Fetched batch 152 (5000 qids) 1s cpu:3s avg:2.5 0:45:16 INF [wikidata] - blocks: [ 40k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 672M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 755k 495/s ] 380M cpus: 1.7 gc: 0% heap: 2.3G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (19/34) -> filter(15% 15% 16% 13% 16% 14% 14% 15% 15% 15% 17%) -> (73k/1M) -> fetch( 1%) 0:45:18 INF [wikidata:fetch] - Fetched batch 153 (5000 qids) 1s cpu:2s avg:1.7 0:45:24 INF [wikidata:fetch] - Fetched batch 154 (5000 qids) 3s cpu:4s avg:1.5 0:45:26 INF [wikidata] - blocks: [ 40k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 687M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 770k 1.4k/s ] 382M cpus: 1.4 gc: 0% heap: 2.7G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 14% 12% 13% 11% 13% 13% 13% 13% 14% 11%) -> (60k/1M) -> fetch( 1%) 0:45:27 INF [wikidata:fetch] - Fetched batch 155 (5000 qids) 1s cpu:3s avg:2.1 0:45:36 INF [wikidata:fetch] - Fetched batch 156 (5000 qids) 8s cpu:13s avg:1.6 0:45:37 INF [wikidata] - blocks: [ 40k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 701M 1.3M/s ] rels: [ 0 0/s ] wiki: [ 775k 477/s ] 382M cpus: 1.6 gc: 0% heap: 3.2G/21G direct: 98k postGC: 1.5G parse() -> pbf( 0%) -> (22/34) -> filter(12% 15% 14% 14% 15% 13% 15% 13% 14% 13% 15%) -> (60k/1M) -> fetch( 1%) 0:45:40 INF [wikidata:fetch] - Fetched batch 157 (5000 qids) 1s cpu:2s avg:1.4 0:45:42 INF [wikidata:fetch] - Fetched batch 158 (5000 qids) 2s cpu:3s avg:1.7 0:45:47 INF [wikidata] - blocks: [ 40k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 717M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 785k 974/s ] 384M cpus: 1.6 gc: 0% heap: 4.3G/21G direct: 98k postGC: 1.5G parse() -> pbf( 0%) -> (23/34) -> filter(14% 15% 14% 15% 13% 14% 12% 14% 14% 12% 16%) -> (54k/1M) -> fetch( 1%) 0:45:48 INF [wikidata:fetch] - Fetched batch 159 (5000 qids) 1s cpu:3s avg:2.5 0:45:52 INF [wikidata:fetch] - Fetched batch 160 (5000 qids) 1s cpu:3s avg:2.4 0:45:54 INF [wikidata:fetch] - Fetched batch 161 (5000 qids) 1s cpu:2s avg:1.4 0:45:56 INF [wikidata:fetch] - Fetched batch 162 (5000 qids) 1s cpu:2s avg:1.7 0:45:57 INF [wikidata] - blocks: [ 40k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 731M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 805k 1.9k/s ] 386M cpus: 1.6 gc: 0% heap: 5.5G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(14% 15% 15% 14% 15% 13% 15% 14% 13% 13% 14%) -> (31k/1M) -> fetch( 2%) 0:45:59 INF [wikidata:fetch] - Fetched batch 163 (5000 qids) 1s cpu:2s avg:1.3 0:46:01 INF [wikidata:fetch] - Fetched batch 164 (5000 qids) 1s cpu:2s avg:2 0:46:06 INF [wikidata:fetch] - Fetched batch 165 (5000 qids) 1s cpu:1s avg:1 0:46:08 INF [wikidata] - blocks: [ 41k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 747M 1.5M/s ] rels: [ 0 0/s ] wiki: [ 820k 1.4k/s ] 387M cpus: 1.6 gc: 1% heap: 2.2G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 15% 13% 13% 15% 14% 12% 14% 15% 14% 14%) -> (16k/1M) -> fetch( 1%) 0:46:09 INF [wikidata:fetch] - Fetched batch 166 (5000 qids) 1s cpu:2s avg:1.8 0:46:13 INF [wikidata:fetch] - Fetched batch 167 (5000 qids) 3s cpu:5s avg:1.5 0:46:18 INF [wikidata] - blocks: [ 41k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 762M 1.3M/s ] rels: [ 0 0/s ] wiki: [ 835k 1.4k/s ] 389M cpus: 1.5 gc: 0% heap: 2.5G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (22/34) -> filter(13% 12% 14% 14% 11% 14% 14% 13% 15% 12% 13%) -> (636/1M) -> fetch( 1%) 0:46:20 INF [wikidata:fetch] - Fetched batch 168 (5000 qids) 1s cpu:2s avg:1.4 0:46:29 INF [wikidata] - blocks: [ 41k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 777M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 840k 471/s ] 390M cpus: 1.6 gc: 0% heap: 4.3G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(15% 13% 14% 15% 12% 13% 14% 14% 13% 15% 14%) -> (437/1M) -> fetch( 1%) 0:46:33 INF [wikidata:fetch] - Fetched batch 169 (5000 qids) 1s cpu:2s avg:1.6 0:46:39 INF [wikidata] - blocks: [ 41k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 793M 1.5M/s ] rels: [ 0 0/s ] wiki: [ 845k 481/s ] 390M cpus: 1.6 gc: 0% heap: 1.5G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 15% 15% 14% 15% 15% 15% 13% 15% 14% 15%) -> (579/1M) -> fetch( 1%) 0:46:46 INF [wikidata:fetch] - Fetched batch 170 (5000 qids) 1s cpu:3s avg:2.1 0:46:49 INF [wikidata] - blocks: [ 41k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 808M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 845k 0/s ] 391M cpus: 1.7 gc: 0% heap: 3.2G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(15% 14% 17% 16% 14% 14% 16% 15% 14% 15% 15%) -> (3k/1M) -> fetch( 1%) 0:47:00 INF [wikidata] - blocks: [ 42k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 823M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 850k 479/s ] 391M cpus: 1.5 gc: 0% heap: 4.1G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 13% 14% 15% 13% 13% 14% 13% 15% 13% 14%) -> (422/1M) -> fetch( 0%) 0:47:02 INF [wikidata:fetch] - Fetched batch 171 (5000 qids) 1s cpu:1s avg:1.1 0:47:10 INF [wikidata] - blocks: [ 42k 21/s ] nodes: [ 9.3B 0/s ] ways: [ 839M 1.5M/s ] rels: [ 0 0/s ] wiki: [ 855k 481/s ] 392M cpus: 1.6 gc: 0% heap: 5.9G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(15% 13% 14% 14% 14% 14% 13% 15% 14% 13% 13%) -> (509/1M) -> fetch( 0%) 0:47:18 INF [wikidata:fetch] - Fetched batch 172 (5000 qids) 1s cpu:1s avg:1.3 0:47:20 INF [wikidata] - blocks: [ 42k 22/s ] nodes: [ 9.3B 0/s ] ways: [ 856M 1.6M/s ] rels: [ 0 0/s ] wiki: [ 855k 0/s ] 392M cpus: 1.7 gc: 0% heap: 3.9G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(16% 14% 15% 16% 16% 15% 15% 16% 15% 15% 15%) -> (1.5k/1M) -> fetch( 0%) 0:47:31 INF [wikidata] - blocks: [ 42k 21/s ] nodes: [ 9.3B 0/s ] ways: [ 872M 1.5M/s ] rels: [ 0 0/s ] wiki: [ 860k 474/s ] 392M cpus: 1.7 gc: 0% heap: 2.1G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (20/34) -> filter(15% 14% 14% 14% 15% 15% 16% 16% 16% 16% 15%) -> (508/1M) -> fetch( 0%) 0:47:34 INF [wikidata:fetch] - Fetched batch 173 (5000 qids) 1s cpu:2s avg:1.6 0:47:41 INF [wikidata] - blocks: [ 42k 20/s ] nodes: [ 9.3B 0/s ] ways: [ 887M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 865k 474/s ] 393M cpus: 1.5 gc: 0% heap: 3G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 14% 14% 13% 13% 14% 13% 13% 15% 15% 14%) -> (649/1M) -> fetch( 0%) 0:47:51 INF [wikidata:fetch] - Fetched batch 174 (5000 qids) 2s cpu:4s avg:1.8 0:47:52 INF [wikidata] - blocks: [ 43k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 902M 1.3M/s ] rels: [ 0 0/s ] wiki: [ 865k 0/s ] 393M cpus: 1.7 gc: 0% heap: 3.6G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(15% 16% 15% 15% 16% 15% 15% 14% 16% 15% 14%) -> (2.5k/1M) -> fetch( 0%) 0:48:02 INF [wikidata] - blocks: [ 43k 19/s ] nodes: [ 9.3B 0/s ] ways: [ 916M 1.4M/s ] rels: [ 0 0/s ] wiki: [ 870k 480/s ] 393M cpus: 1.5 gc: 0% heap: 4.4G/21G direct: 98k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(13% 14% 14% 13% 13% 14% 12% 14% 13% 12% 15%) -> (490/1M) -> fetch( 0%) 0:48:04 INF [wikidata:fetch] - Fetched batch 175 (5000 qids) 1s cpu:2s avg:2.2 0:48:13 INF [wikidata] - blocks: [ 43k 18/s ] nodes: [ 9.3B 0/s ] ways: [ 929M 1.2M/s ] rels: [ 0 0/s ] wiki: [ 875k 481/s ] 394M cpus: 1.4 gc: 0% heap: 3.8G/21G direct: 99k postGC: 1.4G parse() -> pbf( 0%) -> (23/34) -> filter(14% 14% 13% 11% 14% 13% 11% 12% 12% 12% 13%) -> (533/1M) -> fetch( 1%) Worker wikidata_filter-2 died java.io.UncheckedIOException: java.io.IOException: Channel not open for writing - cannot extend file to required size at com.onthegomap.planetiler.reader.osm.OsmInputFile$LazyReader$LazyBlock.decodeElements(OsmInputFile.java:235) at com.onthegomap.planetiler.util.Wikidata.filter(Wikidata.java:227) at com.onthegomap.planetiler.worker.WorkerPipeline$Builder.lambda$addWorker$0(WorkerPipeline.java:249) at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41) at com.onthegomap.planetiler.worker.Worker.lambda$new$1(Worker.java:68) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.io.IOException: Channel not open for writing - cannot extend file to required size at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1282) at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1183) at com.onthegomap.planetiler.reader.osm.OsmInputFile$LazyReader$LazyBlock.decodeElements(OsmInputFile.java:230) ... 8 more Exception in thread "main" com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.IOException: Channel not open for writing - cannot extend file to required size at com.onthegomap.planetiler.util.Exceptions.throwFatalException(Exceptions.java:29) at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:367) at com.onthegomap.planetiler.stats.ProgressLoggers.awaitAndLog(ProgressLoggers.java:352) at com.onthegomap.planetiler.worker.WorkerPipeline.awaitAndLog(WorkerPipeline.java:59) at com.onthegomap.planetiler.util.Wikidata.fetch(Wikidata.java:153) at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:796) at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:58) at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:18) at com.onthegomap.planetiler.Main.main(Main.java:102) Caused by: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.IOException: Channel not open for writing - cannot extend file to required size at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096) at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:361) ... 7 more Caused by: java.io.UncheckedIOException: java.io.IOException: Channel not open for writing - cannot extend file to required size at com.onthegomap.planetiler.reader.osm.OsmInputFile$LazyReader$LazyBlock.decodeElements(OsmInputFile.java:235) at com.onthegomap.planetiler.util.Wikidata.filter(Wikidata.java:227) at com.onthegomap.planetiler.worker.WorkerPipeline$Builder.lambda$addWorker$0(WorkerPipeline.java:249) at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41) at com.onthegomap.planetiler.worker.Worker.lambda$new$1(Worker.java:68) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.io.IOException: Channel not open for writing - cannot extend file to required size at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1282) at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1183) at com.onthegomap.planetiler.reader.osm.OsmInputFile$LazyReader$LazyBlock.decodeElements(OsmInputFile.java:230) ... 8 more

Environment (please complete the following information):

• Hardware: 3.5ghx 6c, 64gbram, 15gb swap

• OS: ubuntu 24.10

• Java version and distribution: openjdk 21.0.4 2024-07-16 OpenJDK Runtime Environment (build 21.0.4+7-Ubuntu-1ubuntu224.04) OpenJDK 64-Bit Server VM (build 21.0.4+7-Ubuntu-1ubuntu224.04, mixed mode, sharing)

• Maven version: 3.8.7

Additional context followed the instructions in PLANET.md (but i skipped the osmium apply-changes and osmium renumber commands due to having too little ram)

[msbarry]

Does it happen consistently or intermittently? Also what error do you get when you run with --osm-lazy-reads=false ?

[HomingHamster]

Consistiently, all the time, I will check with the --osm-lazy-reads=false you suggest shortly and report back.

[HomingHamster]

java -Xmx20g -jar planetiler.jar --area=planet --bounds=planet ----osm-path=/mnt/ele/nominatim-data/planet-latest.osm.pbf --osm-lazy-reads=false --download --download-threads=10 --download-chunk-size-mb=1000 --fetch-wikidata --output=/mnt/ele/planetiler/output.mbtiles --nodemap-type=array --storage=mmap 0:00:00 DEB - argument: config=null (path to config file) 0:00:00 DEB - argument: download_dir=data/sources (download directory) 0:00:00 DEB - argument: area=planet (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet')) 0:00:00 INF - argument: stats=use in-memory stats 0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage) 0:00:00 DEB - argument: storage=mmap (default storage type for temporary data, one of [ram, mmap, direct]) 0:00:00 DEB - argument: threads=12 (num threads) 0:00:00 DEB - argument: write_threads=1 (number of threads to use when writing temp features) 0:00:00 DEB - argument: process_threads=11 (number of threads to use when processing input features) 0:00:00 DEB - argument: bounds=Env[-180.0 : 180.0, -85.0511287798066 : 85.0511287798066] (bounds) 0:00:00 DEB - argument: polygon=null (a .poly file that limits output to tiles intersecting the shape) 0:00:00 DEB - argument: minzoom=0 (minimum zoom level) 0:00:00 DEB - argument: maxzoom=14 (maximum zoom level up to 15) 0:00:00 DEB - argument: render_maxzoom=14 (maximum rendering zoom level up to 15) 0:00:00 DEB - argument: tmpdir=data/tmp (temp directory) 0:00:00 DEB - argument: feature_read_threads=1 (number of threads to use when reading features at tile write time) 0:00:00 DEB - argument: tile_write_threads=1 (number of threads used to write tiles - only supported by [files, csv, tsv, proto, pbf, json]) 0:00:00 DEB - argument: loginterval=10 seconds (time between logs) 0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings) 0:00:00 DEB - argument: append=false (append to the output file - only supported by [files, csv, tsv, proto, pbf, json]) 0:00:00 DEB - argument: compress_temp=false (compress temporary feature storage (uses more CPU, but less disk space)) 0:00:00 DEB - argument: mmap_temp=true (use memory-mapped IO for temp feature files) 0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks) 0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks) 0:00:00 DEB - argument: nodemap_type=array (type of node location map, one of [noop, sortedtable, sparsearray, array]) 0:00:00 DEB - argument: nodemap_storage=mmap (storage for node location map, one of [ram, mmap, direct]) 0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations) 0:00:00 DEB - argument: multipolygon_geometry_storage=mmap (storage for multipolygon geometries, one of [ram, mmap, direct]) 0:00:00 DEB - argument: multipolygon_geometry_madvise=true (use linux madvise(random) for temporary multipolygon geometry storage) 0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP) 0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP) 0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP) 0:00:00 DEB - argument: http_retry_wait=5 seconds (How long to wait before retrying HTTP request) 0:00:00 DEB - argument: download_chunk_size_mb=1000 (Size of file chunks to download in parallel in megabytes) 0:00:00 DEB - argument: download_threads=10 (Number of parallel threads to use when downloading each file) 0:00:00 DEB - argument: download_max_bandwidth= (Maximum bandwidth to consume when downloading files in units mb/s, mbps, kbps, etc.) 0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming) 0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level) 0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming) 0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level) 0:00:00 DEB - argument: osm_lazy_reads=false (Read OSM blocks from disk in worker threads) 0:00:00 DEB - argument: skip_filled_tiles=false (Skip writing tiles containing only polygon fills to the output) 0:00:00 DEB - argument: tile_warning_size_mb=1.0 (Maximum size in megabytes of a tile to emit a warning about) 0:00:00 DEB - argument: color=null (Color the terminal output) 0:00:00 DEB - argument: keep_unzipped=false (keep unzipped sources by default after reading) 0:00:00 DEB - argument: tile_compression=gzip (the tile compression, one of [none, gzip]) 0:00:00 DEB - argument: output_layerstats=false (output a tsv.gz file for each tile/layer size) 0:00:00 DEB - argument: debug_url=https://onthegomap.github.io/planetiler-demo/#{z}/{lat}/{lon} (debug url to use for displaying tiles with {z} {lat} {lon} placeholders) 0:00:00 DEB - argument: tile_weights=data/tile_weights.tsv.gz (tsv.gz file with columns z,x,y,loads to generate weighted average tile size stat) 0:00:00 DEB - argument: max_point_buffer=Infinity (Max tile pixels to include points outside tile bounds. Set to a lower value to reduce tile size for clients that handle label collisions across tiles (most web and native clients). NOTE: Do not reduce if you need to support raster tile rendering) 0:00:00 DEB - argument: log_jts_exceptions=false (Emit verbose details to debug JTS geometry errors) 0:00:00 DEB - argument: feature_source_id_multiplier=10 (Set vector tile feature IDs to (featureId * thisValue) + sourceId where sourceId is 1 for OSM nodes, 2 for ways, 3 for relations, and 0 for other sources. Set to false to disable.) 0:00:00 DEB - argument: only_download=false (download source data then exit) 0:00:00 DEB - argument: tests=null (run test cases in a yaml then quit) 0:00:00 DEB - argument: download=true (download sources) 0:00:00 DEB - argument: refresh_sources=false (download new version of source files if they have changed) 0:00:00 DEB - argument: download_osm_tile_weights=true (download OSM tile weights file) 0:00:00 DEB - argument: temp_nodes=data/tmp/node.db (temp node db location) 0:00:00 DEB - argument: temp_multipolygons=data/tmp/multipolygon.db (temp multipolygon db location) 0:00:00 DEB - argument: temp_features=data/tmp/feature.db (temp feature db location) 0:00:00 DEB - argument: osm_parse_node_bounds=false (parse bounds from OSM nodes instead of header) 0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit) 0:00:00 DEB - argument: fetch_wikidata=true (fetch wikidata translations then continue) 0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file) 0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path) 0:00:00 DEB - argument: refresh_lake_centerlines=false (Download new version of lake_centerlines if changed) 0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: lake_centerlines_url=https://dev.maptiler.download/geodata/omt/lake_centerline.shp.zip (lake_centerlines shapefile url) 0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path) 0:00:00 DEB - argument: refresh_water_polygons=false (Download new version of water_polygons if changed) 0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: water_polygons_url=https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip (water_polygons shapefile url) 0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path) 0:00:00 DEB - argument: refresh_natural_earth=false (Download new version of natural_earth if changed) 0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: natural_earth_url=https://dev.maptiler.download/geodata/omt/natural_earth_vector.sqlite.zip (natural_earth sqlite db url) 0:00:00 DEB - argument: natural_earth_keep_unzipped=false (keep unzipped natural_earth after reading) 0:00:00 DEB - argument: osm_path=/mnt/ele/nominatim-data/planet-latest.osm.pbf (osm OSM input file path) 0:00:00 DEB - argument: refresh_osm=false (Download new version of osm if changed) 0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage)) 0:00:00 DEB - argument: osm_url=aws:latest (osm OSM input file url) 0:00:00 DEB - argument: output=/mnt/ele/planetiler/output.mbtiles (output tile archive path) 0:00:00 DEB - argument: version=false (show version then exit) 0:00:00 INF - Planetiler build git hash: a74f274c3162cd547c9b19a9a5e8705982a61755 0:00:00 INF - Planetiler build version: 0.8.2 0:00:00 INF - Planetiler build timestamp: 2024-07-02T11:38:28.480Z 0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names) 0:00:00 DEB - argument: languages=am,ar,az,be,bg,bn,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fa,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pa,pnb,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,ur,vi,zh,zh-Hant,zh-Hans (languages to use) 0:00:00 DEB - argument: only_layers= (Include only certain layers) 0:00:00 DEB - argument: exclude_layers= (Exclude certain layers) 0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries) 0:00:00 DEB - argument: boundary_osm_only=false (boundary layer: only use OSM, even at low zoom levels) 0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13) 0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13) 0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways) 0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield)) 0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways) 0:00:00 DEB - argument: transportation_name_minor_refs=false (transportation_name layer: include name and refs from minor road networks if not present on a way) 0:00:00 DEB - argument: help=false (show arguments then exit) 0:00:00 DEB - argument: layer_stats=/mnt/ele/planetiler/output.mbtiles.layerstats.tsv.gz (layer stats output path) 0:00:00 INF - Building OpenMapTilesProfile profile into file:///mnt/ele/planetiler/output.mbtiles in these phases: 0:00:00 INF - wikidata: Fetch translations from wikidata query service 0:00:00 INF - lake_centerlines: Process features in data/sources/lake_centerline.shp.zip 0:00:00 INF - water_polygons: Process features in data/sources/water-polygons-split-3857.zip 0:00:00 INF - natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip 0:00:00 INF - osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members) 0:00:00 INF - osm_pass2: Process OpenStreetMap nodes, ways, then relations 0:00:00 INF - sort: Sort rendered features by tile ID 0:00:00 INF - archive: Encode each tile and write to TileArchiveConfig[format=MBTILES, scheme=FILE, uri=file:///mnt/ele/planetiler/output.mbtiles, options={}] 0:00:00 INF [wikidata] - 0:00:00 INF [wikidata] - Starting... 0:00:00 INF [wikidata] - Starting with 11 process threads 0:00:12 INF [wikidata] - loaded from 1880289 mappings from /mnt/ele/planetiler/data/sources/wikidata_names.json in 12s cpu:16s avg:1.3 0:00:12 INF [wikidata] - skipping 1880289 mappings we already have 0:00:30 INF [wikidata] - blocks: [ 134 13/s ] nodes: [ 28M 2.8M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 394M cpus: 1.6 gc: 2% heap: 4G/21G direct: 3.4M postGC: 3.7G parse() -> pbf( 5%) -> (0/34) -> filter( 2% 5% 5% 4% 12% 6% 6% 12% 12% 4% 7%) -> (549/1M) -> fetch( 0%) 0:00:40 INF [wikidata] - blocks: [ 279 14/s ] nodes: [ 59M 3M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 394M cpus: 1.1 gc: 0% heap: 3.4G/21G direct: 3.4M postGC: 1.9G parse() -> pbf( 5%) -> (0/34) -> filter( 4% 4% 0% 7% 5% 7% 6% 3% 8% 7% 10%) -> (445/1M) -> fetch( 0%) 0:00:50 INF [wikidata] - blocks: [ 432 15/s ] nodes: [ 91M 3.1M/s ] ways: [ 0 0/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 394M cpus: 1.1 gc: 0% heap: 2.9G/21G direct: 3.4M postGC: 1.9G

...snip...

0:36:07 INF [wikidata] - blocks: [ 43k 37/s ] nodes: [ 9.3B 0/s ] ways: [ 895M 2.6M/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 394M cpus: 3.4 gc: 1% heap: 2.6G/21G direct: 3.9M postGC: 1.8G parse() -> pbf( 7%) -> (0/34) -> filter(31% 31% 33% 28% 29% 29% 30% 26% 29% 25% 28%) -> (414/1M) -> fetch( 0%) 0:36:17 INF [wikidata] - blocks: [ 43k 27/s ] nodes: [ 9.3B 0/s ] ways: [ 915M 1.9M/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 394M cpus: 1.7 gc: 0% heap: 4.3G/21G direct: 3.9M postGC: 1.8G parse() -> pbf( 5%) -> (0/34) -> filter(12% 18% 18% 17% 17% 16% 13% 14% 13% 9% 14%) -> (623/1M) -> fetch( 0%) 0:36:27 INF [wikidata] - blocks: [ 43k 26/s ] nodes: [ 9.3B 0/s ] ways: [ 933M 1.8M/s ] rels: [ 0 0/s ] wiki: [ 0 0/s ] 394M cpus: 1.6 gc: 0% heap: 4.7G/21G direct: 3.9M postGC: 1.8G parse() -> pbf( 5%) -> (0/34) -> filter(12% 15% 13% 10% 17% 14% 11% 15% 13% 10% 18%) -> (412/1M) -> fetch( 0%) Worker wikidata_pbf-1 died java.io.UncheckedIOException: java.io.IOException: Tried to read 1695330 bytes but only got 1348132 at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:166) at com.onthegomap.planetiler.worker.WorkerPipeline$Empty.lambda$fromGenerator$0(WorkerPipeline.java:167) at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41) at com.onthegomap.planetiler.worker.Worker.lambda$new$1(Worker.java:68) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.io.IOException: Tried to read 1695330 bytes but only got 1348132 at com.onthegomap.planetiler.reader.osm.OsmInputFile.readBytes(OsmInputFile.java:63) at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:155) ... 7 more Exception in thread "main" com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.IOException: Tried to read 1695330 bytes but only got 1348132 at com.onthegomap.planetiler.util.Exceptions.throwFatalException(Exceptions.java:29) at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:367) at com.onthegomap.planetiler.stats.ProgressLoggers.awaitAndLog(ProgressLoggers.java:352) at com.onthegomap.planetiler.worker.WorkerPipeline.awaitAndLog(WorkerPipeline.java:59) at com.onthegomap.planetiler.util.Wikidata.fetch(Wikidata.java:153) at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:796) at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:58) at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:18) at com.onthegomap.planetiler.Main.main(Main.java:102) Caused by: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.IOException: Tried to read 1695330 bytes but only got 1348132 at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096) at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:361) ... 7 more Caused by: java.io.UncheckedIOException: java.io.IOException: Tried to read 1695330 bytes but only got 1348132 at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:166) at com.onthegomap.planetiler.worker.WorkerPipeline$Empty.lambda$fromGenerator$0(WorkerPipeline.java:167) at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41) at com.onthegomap.planetiler.worker.Worker.lambda$new$1(Worker.java:68) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.io.IOException: Tried to read 1695330 bytes but only got 1348132 at com.onthegomap.planetiler.reader.osm.OsmInputFile.readBytes(OsmInputFile.java:63) at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:155) ... 7 more

[HomingHamster]

I noticed when i ran this command

java -Xmx20g -jar planetiler.jar --area=monaco --download --download-threads=10 --download-chunk-size-mb=1000 --fetch-wikidata --output=output.mbtiles --nodemap-type=array --storage=mmap

to compile the maps for monaco, this warning appeared

0:00:30 WAR [wikidata] - Planetiler will use ~76G memory-mapped files for node locations and multipolygon geometries but the OS only has 45G available to cache pages, this may slow the import down. To speed up, run on a machine with more memory or reduce the -Xmx setting.

I'd also like to check, is renumbering with osmtool necessary for any osm.pbf download? Also i noticed it intermittently creates an output.mbtiles file for the original command the bug was about.

[msbarry]

Can you try reading that osm pbf with any other tools? Those errors make it seem like the file is malformed (ie truncated early)

[HomingHamster]

Thanks, I think you are right, I'm now 42 hours into a full planet render, probably got some other settings to adjust, but yes, when i get planetiler to download its own osm pbf it gets past the stage where it was erroring. It was most probably a malformed osm pbf as you say, I must have made a different mistake with the original command to think that the auto downloaded one didn't work either. Thanks again.

[msbarry]

OK got it, I am curious if other tools can read that PBF file to know if this is something it should be able to recover from. I'll close for now but feel free to reopen if you think it's a case planetiler should be able to handle.

Also 42 hours is way slow, a laptop with 8gb of RAM and an SSD should finish in less than a day. Are you running on a spinning metal disk or something?

[HomingHamster]

I'm far from familiar with the PBF ecosystem at this point, and I don't know what other tools you mean. The nominatim server seemed to import from the PBF I was using ok if thats what you mean? Also yes, I'm running on a spinning disk, that would be why, thanks for pointing this out!

[msbarry]

Try installing osmium then osmium fileinfo -e file.pbf should parse the whole thing then print stats at the end.

[HomingHamster]

It looks like you were completely right!

File:
  Name: planet-latest.osm.pbf
  Format: PBF
  Compression: none
  Size: 79550191662
Header:
  Bounding boxes:
    (-180,-90,180,90)
  With history: no
  Options:
    generator=planet-dump-ng 1.2.4
    osmosis_replication_timestamp=2024-09-16T00:00:01Z
    pbf_dense_nodes=true
    pbf_optional_feature_0=Has_Metadata
    pbf_optional_feature_1=Sort.Type_then_ID
    sorting=Type_then_ID
    timestamp=2024-09-16T00:00:01Z
[======================================================================] 100% 
PBF error: unexpected EOF

(venv) nominatim@felix-B550-UD-AC-Y1:/mnt/ele/nominatim-data$ nominatim admin --check-database
2024-10-13 23:01:09: Using project directory: /mnt/ele/nominatim-data
2024-10-13 23:01:09: Checking database
Checking database connection ... OK
Checking database_version matches Nominatim software version ... OK
Checking for placex table ... OK
Checking for placex content ... OK
Checking that tokenizer works ... OK
Checking for wikipedia/wikidata data ... OK
Checking indexing status ... Failed
The indexing didn't finish. 436878 entries are not yet indexed.

To index the remaining entries, run:   nominatim index

Checking that database indexes are complete ... OK
Checking that all database indexes are valid ... OK
Checking TIGER external data table. ... not applicable