No feature limit & no tile size limit

[dmitry-voronoff]

Hi, Is there a way to generate tiles with no feature limit and no tile size limit (f.ex. --no-feature-limit --no-tile-size-limit in tippecanoe)?

Thnak you.

[msbarry]

Hello! In planetiler this logic is controlled by the profile, which decides what zoom levels to include which features at. There is no inherent tile size limit and no density-based feature dropping (yet), it just emits a warning when tiles are larger than 1mb but you can ignore it.

You can make a custom profile with any logic for including features, or tweak the built-in one (or soon specify via a config file #160). For example this is what the built-in profile looks like for Boston with --only-layers=building, tweaked so that buildings show up at every zoom level:

Several tiles of the tiles are over 3mb.

[dmitry-voronoff]

I was managed to configure basemap profile to keep all highways up to zoom 0. But I am stuggling with converting even europe as one peace. DigitalOcean's droplet with 192Gb RAM and 24 cores cannot handle it (process killed). Is there any way to make it work?

I am sorry, I am not Java programmer. May be I am doing something wrong.

java -Xmx170g -XX:MaxHeapFreeRatio=40 -jar planetiler.jar --force --only-layers=transportation,building,water,landcover --nodemap-type=array --storage=mmap --osm_path=europe-latest.osm.pbf --mbtiles=./data/europe.mbtiles

[msbarry]

On a larger machine like that try -Xmx=120g and --storage=ram to store all node locations (80gb) and relation membership (10gb) in RAM.

If you have less than 128gb ram then you'd want --Xmx20g and --storage=mmap to store them in memory mapped files so that the OS has plenty of memory outside the jvm to cache memory-mapped pages.

[dmitry-voronoff]

Just tested

java -Xmx180g -XX:MaxHeapFreeRatio=40 -jar planetiler.jar --force --only-layers=transportation,building,water,landcover --nodemap-type=array --storage=ram --osm_path=us-latest.osm.pbf --mbtiles=./data/usa.mbtiles

Result

0:00:00 INF - Building BasemapProfile profile into ./data/usa.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to ./data/usa.mbtiles
0:00:00 INF - no wikidata translations found, run with --fetch-wikidata to download
0:00:00 DEB - ✓ 29G storage on / (/dev/vda1) requested for read phase disk, 596G available
0:00:00 DEB -  - 29G used for temporary feature storage
0:00:00 DEB - ✓ 44G storage on / (/dev/vda1) requested for write phase disk, 596G available
0:00:00 DEB -  - 29G used for temporary feature storage
0:00:00 DEB -  - 14G used for mbtiles output
0:00:00 DEB - ✓ 80G JVM heap requested for read phase, 193G available
0:00:00 DEB -  - 76G used for array node location cache (switch to sparsearray to reduce size)
0:00:00 DEB -  - 1.3G used for multipolygon way geometries
0:00:00 DEB -  - 2.6G used for temporary profile storage
0:00:00 DEB - ✓ 0 temporary files and 9.3G of free memory for OS to cache them

...

0:08:13 DEB [osm_pass2] - Processed 152,773 blocks:
0:08:13 DEB [osm_pass2] -   nodes: 1,112,677,570 (44M/s) in 25s cpu:9m41s avg:23.1
0:08:13 DEB [osm_pass2] -   ways: 108,544,146 (328k/s) in 5m30s cpu:1h59m42s gc:2s avg:21.8
0:08:13 DEB [osm_pass2] -   relations: 944,779 (25k/s) in 37s cpu:14m9s avg:23
0:08:13 INF [osm_pass2] - Finished in 6m33s cpu:2h23m34s gc:2s avg:21.9
0:08:13 INF [osm_pass2] -   read     1x(2% 7s sys:3s wait:6m17s done:10s)
0:08:13 INF [osm_pass2] -   process 23x(92% 6m sys:1s wait:28s)
0:08:13 INF [osm_pass2] -   write    1x(64% 4m12s sys:48s wait:2m13s)
0:08:13 INF - Deleting node.db to make room for output file
0:08:13 INF [sort] -
0:08:13 INF [sort] - Starting...
0:08:13 INF [sort] - Grouped 45 chunks into 40
0:08:24 INF [sort] -  chunks: [   0 /  40   0% ] 40G
    cpus: 20.9 gc: 27% heap: 126G/193G direct: 52M postGC: 126G
    ->   (16/65) -> worker(49% 37% 43% 49% 13% 36% 32% 33% 35% 40% 52% 36% 28% 33% 22% 30% 18% 20% 14%  3% 16%  8% 16%  3%)
0:08:36 INF [sort] -  chunks: [   1 /  40   3% ] 43G
    cpus: 19.6 gc:  5% heap: 134G/193G direct: 52M postGC: 134G
    ->   (15/65) -> worker(24% 36% 61% 24% 34% 13% 38% 38% 27% 37% 41% 18% 38% 39% 42% 41% 41% 44% 39% 34% 39% 38% 38% 38%)
0:08:51 INF [sort] -  chunks: [   5 /  40  12% ] 45G
    cpus: 17.6 gc:  8% heap: 143G/193G direct: 52M postGC: 143G
    ->   (11/65) -> worker(25% 36% 28% 32% 33% 31% 34% 23% 34% 29% 31% 28% 34% 33% 36% 11% 33% 33% 34% 37% 22% 34% 38% 32%)
0:09:04 INF [sort] -  chunks: [  10 /  40  25% ] 41G
    cpus: 18.7 gc:  1% heap: 147G/193G direct: 52M postGC: 143G
    ->    (6/65) -> worker( 1% 81%  9% 82% 82% 82% 82%  0%  8%  1%  0%  3% 82%  1% 81% 18% 81% 80% 64% 82%  0% 81% 82% 81%)
0:09:24 INF [sort] -  chunks: [  11 /  40  28% ] 42G
    cpus: 10.7 gc:  4% heap: 150G/193G direct: 52M postGC: 150G
    ->    (5/65) -> worker( 1% 53% 52% 52% 53% 53% 53% 16% 48%  5% 12%  0% 51%  6% 52%  1% 53% 52%  0% 53% 17% 52% 53% 24%)
0:09:37 INF [sort] -  chunks: [  14 /  40  35% ] 42G
    cpus: 15.6 gc:  0% heap: 151G/193G direct: 52M postGC: 150G
    ->    (2/65) -> worker( 0% 76% 76% 74% 76% 76% 42%  0% 75%  0%  0%  0% 76%  0% 76%  0% 76% 76% 15% 74%  0% 76% 76% 23%)
0:09:48 INF [sort] -  chunks: [  16 /  40  40% ] 40G
    cpus: 13.8 gc:  0% heap: 151G/193G direct: 52M postGC: 150G
    ->    (0/65) -> worker( 0% 100% 100% 15% 93% 72%  0%  0% 100%  1%  0%  0% 100%  1% 100%  0% 100% 100%  0% 100%  0% 100% 19%  0%)
0:10:06 INF [sort] -  chunks: [  16 /  40  40% ] 44G
    cpus: 13.6 gc:  0% heap: 151G/193G direct: 52M postGC: 150G
    ->    (0/65) -> worker( 0% 57% 57% 22%  0% 12% 26%  0% 57%  1%  0%  0% 29%  1% 55%  0% 57% 57%  0% 31%  0% 38% 24%  0%)
Killed

[dmitry-voronoff]

Another try, but with mmap

java -Xmx20g -XX:MaxHeapFreeRatio=40 -jar planetiler.jar --force --only-layers=transportation,building,water,landcover --nodemap-type=array --storage=mmap --osm_path=us-latest.osm.pbf --mbtiles=./data/usa.mbtiles

Result

0:00:00 INF - Building BasemapProfile profile into ./data/usa.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to ./data/usa.mbtiles
0:00:00 INF - no wikidata translations found, run with --fetch-wikidata to download
0:00:00 DEB - ✓ 107G storage on / (/dev/vda1) requested for read phase disk, 596G available
0:00:00 DEB -  - 76G used for temporary node location cache
0:00:00 DEB -  - 1.3G used for temporary multipolygon geometry cache
0:00:00 DEB -  - 29G used for temporary feature storage
0:00:00 DEB - ✓ 44G storage on / (/dev/vda1) requested for write phase disk, 596G available
0:00:00 DEB -  - 29G used for temporary feature storage
0:00:00 DEB -  - 14G used for mbtiles output
0:00:00 DEB - ✓ 2.6G JVM heap requested for read phase, 21G available
0:00:00 DEB -  - 2.6G used for temporary profile storage
0:00:00 DEB - ✓ 77G storage on / (/dev/vda1) requested for read phase, 596G available
0:00:00 DEB -  - 76G used for array node location cache (switch to sparsearray to reduce size)
0:00:00 DEB -  - 1.3G used for multipolygon way geometries
0:00:00 DEB - ✓ 77G temporary files and 181G of free memory for OS to cache them

...

0:09:06 DEB [osm_pass2] - Processed 152,773 blocks:
0:09:06 DEB [osm_pass2] -   nodes: 1,112,677,570 (45M/s) in 25s cpu:9m24s avg:22.9
0:09:06 DEB [osm_pass2] -   ways: 108,544,146 (325k/s) in 5m34s cpu:2h4m2s gc:7s avg:22.3
0:09:06 DEB [osm_pass2] -   relations: 944,779 (25k/s) in 37s cpu:14m19s avg:23
0:09:06 INF [osm_pass2] - Finished in 6m37s cpu:2h27m50s gc:8s avg:22.4
0:09:06 INF [osm_pass2] -   read     1x(2% 7s sys:3s wait:6m21s done:10s)
0:09:06 INF [osm_pass2] -   process 23x(93% 6m7s sys:1s wait:17s)
0:09:06 INF [osm_pass2] -   write    1x(60% 3m59s sys:45s wait:2m29s)

...

0:10:45 INF [sort] - Finished in 1m32s cpu:23m58s gc:10s avg:15.7
0:10:45 INF [sort] -   worker  5x(50% 46s sys:4s wait:16s done:2s)
0:10:45 INF - read:109s write:131s sort:205s

...

0:10:55 INF [mbtiles] -  features: [  72M  10% 7.2M/s ] 40G   tiles: [    0    0/s ] 20k
    cpus: 6.1 gc: 15% heap: 10G/21G direct: 18M postGC: 10G
    read(84%) ->    (0/1k) -> encode( 0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0% 13%  0%  0%  0%  0% 14%  0%  0%  0%  0%  0%  0%) ->    (1/1k) -> write( 1%)
    last tile: n/a
0:11:05 INF [mbtiles] -  features: [ 110M  15% 3.8M/s ] 40G   tiles: [    0    0/s ] 20k
    cpus: 15.2 gc: 50% heap: 21G/21G direct: 18M postGC: 21G
    read(49%) ->    (0/1k) -> encode( 0%  0%  0%  0%  9%  0%  0%  0%  0%  4%  0% 49%  5%  0%  0%  0% 49%  0%  0% 40%  0%  0%  0%) ->    (8/1k) -> write( 0%)
    last tile: n/a
0:11:19 INF [mbtiles] -  features: [ 110M  15%    0/s ] 40G   tiles: [    0    0/s ] 20k
    cpus: 17.6 gc: 99% heap: 21G/21G direct: 18M postGC: 21G
    read( 1%) ->    (0/1k) -> encode( 0%  0%  0%  0%  1%  0%  0%  0%  0%  0%  0%  1%  0%  0%  0%  0%  1%  0%  0%  1%  0%  0%  0%) ->    (8/1k) -> write( 0%)
    last tile: n/a
0:11:34 INF [mbtiles] -  features: [ 110M  15%    0/s ] 40G   tiles: [    0    0/s ] 20k
    cpus: 17.6 gc: 100% heap: 21G/21G direct: 18M postGC: 21G
    read( 0%) ->    (0/1k) -> encode( 0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%  0%) ->    (8/1k) -> write( 0%)
    last tile: n/a
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.base/java.lang.Integer.toUnsignedString0(Integer.java:367)
    at java.base/java.lang.Integer.toHexString(Integer.java:284)
    at java.management/java.lang.management.ThreadInfo.initialize(ThreadInfo.java:246)
    at java.management/java.lang.management.ThreadInfo.<init>(ThreadInfo.java:200)
    at java.management/sun.management.ThreadImpl.dumpThreads0(Native Method)
    at java.management/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
    at java.management/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
    at com.onthegomap.planetiler.stats.ProcessInfo.getThreadStats(ProcessInfo.java:228)
    at com.onthegomap.planetiler.stats.ProgressLoggers.lambda$addThreadPoolStats$20(ProgressLoggers.java:306)
    at com.onthegomap.planetiler.stats.ProgressLoggers$$Lambda$404/0x00000008010236d8.get(Unknown Source)
    at com.onthegomap.planetiler.stats.ProgressLoggers$WorkerPipelineLogger.toString(ProgressLoggers.java:407)
    at com.onthegomap.planetiler.stats.ProgressLoggers$$Lambda$422/0x000000080104ded8.apply(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
    at com.onthegomap.planetiler.stats.ProgressLoggers.getLog(ProgressLoggers.java:343)
    at com.onthegomap.planetiler.stats.ProgressLoggers.log(ProgressLoggers.java:337)
    at com.onthegomap.planetiler.stats.ProgressLoggers.awaitAndLog(ProgressLoggers.java:375)
    at com.onthegomap.planetiler.mbtiles.MbtilesWriter.writeOutput(MbtilesWriter.java:195)
    at com.onthegomap.planetiler.mbtiles.MbtilesWriter.writeOutput(MbtilesWriter.java:95)
    at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:549)
    at com.onthegomap.planetiler.basemap.BasemapMain.run(BasemapMain.java:51)
    at com.onthegomap.planetiler.basemap.BasemapMain.main(BasemapMain.java:14)
    at com.onthegomap.planetiler.Main$$Lambda$1/0x0000000800c01800.main(Unknown Source)
    at com.onthegomap.planetiler.Main.main(Main.java:44)
Worker mbtiles_encode-5 died
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap.newNode(HashMap.java:1901)
    at java.base/java.util.HashMap.putVal(HashMap.java:629)
    at java.base/java.util.HashMap.put(HashMap.java:610)
    at com.onthegomap.planetiler.collection.FeatureGroup$TileFeatures.decodeVectorTileFeature(FeatureGroup.java:433)
    at com.onthegomap.planetiler.collection.FeatureGroup$TileFeatures.getVectorTileEncoder(FeatureGroup.java:465)
    at com.onthegomap.planetiler.mbtiles.MbtilesWriter.tileEncoder(MbtilesWriter.java:282)
    at com.onthegomap.planetiler.mbtiles.MbtilesWriter.tileEncoderSink(MbtilesWriter.java:248)
    at com.onthegomap.planetiler.mbtiles.MbtilesWriter$$Lambda$663/0x000000080108dcd0.run(Unknown Source)
    at com.onthegomap.planetiler.worker.WorkerPipeline$Builder.lambda$sinkTo$2(WorkerPipeline.java:275)
    at com.onthegomap.planetiler.worker.WorkerPipeline$Builder$$Lambda$374/0x00000008010165c8.run(Unknown Source)
    at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41)
    at com.onthegomap.planetiler.worker.Worker$$Lambda$364/0x0000000801005668.accept(Unknown Source)
    at com.onthegomap.planetiler.worker.Worker.lambda$new$1(Worker.java:68)
    at com.onthegomap.planetiler.worker.Worker$$Lambda$366/0x0000000801005cf8.run(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)    

[msbarry]

There are 2 memory issues going on there, the first one when it got killed was when the JVM got close to 180gb but sqlite was using some unmanaged memory outside the jvm heap, so the OS killed the whole process for using too much, and the "java heap space" one was because 20g isn't enough to handle encoding such large tiles.

On a 192GB machine I'd suggest first trying java -Xmx150g -XX:MaxHeapFreeRatio=40 ... --nodemap-type=array --storage=ram to leave more free RAM outside the JVM, or if you wanted to try mmap again then java -Xmx50g -jar planetiler.jar ... --nodemap-type=array --storage=mmap to give the JVM more space, while still allowing plenty of free RAM outside the JVM for the OS disk cache.

[hayatsingh-geospoc]

Error processing OSM Way 1024819340 how to tackle this issue ,as i am creating custom mbtiles by using yml file ?
here in key: value i am passing like key: render_height value: '$ { match_value * 3.66 } and getting this error