Closed msbarry closed 3 months ago
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code
Base 6c41c388f4c09dd7097dc8dcdc2e77acbefc81b9 | This Branch 7db687d4c91488462f5a70df7d165c30b1d1d88c |
---|---|
``` 0:01:04 DEB [archive] - Tile stats: 0:01:04 DEB [archive] - Biggest tiles (gzipped) 1. 14/4942/6092 (154k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:82k) 2. 9/154/190 (149k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k) 3. 10/308/380 (138k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k) 4. 10/308/381 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k) 5. 14/4941/6092 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:63k) 6. 14/4941/6093 (110k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k) 7. 14/4940/6092 (99k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k) 8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k) 9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k) 10. 11/616/761 (96k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k) 0:01:04 DEB [archive] - Max tile sizes z0 z1 z2 z3 z4 z5 z6 z7 z8 z9 z10 z11 z12 z13 z14 all boundary 154 374 443 583 938 339 433 548 773 1.6k 2.1k 7.2k 6.4k 5.8k 4.5k 7.2k water 7.7k 3.7k 8.6k 5.5k 2.6k 5.1k 15k 18k 16k 26k 15k 13k 17k 15k 12k 26k place 0 0 441 441 441 650 739 1k 1.7k 3.4k 6k 3.3k 1.8k 824 947 6k landuse 0 0 0 0 548 619 1.3k 5.9k 17k 44k 59k 50k 38k 19k 12k 59k transportation 0 0 0 0 468 918 1.2k 6k 8k 24k 17k 19k 65k 48k 33k 65k waterway 0 0 0 0 111 118 0 0 0 3.1k 2.4k 2.1k 2.1k 4.9k 2.4k 4.9k park 0 0 0 0 0 0 1k 3.6k 9.6k 19k 13k 8.2k 4.3k 3.4k 4.4k 19k transportation_name 0 0 0 0 0 0 369 464 1.2k 1.9k 5.5k 4.6k 4.8k 4.6k 18k 18k landcover 0 0 0 0 0 0 0 9.5k 29k 85k 72k 81k 53k 30k 24k 85k mountain_peak 0 0 0 0 0 0 0 1.1k 1.8k 3.4k 4.3k 2.8k 1.4k 1.4k 869 4.3k water_name 0 0 0 0 0 0 0 0 0 486 461 433 452 1.2k 1.5k 1.5k aerodrome_label 0 0 0 0 0 0 0 0 0 0 664 327 273 220 220 664 aeroway 0 0 0 0 0 0 0 0 0 0 1.6k 2.1k 3k 3.4k 2.7k 3.4k poi 0 0 0 0 0 0 0 0 0 0 0 0 501 498 82k 82k building 0 0 0 0 0 0 0 0 0 0 0 0 0 59k 92k 92k housenumber 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35k 35k full tile 7.9k 4k 9.5k 6.5k 3.8k 6.1k 20k 41k 84k 203k 185k 135k 114k 129k 243k 243k gzipped 6.2k 3.5k 7.1k 5.2k 3.2k 4.9k 13k 29k 60k 149k 138k 99k 83k 92k 154k 154k 0:01:04 DEB [archive] - Max tile: 243k (gzipped: 154k) 0:01:04 DEB [archive] - Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic 0:01:04 DEB [archive] - # tiles: 4,115,053 0:01:04 DEB [archive] - # features: 6,087,709 0:01:04 INF [archive] - Finished in 19s cpu:1m10s avg:3.7 0:01:04 INF [archive] - read 1x(3% 0.6s wait:17s) 0:01:04 INF [archive] - encode 4x(56% 11s wait:2s) 0:01:04 INF [archive] - write 1x(21% 4s wait:13s) 0:01:04 INF - Finished in 1m4s cpu:3m30s gc:1s avg:3.3 0:01:04 INF - FINISHED! 0:01:04 INF - 0:01:04 INF - ---------------------------------------- 0:01:04 INF - data errors: 0:01:04 INF - render_snap_fix_input 16,642 0:01:04 INF - osm_multipolygon_missing_way 74 0:01:04 INF - osm_boundary_missing_way 70 0:01:04 INF - merge_snap_fix_input 14 0:01:04 INF - feature_polygon_osm_invalid_multipolygon_empty_after_fix 3 0:01:04 INF - feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix 2 0:01:04 INF - omt_park_area_osm_invalid_multipolygon_empty_after_fix 1 0:01:04 INF - ---------------------------------------- 0:01:04 INF - overall 1m4s cpu:3m30s gc:1s avg:3.3 0:01:04 INF - lake_centerlines 2s cpu:5s avg:2.4 0:01:04 INF - read 1x(22% 0.5s done:2s) 0:01:04 INF - process 4x(0% 0s done:2s) 0:01:04 INF - write 1x(0% 0s done:2s) 0:01:04 INF - water_polygons 15s cpu:42s avg:2.8 0:01:04 INF - read 1x(40% 6s done:8s) 0:01:04 INF - process 4x(28% 4s wait:4s done:5s) 0:01:04 INF - write 1x(4% 0.5s wait:10s done:5s) 0:01:04 INF - natural_earth 6s cpu:12s avg:1.9 0:01:04 INF - read 1x(95% 6s) 0:01:04 INF - process 4x(13% 0.8s wait:6s) 0:01:04 INF - write 1x(0% 0s wait:6s) 0:01:04 INF - osm_pass1 2s cpu:6s avg:3.2 0:01:04 INF - read 1x(2% 0s wait:2s) 0:01:04 INF - parse 4x(36% 0.7s) 0:01:04 INF - process 1x(65% 1s) 0:01:04 INF - osm_pass2 18s cpu:1m9s avg:3.9 0:01:04 INF - read 1x(0% 0s wait:10s done:7s) 0:01:04 INF - process 4x(76% 13s) 0:01:04 INF - write 1x(4% 0.6s wait:17s) 0:01:04 INF - boundaries 0s cpu:0s avg:2.1 0:01:04 INF - agg_stop 0s cpu:0s avg:0 0:01:04 INF - sort 2s cpu:4s avg:2.6 0:01:04 INF - worker 1x(55% 0.9s) 0:01:04 INF - archive 19s cpu:1m10s avg:3.7 0:01:04 INF - read 1x(3% 0.6s wait:17s) 0:01:04 INF - encode 4x(56% 11s wait:2s) 0:01:04 INF - write 1x(21% 4s wait:13s) 0:01:04 INF - ---------------------------------------- 0:01:04 INF - archive 108MB 0:01:04 INF - features 360MB -rw-r--r-- 1 runner docker 70M Apr 3 00:23 run.jar ``` | ``` 0:01:09 DEB [archive] - Tile stats: 0:01:09 DEB [archive] - Biggest tiles (gzipped) 1. 14/4942/6092 (154k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:82k) 2. 9/154/190 (149k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k) 3. 10/308/380 (138k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k) 4. 10/308/381 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k) 5. 14/4941/6092 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:63k) 6. 14/4941/6093 (110k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k) 7. 14/4940/6092 (99k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k) 8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k) 9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k) 10. 11/616/761 (96k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k) 0:01:09 DEB [archive] - Max tile sizes z0 z1 z2 z3 z4 z5 z6 z7 z8 z9 z10 z11 z12 z13 z14 all boundary 154 374 443 583 938 339 433 548 773 1.6k 2.1k 7.2k 6.4k 5.8k 4.5k 7.2k water 7.7k 3.7k 8.6k 5.5k 2.6k 5.1k 15k 18k 16k 26k 15k 13k 17k 15k 12k 26k place 0 0 441 441 441 650 739 1k 1.7k 3.4k 6k 3.3k 1.8k 824 947 6k landuse 0 0 0 0 548 619 1.3k 5.9k 17k 44k 59k 50k 38k 19k 12k 59k transportation 0 0 0 0 468 918 1.2k 6k 8k 24k 17k 19k 65k 48k 33k 65k waterway 0 0 0 0 111 118 0 0 0 3.1k 2.4k 2.1k 2.1k 4.9k 2.4k 4.9k park 0 0 0 0 0 0 1k 3.6k 9.6k 19k 13k 8.2k 4.3k 3.4k 4.4k 19k transportation_name 0 0 0 0 0 0 369 464 1.2k 1.9k 5.5k 4.6k 4.8k 4.6k 18k 18k landcover 0 0 0 0 0 0 0 9.5k 29k 85k 72k 81k 53k 30k 24k 85k mountain_peak 0 0 0 0 0 0 0 1.1k 1.8k 3.4k 4.3k 2.8k 1.4k 1.4k 869 4.3k water_name 0 0 0 0 0 0 0 0 0 486 461 433 452 1.2k 1.5k 1.5k aerodrome_label 0 0 0 0 0 0 0 0 0 0 664 327 273 220 220 664 aeroway 0 0 0 0 0 0 0 0 0 0 1.6k 2.1k 3k 3.4k 2.7k 3.4k poi 0 0 0 0 0 0 0 0 0 0 0 0 501 498 82k 82k building 0 0 0 0 0 0 0 0 0 0 0 0 0 59k 92k 92k housenumber 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35k 35k full tile 7.9k 4k 9.5k 6.5k 3.8k 6.1k 20k 41k 84k 203k 185k 135k 114k 129k 243k 243k gzipped 6.2k 3.5k 7.1k 5.2k 3.2k 4.9k 13k 29k 60k 149k 138k 99k 83k 92k 154k 154k 0:01:09 DEB [archive] - Max tile: 243k (gzipped: 154k) 0:01:09 DEB [archive] - Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic 0:01:09 DEB [archive] - # tiles: 4,115,053 0:01:09 DEB [archive] - # features: 6,087,709 0:01:09 INF [archive] - Finished in 19s cpu:1m9s avg:3.7 0:01:09 INF [archive] - read 1x(3% 0.6s wait:17s done:1s) 0:01:09 INF [archive] - encode 4x(56% 11s wait:2s) 0:01:09 INF [archive] - write 1x(21% 4s wait:13s) 0:01:09 INF - Finished in 1m9s cpu:3m31s gc:1s avg:3 0:01:09 INF - FINISHED! 0:01:09 INF - 0:01:09 INF - ---------------------------------------- 0:01:09 INF - data errors: 0:01:09 INF - render_snap_fix_input 16,642 0:01:09 INF - osm_multipolygon_missing_way 74 0:01:09 INF - osm_boundary_missing_way 70 0:01:09 INF - merge_snap_fix_input 14 0:01:09 INF - feature_polygon_osm_invalid_multipolygon_empty_after_fix 3 0:01:09 INF - feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix 2 0:01:09 INF - omt_park_area_osm_invalid_multipolygon_empty_after_fix 1 0:01:09 INF - ---------------------------------------- 0:01:09 INF - overall 1m9s cpu:3m31s gc:1s avg:3 0:01:09 INF - lake_centerlines 3s cpu:6s avg:2 0:01:09 INF - read 1x(16% 0.5s done:2s) 0:01:09 INF - process 4x(0% 0s done:2s) 0:01:09 INF - write 1x(0% 0s done:2s) 0:01:09 INF - water_polygons 14s cpu:38s avg:2.7 0:01:09 INF - read 1x(41% 6s done:7s) 0:01:09 INF - process 4x(26% 4s wait:4s done:5s) 0:01:09 INF - write 1x(4% 0.5s wait:9s done:5s) 0:01:09 INF - natural_earth 12s cpu:18s avg:1.5 0:01:09 INF - read 1x(52% 6s done:5s) 0:01:09 INF - process 4x(7% 0.8s wait:6s done:5s) 0:01:09 INF - write 1x(0% 0s wait:6s done:5s) 0:01:09 INF - osm_pass1 2s cpu:6s avg:3.3 0:01:09 INF - read 1x(2% 0s wait:2s) 0:01:09 INF - parse 4x(34% 0.6s) 0:01:09 INF - process 1x(69% 1s) 0:01:09 INF - osm_pass2 18s cpu:1m9s avg:3.9 0:01:09 INF - read 1x(0% 0s wait:10s done:7s) 0:01:09 INF - process 4x(75% 13s) 0:01:09 INF - write 1x(3% 0.6s wait:17s) 0:01:09 INF - boundaries 0s cpu:0s avg:1 0:01:09 INF - agg_stop 0s cpu:0s avg:0 0:01:09 INF - sort 2s cpu:4s avg:2.4 0:01:09 INF - worker 1x(50% 0.9s) 0:01:09 INF - archive 19s cpu:1m9s avg:3.7 0:01:09 INF - read 1x(3% 0.6s wait:17s done:1s) 0:01:09 INF - encode 4x(56% 11s wait:2s) 0:01:09 INF - write 1x(21% 4s wait:13s) 0:01:09 INF - ---------------------------------------- 0:01:09 INF - archive 108MB 0:01:09 INF - features 360MB -rw-r--r-- 1 runner docker 70M Apr 3 00:22 run.jar ``` |
https://github.com/onthegomap/planetiler/actions/runs/8531224355
There was a memory leak when generating TileSizeStats TSV output during a planetiler run. The jackson
ObjectWriter#writeValueAsString
methodJsonFactory#_getBufferRecycler()
which usesLockFreePool#acquireAndLinkPooled
to get aBufferRecycler
. This tried to share up to 4 recyclers between threads, but somewhere around z13-15 tile generation, it stops working and starts creating a newBufferRecycler
per row written, which ends up making a lot ofchar[]
instances, causing the JVM to run out of memory. This change uses a newCsvWriter
andObjectWriter
per thread so there's no race condition to cause out of control allocations.