Closed 08-15at closed 3 months ago
Have done a whole Europe import and am now trying to prerender the tiles using reder_list. This works fine until ZoomLevel 10. I have 6 threads running and renderd consumes about 10-15GB. around tile 528 memory consumption jumps to more than 48GB which causes oom_killer (renderd runs in a proxmox CT) to kill renderd. Jun 5 08:38:17 renderd renderd[1882]: Rendering projected coordinates 10 528 368 -> 626172.135713|5322463.153556 939258.203569|5635549.221413 to a 8 x 8 tile Jun 5 08:40:29 renderd systemd[1]: renderd.service: Main process exited, code=killed, status=9/KILL Is this a mod_tile/renderd or mapnik issue. Can I do something about this ? thx
I can reproduce the descriped behaviour with my installation. I am using docker project for setup of the tileserver and observed high RAM memory consumption of the container. A closer look into container shows that memory consumtion of renderd service seems to grow up over the time until oom killer stops the service. Depending from the load it will happen after hours or weeks on my system. Have found a solution / workaround for that?
note: There is another user who reportet this issue.
@stevo01 Can you describe in a bit more detail what you are doing? Can you reproduce the problem in a smaller system? I don't have anywhere I could import Europe in and allocate 48Gb to, but if you could reproduce the problem in a smaller container (with less data and less memory) other people might be able to investigate.
Hi,
@stevo01 Can you describe in a bit more detail what you are doing? Can you reproduce the problem in a smaller system? I don't have anywhere I could import Europe in and allocate 48Gb to, but if you could reproduce the problem in a smaller container (with less data and less memory) other people might be able to investigate.
I wrote description how to setup the docker based tile server with import of osm extract (just germany). This allows you to setup the server in some minutes if docker is available. The initial import of germany needs around 3 hours on my workstation (amd ryzen 1700).
I triggert the renderer with command render_list -a -m ajt -z 11 -Z 11 -l 800 -n 12
and analysed memory usage of renderd with command: pmap 1370 > renderd.memory.001.txt before tiles where rendered and command pmap 1370 > renderd.memory.004.txt after renderd handled around 65536 tiles.
The memory consumtion after renderd startup is 2089460K. The memory consumtion after renderd produced the 65536 tiles is 7725556K.
See attachments for details. renderd.memory.001.txt renderd.memory.004.txt renderd.stats.txt
Having the same issue
This is my render command: render_list -a -f -m ajt -z 15 -Z 20 -n 18
I am using this docker image: Overv/openstreetmap-tile-server
The docker image is currently consuming 16.82GiB and rising.
There are currently 20 pids running renderd -f -c /usr/local/etc/renderd.conf
:
This is the output of pmap 2259
2259: renderd -f -c /usr/local/etc/renderd.conf
00005631de76d000 76K r-x-- renderd
00005631de97f000 4K r---- renderd
00005631de980000 4K rw--- renderd
00005631df322000 17572K rw--- [ anon ]
00007f3970000000 94944K rw--- [ anon ]
00007f3975cb8000 36128K ----- [ anon ]
00007f3978000000 65456K rw--- [ anon ]
00007f397bfec000 80K ----- [ anon ]
00007f397c000000 65536K rw--- [ anon ]
00007f3980000000 65536K rw--- [ anon ]
00007f3988000000 65536K rw--- [ anon ]
00007f3990000000 65536K rw--- [ anon ]
00007f3998000000 65536K rw--- [ anon ]
00007f39a0000000 65452K rw--- [ anon ]
00007f39a3feb000 84K ----- [ anon ]
00007f39a8000000 65536K rw--- [ anon ]
00007f39b0000000 65536K rw--- [ anon ]
00007f39b8000000 65536K rw--- [ anon ]
00007f39c0000000 65536K rw--- [ anon ]
00007f39c8000000 119844K rw--- [ anon ]
00007f39cf509000 11228K ----- [ anon ]
00007f39d0000000 65536K rw--- [ anon ]
00007f39d8000000 65536K rw--- [ anon ]
00007f39e0000000 65532K rw--- [ anon ]
00007f39e3fff000 4K ----- [ anon ]
00007f39e8000000 65536K rw--- [ anon ]
00007f39f0000000 65536K rw--- [ anon ]
00007f39f8000000 65536K rw--- [ anon ]
00007f3a00000000 65536K rw--- [ anon ]
00007f3a08000000 131072K rw--- [ anon ]
00007f3a10000000 131072K rw--- [ anon ]
00007f3a30000000 65536K rw--- [ anon ]
00007f3a38000000 65536K rw--- [ anon ]
00007f3a40000000 65532K rw--- [ anon ]
00007f3a43fff000 4K ----- [ anon ]
00007f3a48000000 65536K rw--- [ anon ]
00007f3a50000000 59012K rw--- [ anon ]
00007f3a539a1000 6524K ----- [ anon ]
00007f3a58000000 64548K rw--- [ anon ]
00007f3a5bf09000 988K ----- [ anon ]
00007f3a60000000 65500K rw--- [ anon ]
00007f3a63ff7000 36K ----- [ anon ]
00007f3a64000000 65536K rw--- [ anon ]
00007f3a68000000 2832K rw--- [ anon ]
00007f3a682c4000 62704K ----- [ anon ]
00007f3a70000000 131072K rw--- [ anon ]
00007f3a78000000 131072K rw--- [ anon ]
00007f3a80000000 131072K rw--- [ anon ]
00007f3a88000000 65536K rw--- [ anon ]
00007f3a90000000 131072K rw--- [ anon ]
00007f3a98000000 65536K rw--- [ anon ]
00007f3aa0000000 65536K rw--- [ anon ]
00007f3aa8000000 65508K rw--- [ anon ]
00007f3aabff9000 28K ----- [ anon ]
00007f3ab0000000 131072K rw--- [ anon ]
00007f3ab8000000 65536K rw--- [ anon ]
00007f3ac0000000 65536K rw--- [ anon ]
00007f3ac8000000 65536K rw--- [ anon ]
00007f3ad0000000 65536K rw--- [ anon ]
00007f3ad8000000 58072K rw--- [ anon ]
00007f3adb8b6000 7464K ----- [ anon ]
00007f3ae0000000 65536K rw--- [ anon ]
00007f3ae8000000 65536K rw--- [ anon ]
00007f3af0000000 65536K rw--- [ anon ]
00007f3af8000000 65536K rw--- [ anon ]
00007f3b00000000 65536K rw--- [ anon ]
00007f3b08000000 65536K rw--- [ anon ]
00007f3b10000000 65536K rw--- [ anon ]
00007f3b14000000 65108K rw--- [ anon ]
00007f3b17f95000 428K ----- [ anon ]
00007f3b18000000 20228K rw--- [ anon ]
00007f3b193c1000 45308K ----- [ anon ]
00007f3b1c000000 65536K rw--- [ anon ]
00007f3b20000000 65536K rw--- [ anon ]
00007f3b28000000 65536K rw--- [ anon ]
00007f3b2c000000 65536K rw--- [ anon ]
00007f3b30000000 65536K rw--- [ anon ]
00007f3b34000000 65536K rw--- [ anon ]
00007f3b38000000 65536K rw--- [ anon ]
00007f3b3c000000 65536K rw--- [ anon ]
00007f3b40000000 131072K rw--- [ anon ]
00007f3b48000000 65532K rw--- [ anon ]
00007f3b4bfff000 4K ----- [ anon ]
00007f3b50000000 43144K rw--- [ anon ]
00007f3b52a22000 22392K ----- [ anon ]
00007f3b58000000 131072K rw--- [ anon ]
00007f3b60000000 131072K rw--- [ anon ]
00007f3b68000000 131072K rw--- [ anon ]
00007f3b70000000 130488K rw--- [ anon ]
00007f3b77f6e000 584K ----- [ anon ]
00007f3b78000000 65536K rw--- [ anon ]
00007f3b80000000 65536K rw--- [ anon ]
00007f3b88000000 131072K rw--- [ anon ]
00007f3b90000000 65460K rw--- [ anon ]
00007f3b93fed000 76K ----- [ anon ]
00007f3b98000000 65536K rw--- [ anon ]
00007f3ba0000000 65536K rw--- [ anon ]
00007f3ba8000000 93892K rw--- [ anon ]
00007f3badbb1000 37180K ----- [ anon ]
00007f3bb0000000 123660K rw--- [ anon ]
00007f3bb78c3000 7412K ----- [ anon ]
00007f3bb8000000 62964K rw--- [ anon ]
00007f3bbbd7d000 2572K ----- [ anon ]
00007f3bbc000000 63100K rw--- [ anon ]
00007f3bbfd9f000 2436K ----- [ anon ]
00007f3bc0000000 35560K rw--- [ anon ]
00007f3bc22ba000 29976K ----- [ anon ]
00007f3bc8000000 65536K rw--- [ anon ]
00007f3bcc000000 65532K rw--- [ anon ]
00007f3bcffff000 4K ----- [ anon ]
00007f3bd0000000 131072K rw--- [ anon ]
00007f3bd8000000 65536K rw--- [ anon ]
00007f3bdc000000 62816K rw--- [ anon ]
00007f3bdfd58000 2720K ----- [ anon ]
00007f3be0000000 65536K rw--- [ anon ]
00007f3be8000000 65536K rw--- [ anon ]
00007f3bec000000 65536K rw--- [ anon ]
00007f3bf0000000 131072K rw--- [ anon ]
00007f3bf8000000 111612K rw--- [ anon ]
00007f3bfecff000 19460K ----- [ anon ]
00007f3c00000000 131072K rw--- [ anon ]
00007f3c08000000 65536K rw--- [ anon ]
00007f3c0c000000 53256K rw--- [ anon ]
00007f3c0f402000 12280K ----- [ anon ]
00007f3c10000000 48780K rw--- [ anon ]
00007f3c12fa3000 16756K ----- [ anon ]
00007f3c14000000 65536K rw--- [ anon ]
00007f3c18000000 65536K rw--- [ anon ]
00007f3c1c000000 65536K rw--- [ anon ]
00007f3c20000000 65520K rw--- [ anon ]
00007f3c23ffc000 16K ----- [ anon ]
00007f3c24000000 65536K rw--- [ anon ]
00007f3c28000000 65536K rw--- [ anon ]
00007f3c2c000000 65536K rw--- [ anon ]
00007f3c30000000 65536K rw--- [ anon ]
00007f3c38000000 65536K rw--- [ anon ]
00007f3c3c000000 63220K rw--- [ anon ]
00007f3c3fdbd000 2316K ----- [ anon ]
00007f3c40000000 65536K rw--- [ anon ]
00007f3c44000000 25504K rw--- [ anon ]
00007f3c458e8000 40032K ----- [ anon ]
00007f3c48000000 65536K rw--- [ anon ]
00007f3c50000000 131072K rw--- [ anon ]
00007f3c58000000 131072K rw--- [ anon ]
00007f3c60000000 65536K rw--- [ anon ]
00007f3c64000000 2864K rw--- [ anon ]
00007f3c642cc000 62672K ----- [ anon ]
00007f3c68000000 65536K rw--- [ anon ]
00007f3c6c000000 65536K rw--- [ anon ]
00007f3c70000000 64312K rw--- [ anon ]
00007f3c73ece000 1224K ----- [ anon ]
00007f3c74000000 65536K rw--- [ anon ]
00007f3c78000000 65536K rw--- [ anon ]
00007f3c7c000000 65536K rw--- [ anon ]
00007f3c80000000 131072K rw--- [ anon ]
00007f3c88000000 30980K rw--- [ anon ]
00007f3c89e41000 34556K ----- [ anon ]
00007f3c8c000000 65536K rw--- [ anon ]
00007f3c90000000 65536K rw--- [ anon ]
00007f3c94000000 65536K rw--- [ anon ]
00007f3c98000000 65536K rw--- [ anon ]
00007f3ca0000000 25520K rw--- [ anon ]
00007f3ca18ec000 40016K ----- [ anon ]
00007f3ca8000000 65512K rw--- [ anon ]
00007f3cabffa000 24K ----- [ anon ]
00007f3cac000000 65304K rw--- [ anon ]
00007f3caffc6000 232K ----- [ anon ]
00007f3cb0000000 65536K rw--- [ anon ]
00007f3cb4000000 65528K rw--- [ anon ]
00007f3cb7ffe000 8K ----- [ anon ]
00007f3cb8000000 65528K rw--- [ anon ]
00007f3cbbffe000 8K ----- [ anon ]
00007f3cbc000000 65536K rw--- [ anon ]
00007f3cc0000000 131072K rw--- [ anon ]
00007f3cc8000000 128632K rw--- [ anon ]
00007f3ccfd9e000 2440K ----- [ anon ]
00007f3cd0000000 131072K rw--- [ anon ]
00007f3cd8000000 131072K rw--- [ anon ]
00007f3ce0000000 65536K rw--- [ anon ]
00007f3ce8000000 65536K rw--- [ anon ]
00007f3cf0000000 65536K rw--- [ anon ]
00007f3cf8000000 65536K rw--- [ anon ]
00007f3d00000000 65536K rw--- [ anon ]
00007f3d08000000 65536K rw--- [ anon ]
00007f3d10000000 65536K rw--- [ anon ]
00007f3d18000000 65536K rw--- [ anon ]
00007f3d1c000000 65536K rw--- [ anon ]
00007f3d20000000 131072K rw--- [ anon ]
00007f3d2c000000 55192K rw--- [ anon ]
00007f3d2f5e6000 10344K ----- [ anon ]
00007f3d30000000 65536K rw--- [ anon ]
00007f3d38000000 65536K rw--- [ anon ]
00007f3d3c000000 65536K rw--- [ anon ]
00007f3d44000000 65536K rw--- [ anon ]
00007f3d4c000000 131072K rw--- [ anon ]
00007f3d54000000 65536K rw--- [ anon ]
00007f3d5c000000 65536K rw--- [ anon ]
00007f3d64000000 65536K rw--- [ anon ]
00007f3d6c000000 65536K rw--- [ anon ]
00007f3d74000000 65536K rw--- [ anon ]
00007f3d7c000000 65536K rw--- [ anon ]
00007f3d80000000 65536K rw--- [ anon ]
00007f3d84000000 65536K rw--- [ anon ]
00007f3d88000000 53780K rw--- [ anon ]
00007f3d8b485000 11756K ----- [ anon ]
00007f3d90000000 65536K rw--- [ anon ]
00007f3d94000000 65536K rw--- [ anon ]
00007f3d98000000 64940K rw--- [ anon ]
00007f3d9bf6b000 596K ----- [ anon ]
00007f3da0000000 65536K rw--- [ anon ]
00007f3da4000000 65536K rw--- [ anon ]
00007f3dac000000 65536K rw--- [ anon ]
00007f3db4000000 65536K rw--- [ anon ]
00007f3dbc000000 64288K rw--- [ anon ]
00007f3dbfec8000 1248K ----- [ anon ]
00007f3dc4000000 65536K rw--- [ anon ]
00007f3dc8000000 65532K rw--- [ anon ]
00007f3dcbfff000 4K ----- [ anon ]
00007f3dd2ece000 70980K r--s- icesheet_outlines.shp
00007f3dd741f000 861580K r--s- water_polygons.shp
00007f3e0bd82000 68088K r--s- icesheet_polygons.shp
00007f3e10000000 56788K rw--- [ anon ]
00007f3e13775000 8748K ----- [ anon ]
00007f3e14000000 56888K rw--- [ anon ]
00007f3e1778e000 8648K ----- [ anon ]
00007f3e18000000 56892K rw--- [ anon ]
00007f3e1b78f000 8644K ----- [ anon ]
00007f3e1c000000 56788K rw--- [ anon ]
00007f3e1f775000 8748K ----- [ anon ]
00007f3e20000000 56888K rw--- [ anon ]
00007f3e2378e000 8648K ----- [ anon ]
00007f3e24000000 56788K rw--- [ anon ]
00007f3e27775000 8748K ----- [ anon ]
00007f3e28000000 56788K rw--- [ anon ]
00007f3e2b775000 8748K ----- [ anon ]
00007f3e2d2d4000 29864K r--s- simplified_water_polygons.shp
00007f3e2effe000 4K ----- [ anon ]
00007f3e2efff000 8192K rw--- [ anon ]
00007f3e30000000 56892K rw--- [ anon ]
00007f3e3378f000 8644K ----- [ anon ]
00007f3e349d6000 6288K r--s- icesheet_outlines.dbf
00007f3e34ffa000 4K ----- [ anon ]
00007f3e34ffb000 8192K rw--- [ anon ]
00007f3e357fb000 4K ----- [ anon ]
00007f3e357fc000 8192K rw--- [ anon ]
00007f3e35ffc000 4K ----- [ anon ]
00007f3e35ffd000 8192K rw--- [ anon ]
00007f3e367fd000 4K ----- [ anon ]
00007f3e367fe000 8192K rw--- [ anon ]
00007f3e36ffe000 4K ----- [ anon ]
00007f3e36fff000 8192K rw--- [ anon ]
00007f3e377ff000 4K ----- [ anon ]
00007f3e37800000 8192K rw--- [ anon ]
00007f3e38000000 56892K rw--- [ anon ]
00007f3e3b78f000 8644K ----- [ anon ]
00007f3e3c000000 56788K rw--- [ anon ]
00007f3e3f775000 8748K ----- [ anon ]
00007f3e40000000 56888K rw--- [ anon ]
00007f3e4378e000 8648K ----- [ anon ]
00007f3e44000000 56796K rw--- [ anon ]
00007f3e47777000 8740K ----- [ anon ]
00007f3e48000000 56940K rw--- [ anon ]
00007f3e4b79b000 8596K ----- [ anon ]
00007f3e4c000000 56888K rw--- [ anon ]
00007f3e4f78e000 8648K ----- [ anon ]
00007f3e50000000 56888K rw--- [ anon ]
00007f3e5378e000 8648K ----- [ anon ]
00007f3e54000000 56952K rw--- [ anon ]
00007f3e5779e000 8584K ----- [ anon ]
00007f3e5844d000 1016K r--s- icesheet_outlines.index
00007f3e5854b000 624K r--s- icesheet_outlines.shx
00007f3e585e7000 44K r-x-- libnss_files-2.27.so
00007f3e585f2000 2044K ----- libnss_files-2.27.so
00007f3e587f1000 4K r---- libnss_files-2.27.so
00007f3e587f2000 4K rw--- libnss_files-2.27.so
00007f3e587f3000 24K rw--- [ anon ]
00007f3e587f9000 4K ----- [ anon ]
00007f3e587fa000 8192K rw--- [ anon ]
00007f3e58ffa000 4K ----- [ anon ]
00007f3e58ffb000 8192K rw--- [ anon ]
00007f3e597fb000 4K ----- [ anon ]
00007f3e597fc000 8192K rw--- [ anon ]
00007f3e59ffc000 4K ----- [ anon ]
00007f3e59ffd000 8192K rw--- [ anon ]
00007f3e5a7fd000 4K ----- [ anon ]
00007f3e5a7fe000 8192K rw--- [ anon ]
00007f3e5affe000 4K ----- [ anon ]
00007f3e5afff000 8192K rw--- [ anon ]
00007f3e5b7ff000 4K ----- [ anon ]
00007f3e5b800000 8192K rw--- [ anon ]
00007f3e5c000000 56788K rw--- [ anon ]
00007f3e5f775000 8748K ----- [ anon ]
00007f3e60000000 56788K rw--- [ anon ]
00007f3e63775000 8748K ----- [ anon ]
00007f3e64000000 132K rw--- [ anon ]
00007f3e64021000 65404K ----- [ anon ]
00007f3e6809b000 492K r--s- water_polygons.index
00007f3e68116000 92K r-x-- libnsl-2.27.so
00007f3e6812d000 2044K ----- libnsl-2.27.so
00007f3e6832c000 4K r---- libnsl-2.27.so
00007f3e6832d000 4K rw--- libnsl-2.27.so
00007f3e6832e000 8K rw--- [ anon ]
00007f3e68330000 44K r-x-- libnss_nis-2.27.so
00007f3e6833b000 2044K ----- libnss_nis-2.27.so
00007f3e6853a000 4K r---- libnss_nis-2.27.so
00007f3e6853b000 4K rw--- libnss_nis-2.27.so
00007f3e6853c000 32K r-x-- libnss_compat-2.27.so
00007f3e68544000 2048K ----- libnss_compat-2.27.so
00007f3e68744000 4K r---- libnss_compat-2.27.so
00007f3e68745000 4K rw--- libnss_compat-2.27.so
00007f3e68746000 4K ----- [ anon ]
00007f3e68747000 8192K rw--- [ anon ]
00007f3e68f47000 4K ----- [ anon ]
00007f3e68f48000 8192K rw--- [ anon ]
00007f3e69748000 4K ----- [ anon ]
00007f3e69749000 8192K rw--- [ anon ]
00007f3e69f49000 4K ----- [ anon ]
00007f3e69f4a000 8192K rw--- [ anon ]
00007f3e6a74a000 4K ----- [ anon ]
00007f3e6a74b000 8192K rw--- [ anon ]
00007f3e6af4b000 168K r-x-- shape.input
00007f3e6af75000 2048K ----- shape.input
00007f3e6b175000 4K r---- shape.input
00007f3e6b176000 4K rw--- shape.input
00007f3e6b177000 192K r-x-- postgis.input
00007f3e6b1a7000 2044K ----- postgis.input
00007f3e6b3a6000 8K r---- postgis.input
00007f3e6b3a8000 4K rw--- postgis.input
00007f3e6b3a9000 384K r-x-- geojson.input
00007f3e6b409000 2048K ----- geojson.input
00007f3e6b609000 8K r---- geojson.input
00007f3e6b60b000 4K rw--- geojson.input
00007f3e6b60c000 16K rw--- [ anon ]
00007f3e6b610000 152K r-x-- pgraster.input
00007f3e6b636000 2048K ----- pgraster.input
00007f3e6b836000 4K r---- pgraster.input
00007f3e6b837000 4K rw--- pgraster.input
00007f3e6b838000 428K r-x-- csv.input
00007f3e6b8a3000 2044K ----- csv.input
00007f3e6baa2000 8K r---- csv.input
00007f3e6baa4000 4K rw--- csv.input
00007f3e6baa5000 4K rw--- [ anon ]
00007f3e6baa6000 136K r-x-- ogr.input
00007f3e6bac8000 2048K ----- ogr.input
00007f3e6bcc8000 4K r---- ogr.input
00007f3e6bcc9000 4K rw--- ogr.input
00007f3e6bcca000 128K r-x-- sqlite.input
00007f3e6bcea000 2048K ----- sqlite.input
00007f3e6beea000 4K r---- sqlite.input
00007f3e6beeb000 4K rw--- sqlite.input
00007f3e6beec000 252K r-x-- libquadmath.so.0.0.0
00007f3e6bf2b000 2044K ----- libquadmath.so.0.0.0
00007f3e6c12a000 4K r---- libquadmath.so.0.0.0
00007f3e6c12b000 4K rw--- libquadmath.so.0.0.0
00007f3e6c12c000 28K r-x-- libaec.so.0.0.3
00007f3e6c133000 2044K ----- libaec.so.0.0.3
00007f3e6c332000 4K r---- libaec.so.0.0.3
00007f3e6c333000 4K rw--- libaec.so.0.0.3
00007f3e6c334000 12K r-x-- libplds4.so
00007f3e6c337000 2044K ----- libplds4.so
00007f3e6c536000 4K r---- libplds4.so
00007f3e6c537000 4K rw--- libplds4.so
00007f3e6c538000 16K r-x-- libplc4.so
00007f3e6c53c000 2044K ----- libplc4.so
00007f3e6c73b000 4K r---- libplc4.so
00007f3e6c73c000 4K rw--- libplc4.so
00007f3e6c73d000 160K r-x-- libnssutil3.so
00007f3e6c765000 2044K ----- libnssutil3.so
00007f3e6c964000 28K r---- libnssutil3.so
00007f3e6c96b000 4K rw--- libnssutil3.so
00007f3e6c96c000 1904K r-x-- libgfortran.so.4.0.0
00007f3e6cb48000 2048K ----- libgfortran.so.4.0.0
00007f3e6cd48000 4K r---- libgfortran.so.4.0.0
00007f3e6cd49000 4K rw--- libgfortran.so.4.0.0
00007f3e6cd4a000 4K rw--- [ anon ]
00007f3e6cd4b000 24K r-x-- libfygm.so.0.0.0
00007f3e6cd51000 2044K ----- libfygm.so.0.0.0
00007f3e6cf50000 4K r---- libfygm.so.0.0.0
00007f3e6cf51000 4K rw--- libfygm.so.0.0.0
00007f3e6cf52000 32K r-x-- libfyut.so.0.0.0
00007f3e6cf5a000 2044K ----- libfyut.so.0.0.0
00007f3e6d159000 4K r---- libfyut.so.0.0.0
00007f3e6d15a000 4K rw--- libfyut.so.0.0.0
00007f3e6d15b000 2668K r-x-- libcrypto.so.1.1
00007f3e6d3f6000 2044K ----- libcrypto.so.1.1
00007f3e6d5f5000 176K r---- libcrypto.so.1.1
00007f3e6d621000 8K rw--- libcrypto.so.1.1
00007f3e6d623000 12K rw--- [ anon ]
00007f3e6d626000 512K r-x-- libssl.so.1.1
00007f3e6d6a6000 2048K ----- libssl.so.1.1
00007f3e6d8a6000 36K r---- libssl.so.1.1
00007f3e6d8af000 16K rw--- libssl.so.1.1
00007f3e6d8b3000 8K r-x-- libsz.so.2.0.1
00007f3e6d8b5000 2044K ----- libsz.so.2.0.1
00007f3e6dab4000 4K r---- libsz.so.2.0.1
00007f3e6dab5000 4K rw--- libsz.so.2.0.1
00007f3e6dab6000 132K r-x-- libhdf5_serial_hl.so.100.0.0
00007f3e6dad7000 2044K ----- libhdf5_serial_hl.so.100.0.0
00007f3e6dcd6000 4K r---- libhdf5_serial_hl.so.100.0.0
00007f3e6dcd7000 4K rw--- libhdf5_serial_hl.so.100.0.0
00007f3e6dcd8000 4K rw--- [ anon ]
00007f3e6dcd9000 104K r-x-- liburiparser.so.1.0.20
00007f3e6dcf3000 2044K ----- liburiparser.so.1.0.20
00007f3e6def2000 4K r---- liburiparser.so.1.0.20
00007f3e6def3000 4K rw--- liburiparser.so.1.0.20
00007f3e6def4000 40K r-x-- libminizip.so.1.0.0
00007f3e6defe000 2044K ----- libminizip.so.1.0.0
00007f3e6e0fd000 4K r---- libminizip.so.1.0.0
00007f3e6e0fe000 4K rw--- libminizip.so.1.0.0
00007f3e6e0ff000 36K r-x-- libltdl.so.7.3.1
00007f3e6e108000 2044K ----- libltdl.so.7.3.1
00007f3e6e307000 4K r---- libltdl.so.7.3.1
00007f3e6e308000 4K rw--- libltdl.so.7.3.1
00007f3e6e309000 1628K r-x-- libgeos-3.7.1.so
00007f3e6e4a0000 2048K ----- libgeos-3.7.1.so
00007f3e6e6a0000 44K r---- libgeos-3.7.1.so
00007f3e6e6ab000 4K rw--- libgeos-3.7.1.so
00007f3e6e6ac000 328K r-x-- liblcms2.so.2.0.8
00007f3e6e6fe000 2048K ----- liblcms2.so.2.0.8
00007f3e6e8fe000 8K r---- liblcms2.so.2.0.8
00007f3e6e900000 12K rw--- liblcms2.so.2.0.8
00007f3e6e903000 4K rw--- [ anon ]
00007f3e6e904000 228K r-x-- libnspr4.so
00007f3e6e93d000 2044K ----- libnspr4.so
00007f3e6eb3c000 4K r---- libnspr4.so
00007f3e6eb3d000 4K rw--- libnspr4.so
00007f3e6eb3e000 12K rw--- [ anon ]
00007f3e6eb41000 164K r-x-- libsmime3.so
00007f3e6eb6a000 2044K ----- libsmime3.so
00007f3e6ed69000 12K r---- libsmime3.so
00007f3e6ed6c000 4K rw--- libsmime3.so
00007f3e6ed6d000 1264K r-x-- libnss3.so
00007f3e6eea9000 2048K ----- libnss3.so
00007f3e6f0a9000 20K r---- libnss3.so
00007f3e6f0ae000 8K rw--- libnss3.so
00007f3e6f0b0000 4K rw--- [ anon ]
00007f3e6f0b1000 444K r-x-- libsuperlu.so.5.2.1
00007f3e6f120000 2044K ----- libsuperlu.so.5.2.1
00007f3e6f31f000 4K r---- libsuperlu.so.5.2.1
00007f3e6f320000 4K rw--- libsuperlu.so.5.2.1
00007f3e6f321000 292K r-x-- libarpack.so.2.0.0
00007f3e6f36a000 2044K ----- libarpack.so.2.0.0
00007f3e6f569000 4K r---- libarpack.so.2.0.0
00007f3e6f56a000 4K rw--- libarpack.so.2.0.0
00007f3e6f56b000 6752K r-x-- liblapack.so.3.7.1
00007f3e6fc03000 2048K ----- liblapack.so.3.7.1
00007f3e6fe03000 4K r---- liblapack.so.3.7.1
00007f3e6fe04000 16K rw--- liblapack.so.3.7.1
00007f3e6fe08000 4K rw--- [ anon ]
00007f3e6fe09000 428K r-x-- libblas.so.3.7.1
00007f3e6fe74000 2048K ----- libblas.so.3.7.1
00007f3e70074000 4K r---- libblas.so.3.7.1
00007f3e70075000 4K rw--- libblas.so.3.7.1
00007f3e70076000 3312K r-x-- libmysqlclient.so.20.3.15
00007f3e703b2000 2048K ----- libmysqlclient.so.20.3.15
00007f3e705b2000 16K r---- libmysqlclient.so.20.3.15
00007f3e705b6000 456K rw--- libmysqlclient.so.20.3.15
00007f3e70628000 24K rw--- [ anon ]
00007f3e7062e000 212K r-x-- libfyba.so.0.0.0
00007f3e70663000 2044K ----- libfyba.so.0.0.0
00007f3e70862000 4K r---- libfyba.so.0.0.0
00007f3e70863000 4K rw--- libfyba.so.0.0.0
00007f3e70864000 124K rw--- [ anon ]
00007f3e70883000 5624K r-x-- libspatialite.so.7.1.0
00007f3e70e01000 2044K ----- libspatialite.so.7.1.0
00007f3e71000000 4K r---- libspatialite.so.7.1.0
00007f3e71001000 8K rw--- libspatialite.so.7.1.0
00007f3e71003000 8K rw--- [ anon ]
00007f3e71005000 1560K r-x-- libdap.so.25.0.1
00007f3e7118b000 2048K ----- libdap.so.25.0.1
00007f3e7138b000 32K r---- libdap.so.25.0.1
00007f3e71393000 4K rw--- libdap.so.25.0.1
00007f3e71394000 4K rw--- [ anon ]
00007f3e71395000 252K r-x-- libdapclient.so.6.1.7
00007f3e713d4000 2044K ----- libdapclient.so.6.1.7
00007f3e715d3000 8K r---- libdapclient.so.6.1.7
00007f3e715d5000 4K rw--- libdapclient.so.6.1.7
00007f3e715d6000 296K r-x-- libpq.so.5.12
00007f3e71620000 2048K ----- libpq.so.5.12
00007f3e71820000 12K r---- libpq.so.5.12
00007f3e71823000 4K rw--- libpq.so.5.12
00007f3e71824000 48K rw--- [ anon ]
00007f3e71830000 180K r-x-- libgeotiff.so.2.1.2
00007f3e7185d000 2044K ----- libgeotiff.so.2.1.2
00007f3e71a5c000 36K r---- libgeotiff.so.2.1.2
00007f3e71a65000 4K rw--- libgeotiff.so.2.1.2
00007f3e71a66000 32K r-x-- libgif.so.7.0.0
00007f3e71a6e000 2044K ----- libgif.so.7.0.0
00007f3e71c6d000 4K r---- libgif.so.7.0.0
00007f3e71c6e000 4K rw--- libgif.so.7.0.0
00007f3e71c6f000 124K r-x-- libogdi.so.3.2
00007f3e71c8e000 2044K ----- libogdi.so.3.2
00007f3e71e8d000 4K r---- libogdi.so.3.2
00007f3e71e8e000 4K rw--- libogdi.so.3.2
00007f3e71e8f000 480K r-x-- libdfalt.so.0.0.0
00007f3e71f07000 2044K ----- libdfalt.so.0.0.0
00007f3e72106000 8K r---- libdfalt.so.0.0.0
00007f3e72108000 4K rw--- libdfalt.so.0.0.0
00007f3e72109000 164K rw--- [ anon ]
00007f3e72132000 144K r-x-- libmfhdfalt.so.0.0.0
00007f3e72156000 2044K ----- libmfhdfalt.so.0.0.0
00007f3e72355000 4K r---- libmfhdfalt.so.0.0.0
00007f3e72356000 4K rw--- libmfhdfalt.so.0.0.0
00007f3e72357000 4K rw--- [ anon ]
00007f3e72358000 3360K r-x-- libhdf5_serial.so.100.0.1
00007f3e726a0000 2048K ----- libhdf5_serial.so.100.0.1
00007f3e728a0000 32K r---- libhdf5_serial.so.100.0.1
00007f3e728a8000 20K rw--- libhdf5_serial.so.100.0.1
00007f3e728ad000 12K rw--- [ anon ]
00007f3e728b0000 1104K r-x-- libnetcdf.so.13
00007f3e729c4000 2044K ----- libnetcdf.so.13
00007f3e72bc3000 8K r---- libnetcdf.so.13
00007f3e72bc5000 12K rw--- libnetcdf.so.13
00007f3e72bc8000 40K rw--- [ anon ]
00007f3e72bd2000 336K r-x-- libopenjp2.so.2.3.0
00007f3e72c26000 2048K ----- libopenjp2.so.2.3.0
00007f3e72e26000 4K r---- libopenjp2.so.2.3.0
00007f3e72e27000 4K rw--- libopenjp2.so.2.3.0
00007f3e72e28000 3324K r-x-- libxerces-c-3.2.so
00007f3e73167000 2048K ----- libxerces-c-3.2.so
00007f3e73367000 76K r---- libxerces-c-3.2.so
00007f3e7337a000 136K rw--- libxerces-c-3.2.so
00007f3e7339c000 216K r-x-- libkmlengine.so.1.3.0
00007f3e733d2000 2048K ----- libkmlengine.so.1.3.0
00007f3e735d2000 4K r---- libkmlengine.so.1.3.0
00007f3e735d3000 4K rw--- libkmlengine.so.1.3.0
00007f3e735d4000 700K r-x-- libkmldom.so.1.3.0
00007f3e73683000 2048K ----- libkmldom.so.1.3.0
00007f3e73883000 24K r---- libkmldom.so.1.3.0
00007f3e73889000 8K rw--- libkmldom.so.1.3.0
00007f3e7388b000 100K r-x-- libkmlbase.so.1.3.0
00007f3e738a4000 2048K ----- libkmlbase.so.1.3.0
00007f3e73aa4000 4K r---- libkmlbase.so.1.3.0
00007f3e73aa5000 4K rw--- libkmlbase.so.1.3.0
00007f3e73aa6000 68K r-x-- libodbcinst.so.2.0.0
00007f3e73ab7000 2044K ----- libodbcinst.so.2.0.0
00007f3e73cb6000 4K r---- libodbcinst.so.2.0.0
00007f3e73cb7000 4K rw--- libodbcinst.so.2.0.0
00007f3e73cb8000 12K rw--- [ anon ]
00007f3e73cbb000 392K r-x-- libodbc.so.2.0.0
00007f3e73d1d000 2044K ----- libodbc.so.2.0.0
00007f3e73f1c000 4K r---- libodbc.so.2.0.0
00007f3e73f1d000 28K rw--- libodbc.so.2.0.0
00007f3e73f24000 16K rw--- [ anon ]
00007f3e73f28000 76K r-x-- libepsilon.so.1.0.0
00007f3e73f3b000 2044K ----- libepsilon.so.1.0.0
00007f3e7413a000 4K r---- libepsilon.so.1.0.0
00007f3e7413b000 20K rw--- libepsilon.so.1.0.0
00007f3e74140000 192K r-x-- libgeos_c.so.1.11.1
00007f3e74170000 2044K ----- libgeos_c.so.1.11.1
00007f3e7436f000 4K r---- libgeos_c.so.1.11.1
00007f3e74370000 4K rw--- libgeos_c.so.1.11.1
00007f3e74371000 352K r-x-- libqhull.so.7.2.0
00007f3e743c9000 2048K ----- libqhull.so.7.2.0
00007f3e745c9000 4K r---- libqhull.so.7.2.0
00007f3e745ca000 4K rw--- libqhull.so.7.2.0
00007f3e745cb000 32K r-x-- libfreexl.so.1.1.0
00007f3e745d3000 2044K ----- libfreexl.so.1.1.0
00007f3e747d2000 4K r---- libfreexl.so.1.1.0
00007f3e747d3000 4K rw--- libfreexl.so.1.1.0
00007f3e747d4000 40K r-x-- libjson-c.so.3.0.1
00007f3e747de000 2044K ----- libjson-c.so.3.0.1
00007f3e749dd000 4K r---- libjson-c.so.3.0.1
00007f3e749de000 4K rw--- libjson-c.so.3.0.1
00007f3e749df000 2376K r-x-- libpoppler.so.73.0.0
00007f3e74c31000 2048K ----- libpoppler.so.73.0.0
00007f3e74e31000 120K r---- libpoppler.so.73.0.0
00007f3e74e4f000 152K rw--- libpoppler.so.73.0.0
00007f3e74e75000 32K r-x-- libarmadillo.so.8.400.0
00007f3e74e7d000 2044K ----- libarmadillo.so.8.400.0
00007f3e7507c000 4K r---- libarmadillo.so.8.400.0
00007f3e7507d000 4K rw--- libarmadillo.so.8.400.0
00007f3e7507e000 15356K r-x-- libgdal.so.20.3.2
00007f3e75f7d000 2048K ----- libgdal.so.20.3.2
00007f3e7617d000 812K r---- libgdal.so.20.3.2
00007f3e76248000 4K rw--- libgdal.so.20.3.2
00007f3e76249000 268K rw--- [ anon ]
00007f3e7628c000 60K r-x-- gdal.input
00007f3e7629b000 2048K ----- gdal.input
00007f3e7649b000 4K r---- gdal.input
00007f3e7649c000 4K rw--- gdal.input
00007f3e7649d000 316K r-x-- topojson.input
00007f3e764ec000 2044K ----- topojson.input
00007f3e766eb000 8K r---- topojson.input
00007f3e766ed000 4K rw--- topojson.input
00007f3e766ee000 120K r-x-- raster.input
00007f3e7670c000 2048K ----- raster.input
00007f3e7690c000 8K r---- raster.input
00007f3e7690e000 4K rw--- raster.input
00007f3e7690f000 36K r-x-- libcrypt-2.27.so
00007f3e76918000 2044K ----- libcrypt-2.27.so
00007f3e76b17000 4K r---- libcrypt-2.27.so
00007f3e76b18000 4K rw--- libcrypt-2.27.so
00007f3e76b19000 184K rw--- [ anon ]
00007f3e76b47000 1040K r-x-- libsqlite3.so.0.8.6
00007f3e76c4b000 2044K ----- libsqlite3.so.0.8.6
00007f3e76e4a000 12K r---- libsqlite3.so.0.8.6
00007f3e76e4d000 8K rw--- libsqlite3.so.0.8.6
00007f3e76e4f000 4K rw--- [ anon ]
00007f3e76e50000 280K r-x-- libhx509.so.5.0.0
00007f3e76e96000 2044K ----- libhx509.so.5.0.0
00007f3e77095000 12K r---- libhx509.so.5.0.0
00007f3e77098000 4K rw--- libhx509.so.5.0.0
00007f3e77099000 4K rw--- [ anon ]
00007f3e7709a000 56K r-x-- libheimbase.so.1.0.0
00007f3e770a8000 2044K ----- libheimbase.so.1.0.0
00007f3e772a7000 4K r---- libheimbase.so.1.0.0
00007f3e772a8000 4K rw--- libheimbase.so.1.0.0
00007f3e772a9000 160K r-x-- libwind.so.0.0.0
00007f3e772d1000 2044K ----- libwind.so.0.0.0
00007f3e774d0000 4K r---- libwind.so.0.0.0
00007f3e774d1000 4K rw--- libwind.so.0.0.0
00007f3e774d2000 76K r-x-- libbsd.so.0.8.7
00007f3e774e5000 2044K ----- libbsd.so.0.8.7
00007f3e776e4000 4K r---- libbsd.so.0.8.7
00007f3e776e5000 4K rw--- libbsd.so.0.8.7
00007f3e776e6000 4K rw--- [ anon ]
00007f3e776e7000 84K r-x-- libroken.so.18.1.0
00007f3e776fc000 2044K ----- libroken.so.18.1.0
00007f3e778fb000 4K r---- libroken.so.18.1.0
00007f3e778fc000 4K rw--- libroken.so.18.1.0
00007f3e778fd000 204K r-x-- libhcrypto.so.4.1.0
00007f3e77930000 2044K ----- libhcrypto.so.4.1.0
00007f3e77b2f000 8K r---- libhcrypto.so.4.1.0
00007f3e77b31000 4K rw--- libhcrypto.so.4.1.0
00007f3e77b32000 4K rw--- [ anon ]
00007f3e77b33000 632K r-x-- libasn1.so.8.0.0
00007f3e77bd1000 2048K ----- libasn1.so.8.0.0
00007f3e77dd1000 4K r---- libasn1.so.8.0.0
00007f3e77dd2000 12K rw--- libasn1.so.8.0.0
00007f3e77dd5000 540K r-x-- libkrb5.so.26.0.0
00007f3e77e5c000 2044K ----- libkrb5.so.26.0.0
00007f3e7805b000 16K r---- libkrb5.so.26.0.0
00007f3e7805f000 8K rw--- libkrb5.so.26.0.0
00007f3e78061000 4K rw--- [ anon ]
00007f3e78062000 32K r-x-- libheimntlm.so.0.1.0
00007f3e7806a000 2044K ----- libheimntlm.so.0.1.0
00007f3e78269000 4K r---- libheimntlm.so.0.1.0
00007f3e7826a000 4K rw--- libheimntlm.so.0.1.0
00007f3e7826b000 12K r-x-- libkeyutils.so.1.5
00007f3e7826e000 2044K ----- libkeyutils.so.1.5
00007f3e7846d000 4K r---- libkeyutils.so.1.5
00007f3e7846e000 4K rw--- libkeyutils.so.1.5
00007f3e7846f000 28K r-x-- libffi.so.6.0.4
00007f3e78476000 2044K ----- libffi.so.6.0.4
00007f3e78675000 4K r---- libffi.so.6.0.4
00007f3e78676000 4K rw--- libffi.so.6.0.4
00007f3e78677000 448K r-x-- libpcre.so.3.13.3
00007f3e786e7000 2048K ----- libpcre.so.3.13.3
00007f3e788e7000 4K r---- libpcre.so.3.13.3
00007f3e788e8000 4K rw--- libpcre.so.3.13.3
00007f3e788e9000 20K r-x-- libXdmcp.so.6.0.0
00007f3e788ee000 2044K ----- libXdmcp.so.6.0.0
00007f3e78aed000 4K r---- libXdmcp.so.6.0.0
00007f3e78aee000 4K rw--- libXdmcp.so.6.0.0
00007f3e78aef000 8K r-x-- libXau.so.6.0.0
00007f3e78af1000 2048K ----- libXau.so.6.0.0
00007f3e78cf1000 4K r---- libXau.so.6.0.0
00007f3e78cf2000 4K rw--- libXau.so.6.0.0
00007f3e78cf3000 188K r-x-- libexpat.so.1.6.7
00007f3e78d22000 2048K ----- libexpat.so.1.6.7
00007f3e78f22000 8K r---- libexpat.so.1.6.7
00007f3e78f24000 4K rw--- libexpat.so.1.6.7
00007f3e78f25000 244K r-x-- libgssapi.so.3.0.0
00007f3e78f62000 2048K ----- libgssapi.so.3.0.0
00007f3e79162000 8K r---- libgssapi.so.3.0.0
00007f3e79164000 8K rw--- libgssapi.so.3.0.0
00007f3e79166000 100K r-x-- libsasl2.so.2.0.25
00007f3e7917f000 2048K ----- libsasl2.so.2.0.25
00007f3e7937f000 4K r---- libsasl2.so.2.0.25
00007f3e79380000 4K rw--- libsasl2.so.2.0.25
00007f3e79381000 92K r-x-- libresolv-2.27.so
00007f3e79398000 2048K ----- libresolv-2.27.so
00007f3e79598000 4K r---- libresolv-2.27.so
00007f3e79599000 4K rw--- libresolv-2.27.so
00007f3e7959a000 8K rw--- [ anon ]
00007f3e7959c000 40K r-x-- libkrb5support.so.0.1
00007f3e795a6000 2044K ----- libkrb5support.so.0.1
00007f3e797a5000 4K r---- libkrb5support.so.0.1
00007f3e797a6000 4K rw--- libkrb5support.so.0.1
00007f3e797a7000 12K r-x-- libcom_err.so.2.1
00007f3e797aa000 2044K ----- libcom_err.so.2.1
00007f3e799a9000 4K r---- libcom_err.so.2.1
00007f3e799aa000 4K rw--- libcom_err.so.2.1
00007f3e799ab000 184K r-x-- libk5crypto.so.3.1
00007f3e799d9000 2048K ----- libk5crypto.so.3.1
00007f3e79bd9000 8K r---- libk5crypto.so.3.1
00007f3e79bdb000 4K rw--- libk5crypto.so.3.1
00007f3e79bdc000 4K rw--- [ anon ]
00007f3e79bdd000 792K r-x-- libkrb5.so.3.3
00007f3e79ca3000 2048K ----- libkrb5.so.3.3
00007f3e79ea3000 56K r---- libkrb5.so.3.3
00007f3e79eb1000 8K rw--- libkrb5.so.3.3
00007f3e79eb3000 68K r-x-- libtasn1.so.6.5.5
00007f3e79ec4000 2048K ----- libtasn1.so.6.5.5
00007f3e7a0c4000 4K r---- libtasn1.so.6.5.5
00007f3e7a0c5000 4K rw--- libtasn1.so.6.5.5
00007f3e7a0c6000 1128K r-x-- libp11-kit.so.0.3.0
00007f3e7a1e0000 2048K ----- libp11-kit.so.0.3.0
00007f3e7a3e0000 40K r---- libp11-kit.so.0.3.0
00007f3e7a3ea000 40K rw--- libp11-kit.so.0.3.0
00007f3e7a3f4000 4K rw--- [ anon ]
00007f3e7a3f5000 508K r-x-- libgmp.so.10.3.2
00007f3e7a474000 2048K ----- libgmp.so.10.3.2
00007f3e7a674000 4K r---- libgmp.so.10.3.2
00007f3e7a675000 4K rw--- libgmp.so.10.3.2
00007f3e7a676000 204K r-x-- libhogweed.so.4.4
00007f3e7a6a9000 2044K ----- libhogweed.so.4.4
00007f3e7a8a8000 4K r---- libhogweed.so.4.4
00007f3e7a8a9000 4K rw--- libhogweed.so.4.4
00007f3e7a8aa000 1512K r-x-- libunistring.so.2.1.0
00007f3e7aa24000 2048K ----- libunistring.so.2.1.0
00007f3e7ac24000 12K r---- libunistring.so.2.1.0
00007f3e7ac27000 4K rw--- libunistring.so.2.1.0
00007f3e7ac28000 26272K r-x-- libicudata.so.60.2
00007f3e7c5d0000 2044K ----- libicudata.so.60.2
00007f3e7c7cf000 4K r---- libicudata.so.60.2
00007f3e7c7d0000 4K rw--- libicudata.so.60.2
00007f3e7c7d1000 172K r-x-- libgraphite2.so.3.0.1
00007f3e7c7fc000 2044K ----- libgraphite2.so.3.0.1
00007f3e7c9fb000 8K r---- libgraphite2.so.3.0.1
00007f3e7c9fd000 4K rw--- libgraphite2.so.3.0.1
00007f3e7c9fe000 1104K r-x-- libglib-2.0.so.0.5600.4
00007f3e7cb12000 2048K ----- libglib-2.0.so.0.5600.4
00007f3e7cd12000 4K r---- libglib-2.0.so.0.5600.4
00007f3e7cd13000 4K rw--- libglib-2.0.so.0.5600.4
00007f3e7cd14000 4K rw--- [ anon ]
00007f3e7cd15000 44K r-x-- libjbig.so.0
00007f3e7cd20000 2044K ----- libjbig.so.0
00007f3e7cf1f000 4K r---- libjbig.so.0
00007f3e7cf20000 12K rw--- libjbig.so.0
00007f3e7cf23000 144K r-x-- liblzma.so.5.2.2
00007f3e7cf47000 2048K ----- liblzma.so.5.2.2
00007f3e7d147000 4K r---- liblzma.so.5.2.2
00007f3e7d148000 4K rw--- liblzma.so.5.2.2
00007f3e7d149000 28K r-x-- librt-2.27.so
00007f3e7d150000 2044K ----- librt-2.27.so
00007f3e7d34f000 4K r---- librt-2.27.so
00007f3e7d350000 4K rw--- librt-2.27.so
00007f3e7d351000 68K r-x-- libXext.so.6.4.0
00007f3e7d362000 2044K ----- libXext.so.6.4.0
00007f3e7d561000 4K r---- libXext.so.6.4.0
00007f3e7d562000 4K rw--- libXext.so.6.4.0
00007f3e7d563000 1228K r-x-- libX11.so.6.3.0
00007f3e7d696000 2048K ----- libX11.so.6.3.0
00007f3e7d896000 4K r---- libX11.so.6.3.0
00007f3e7d897000 16K rw--- libX11.so.6.3.0
00007f3e7d89b000 36K r-x-- libXrender.so.1.3.0
00007f3e7d8a4000 2044K ----- libXrender.so.1.3.0
00007f3e7daa3000 4K r---- libXrender.so.1.3.0
00007f3e7daa4000 4K rw--- libXrender.so.1.3.0
00007f3e7daa5000 48K r-x-- libxcb-render.so.0.0.0
00007f3e7dab1000 2044K ----- libxcb-render.so.0.0.0
00007f3e7dcb0000 4K r---- libxcb-render.so.0.0.0
00007f3e7dcb1000 4K rw--- libxcb-render.so.0.0.0
00007f3e7dcb2000 152K r-x-- libxcb.so.1.1.0
00007f3e7dcd8000 2048K ----- libxcb.so.1.1.0
00007f3e7ded8000 4K r---- libxcb.so.1.1.0
00007f3e7ded9000 4K rw--- libxcb.so.1.1.0
00007f3e7deda000 8K r-x-- libxcb-shm.so.0.0.0
00007f3e7dedc000 2044K ----- libxcb-shm.so.0.0.0
00007f3e7e0db000 4K r---- libxcb-shm.so.0.0.0
00007f3e7e0dc000 4K rw--- libxcb-shm.so.0.0.0
00007f3e7e0dd000 248K r-x-- libfontconfig.so.1.10.1
00007f3e7e11b000 2048K ----- libfontconfig.so.1.10.1
00007f3e7e31b000 8K r---- libfontconfig.so.1.10.1
00007f3e7e31d000 20K rw--- libfontconfig.so.1.10.1
00007f3e7e322000 628K r-x-- libpixman-1.so.0.34.0
00007f3e7e3bf000 2044K ----- libpixman-1.so.0.34.0
00007f3e7e5be000 32K r---- libpixman-1.so.0.34.0
00007f3e7e5c6000 4K rw--- libpixman-1.so.0.34.0
00007f3e7e5c7000 2632K r-x-- libicui18n.so.60.2
00007f3e7e859000 2044K ----- libicui18n.so.60.2
00007f3e7ea58000 60K r---- libicui18n.so.60.2
00007f3e7ea67000 4K rw--- libicui18n.so.60.2
00007f3e7ea68000 112K r-x-- libz.so.1.2.11
00007f3e7ea84000 2044K ----- libz.so.1.2.11
00007f3e7ec83000 4K r---- libz.so.1.2.11
00007f3e7ec84000 4K rw--- libz.so.1.2.11
00007f3e7ec85000 52K r-x-- liblber-2.4.so.2.10.8
00007f3e7ec92000 2044K ----- liblber-2.4.so.2.10.8
00007f3e7ee91000 4K r---- liblber-2.4.so.2.10.8
00007f3e7ee92000 4K rw--- liblber-2.4.so.2.10.8
00007f3e7ee93000 312K r-x-- libldap_r-2.4.so.2.10.8
00007f3e7eee1000 2044K ----- libldap_r-2.4.so.2.10.8
00007f3e7f0e0000 8K r---- libldap_r-2.4.so.2.10.8
00007f3e7f0e2000 4K rw--- libldap_r-2.4.so.2.10.8
00007f3e7f0e3000 8K rw--- [ anon ]
00007f3e7f0e5000 288K r-x-- libgssapi_krb5.so.2.2
00007f3e7f12d000 2044K ----- libgssapi_krb5.so.2.2
00007f3e7f32c000 8K r---- libgssapi_krb5.so.2.2
00007f3e7f32e000 8K rw--- libgssapi_krb5.so.2.2
00007f3e7f330000 1376K r-x-- libgnutls.so.30.14.10
00007f3e7f488000 2044K ----- libgnutls.so.30.14.10
00007f3e7f687000 48K r---- libgnutls.so.30.14.10
00007f3e7f693000 4K rw--- libgnutls.so.30.14.10
00007f3e7f694000 4K rw--- [ anon ]
00007f3e7f695000 208K r-x-- libnettle.so.6.4
00007f3e7f6c9000 2044K ----- libnettle.so.6.4
00007f3e7f8c8000 8K r---- libnettle.so.6.4
00007f3e7f8ca000 4K rw--- libnettle.so.6.4
00007f3e7f8cb000 52K r-x-- libpsl.so.5.2.0
00007f3e7f8d8000 2044K ----- libpsl.so.5.2.0
00007f3e7fad7000 4K r---- libpsl.so.5.2.0
00007f3e7fad8000 4K rw--- libpsl.so.5.2.0
00007f3e7fad9000 108K r-x-- librtmp.so.1
00007f3e7faf4000 2044K ----- librtmp.so.1
00007f3e7fcf3000 4K r---- librtmp.so.1
00007f3e7fcf4000 4K rw--- librtmp.so.1
00007f3e7fcf5000 112K r-x-- libidn2.so.0.3.3
00007f3e7fd11000 2044K ----- libidn2.so.0.3.3
00007f3e7ff10000 4K r---- libidn2.so.0.3.3
00007f3e7ff11000 4K rw--- libidn2.so.0.3.3
00007f3e7ff12000 140K r-x-- libnghttp2.so.14.15.2
00007f3e7ff35000 2044K ----- libnghttp2.so.14.15.2
00007f3e80134000 4K r---- libnghttp2.so.14.15.2
00007f3e80135000 8K rw--- libnghttp2.so.14.15.2
00007f3e80137000 1652K r-x-- libm-2.27.so
00007f3e802d4000 2044K ----- libm-2.27.so
00007f3e804d3000 4K r---- libm-2.27.so
00007f3e804d4000 4K rw--- libm-2.27.so
00007f3e804d5000 12K r-x-- libdl-2.27.so
00007f3e804d8000 2044K ----- libdl-2.27.so
00007f3e806d7000 4K r---- libdl-2.27.so
00007f3e806d8000 4K rw--- libdl-2.27.so
00007f3e806d9000 692K r-x-- libfreetype.so.6.15.0
00007f3e80786000 2044K ----- libfreetype.so.6.15.0
00007f3e80985000 28K r---- libfreetype.so.6.15.0
00007f3e8098c000 4K rw--- libfreetype.so.6.15.0
00007f3e8098d000 1676K r-x-- libicuuc.so.60.2
00007f3e80b30000 2044K ----- libicuuc.so.60.2
00007f3e80d2f000 76K r---- libicuuc.so.60.2
00007f3e80d42000 4K rw--- libicuuc.so.60.2
00007f3e80d43000 4K rw--- [ anon ]
00007f3e80d44000 412K r-x-- libjpeg.so.8.1.2
00007f3e80dab000 2044K ----- libjpeg.so.8.1.2
00007f3e80faa000 4K r---- libjpeg.so.8.1.2
00007f3e80fab000 4K rw--- libjpeg.so.8.1.2
00007f3e80fac000 624K r-x-- libharfbuzz.so.0.10702.0
00007f3e81048000 2048K ----- libharfbuzz.so.0.10702.0
00007f3e81248000 4K r---- libharfbuzz.so.0.10702.0
00007f3e81249000 4K rw--- libharfbuzz.so.0.10702.0
00007f3e8124a000 16K r-x-- libboost_system.so.1.65.1
00007f3e8124e000 2044K ----- libboost_system.so.1.65.1
00007f3e8144d000 4K r---- libboost_system.so.1.65.1
00007f3e8144e000 4K rw--- libboost_system.so.1.65.1
00007f3e8144f000 1752K r-x-- libxml2.so.2.9.4
00007f3e81605000 2048K ----- libxml2.so.2.9.4
00007f3e81805000 32K r---- libxml2.so.2.9.4
00007f3e8180d000 8K rw--- libxml2.so.2.9.4
00007f3e8180f000 4K rw--- [ anon ]
00007f3e81810000 408K r-x-- libwebp.so.6.0.2
00007f3e81876000 2044K ----- libwebp.so.6.0.2
00007f3e81a75000 4K r---- libwebp.so.6.0.2
00007f3e81a76000 4K rw--- libwebp.so.6.0.2
00007f3e81a77000 8K rw--- [ anon ]
00007f3e81a79000 460K r-x-- libtiff.so.5.3.0
00007f3e81aec000 2044K ----- libtiff.so.5.3.0
00007f3e81ceb000 16K r---- libtiff.so.5.3.0
00007f3e81cef000 4K rw--- libtiff.so.5.3.0
00007f3e81cf0000 404K r-x-- libproj.so.12.0.0
00007f3e81d55000 2048K ----- libproj.so.12.0.0
00007f3e81f55000 4K r---- libproj.so.12.0.0
00007f3e81f56000 12K rw--- libproj.so.12.0.0
00007f3e81f59000 196K r-x-- libpng16.so.16.34.0
00007f3e81f8a000 2044K ----- libpng16.so.16.34.0
00007f3e82189000 4K r---- libpng16.so.16.34.0
00007f3e8218a000 4K rw--- libpng16.so.16.34.0
00007f3e8218b000 1120K r-x-- libcairo.so.2.11510.0
00007f3e822a3000 2044K ----- libcairo.so.2.11510.0
00007f3e824a2000 12K r---- libcairo.so.2.11510.0
00007f3e824a5000 8K rw--- libcairo.so.2.11510.0
00007f3e824a7000 4K rw--- [ anon ]
00007f3e824a8000 1036K r-x-- libboost_regex.so.1.65.1
00007f3e825ab000 2044K ----- libboost_regex.so.1.65.1
00007f3e827aa000 20K r---- libboost_regex.so.1.65.1
00007f3e827af000 4K rw--- libboost_regex.so.1.65.1
00007f3e827b0000 100K r-x-- libboost_filesystem.so.1.65.1
00007f3e827c9000 2044K ----- libboost_filesystem.so.1.65.1
00007f3e829c8000 4K r---- libboost_filesystem.so.1.65.1
00007f3e829c9000 4K rw--- libboost_filesystem.so.1.65.1
00007f3e829ca000 1948K r-x-- libc-2.27.so
00007f3e82bb1000 2048K ----- libc-2.27.so
00007f3e82db1000 16K r---- libc-2.27.so
00007f3e82db5000 8K rw--- libc-2.27.so
00007f3e82db7000 16K rw--- [ anon ]
00007f3e82dbb000 104K r-x-- libpthread-2.27.so
00007f3e82dd5000 2044K ----- libpthread-2.27.so
00007f3e82fd4000 4K r---- libpthread-2.27.so
00007f3e82fd5000 4K rw--- libpthread-2.27.so
00007f3e82fd6000 16K rw--- [ anon ]
00007f3e82fda000 92K r-x-- libgcc_s.so.1
00007f3e82ff1000 2044K ----- libgcc_s.so.1
00007f3e831f0000 4K r---- libgcc_s.so.1
00007f3e831f1000 4K rw--- libgcc_s.so.1
00007f3e831f2000 1508K r-x-- libstdc++.so.6.0.25
00007f3e8336b000 2048K ----- libstdc++.so.6.0.25
00007f3e8356b000 40K r---- libstdc++.so.6.0.25
00007f3e83575000 8K rw--- libstdc++.so.6.0.25
00007f3e83577000 16K rw--- [ anon ]
00007f3e8357b000 12K r-x-- libiniparser.so.3.0.0
00007f3e8357e000 2044K ----- libiniparser.so.3.0.0
00007f3e8377d000 4K r---- libiniparser.so.3.0.0
00007f3e8377e000 4K rw--- libiniparser.so.3.0.0
00007f3e8377f000 488K r-x-- libcurl-gnutls.so.4.5.0
00007f3e837f9000 2044K ----- libcurl-gnutls.so.4.5.0
00007f3e839f8000 12K r---- libcurl-gnutls.so.4.5.0
00007f3e839fb000 4K rw--- libcurl-gnutls.so.4.5.0
00007f3e839fc000 10460K r-x-- libmapnik.so.3.0.19
00007f3e84433000 2048K ----- libmapnik.so.3.0.19
00007f3e84633000 32K r---- libmapnik.so.3.0.19
00007f3e8463b000 8K rw--- libmapnik.so.3.0.19
00007f3e8463d000 28K rw--- [ anon ]
00007f3e84644000 156K r-x-- ld-2.27.so
00007f3e8468c000 56K r--s- ne_110m_admin_0_boundary_lines_land.shp
00007f3e8469a000 172K r--s- water_polygons.dbf
00007f3e846c5000 116K r--s- water_polygons.shx
00007f3e846e2000 488K r--s- simplified_water_polygons.index
00007f3e8475c000 168K r--s- simplified_water_polygons.dbf
00007f3e84786000 112K r--s- simplified_water_polygons.shx
00007f3e847a2000 244K r--s- icesheet_polygons.index
00007f3e847df000 200K r--s- icesheet_polygons.dbf
00007f3e84811000 132K r--s- icesheet_polygons.shx
00007f3e84832000 180K rw--- [ anon ]
00007f3e8485f000 12K r--s- ne_110m_admin_0_boundary_lines_land.index
00007f3e84862000 32K r--s- ne_110m_admin_0_boundary_lines_land.dbf
00007f3e8486a000 4K r--s- ne_110m_admin_0_boundary_lines_land.shx
00007f3e8486b000 4K r---- ld-2.27.so
00007f3e8486c000 4K rw--- ld-2.27.so
00007f3e8486d000 4K rw--- [ anon ]
00007ffc5cd10000 284K rw--- [ stack ]
00007ffc5cd8c000 12K r---- [ anon ]
00007ffc5cd8f000 8K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
gesamt 15926748K
We also encounter this problem.
It seems we could mitigate this by adding memory limits in systemd's service definition:
[Unit]
Description=Renderd
After=network.target auditd.service
[Service]
User=gis
Group=gis
Type=simple
RuntimeDirectory=./renderd
RuntimeDirectoryMode=0755
Restart=always
ExecStart=/usr/local/bin/renderd --foreground -c /etc/renderd.conf
# here is the trick. It must be adapted to the available RAM
MemoryHigh=50G
MemoryMax=60G
While the level of memory approaches, systemd seems to get memory back from render daemon.
renderd does not crash, and we do not have oom killer issues. But when rendering expired tiles, it may still happens that the rendering is stopped early because systemd has destroyed some memory.
Adding a memory limit will not reduce the memory consumption. What might happen is that the increased memory pressure on this cgroup is leading to pages being dropped increasing the available memory. Either file-backed pages are dropped or anonymous memory relocated to the swap device.
You could try trading CPU for memory by using zram as a target for swapping. Hopefully most of the allocated memory is not part of the active working set, so the extra cost for swapping to RAM could be tolerabe.
Thanks @stephankn . I am going to investigate this.
Hi, is there any progress? I am experiencing unlimited memory consumption already for some years. Here is am image showing current total memory comsumption of my machine (orange): The footprint is mainly due to the renderd service. The two jumps upwards are caused by map serve/render request threads. The jump down caused by oom killing renderd. The memory consumption is somehow very similar to the size of map tiles served. Are there tile caching mechanisms used? I found code using memcached but libmemcached is not installed on my machine. Are there other cache mechanisms?
Since either restarting postgresql or restarting renderd releases huge amount of memory, there must be an issue with the connection of both processes.
I found an interesting post regarding long running connections:
The most likely cause for this is that PostgreSQL keeps a per-connection cache of metadata about all the database objects (tables, indexes, etc.) it has touched during the connection lifetime. There is no upper limit on how big this cache can get, and no mechanism for expiring it. (https://dba.stackexchange.com/questions/160887/how-can-i-find-the-source-of-postgresql-per-connection-memory-leaks/222815#222815)
The usual solutions seems is to set a maximum life time for a single connection. Is there such an option in renderd?
I found an interesting post regarding long running connections:
The most likely cause for this is that PostgreSQL keeps a per-connection cache of metadata about all the database objects (tables, indexes, etc.) it has touched during the connection lifetime. There is no upper limit on how big this cache can get, and no mechanism for expiring it. (https://dba.stackexchange.com/questions/160887/how-can-i-find-the-source-of-postgresql-per-connection-memory-leaks/222815#222815)
The usual solutions seems is to set a maximum life time for a single connection. Is there such an option in renderd?
It is unlikely this is the problem, because most rendering databases contain a small number of large objects, not many small objects.
In my case there are four tables with an index for each of them, like osm2pgsql creates. A geometry is indexed using postgis plugin.
I did not dig into detail of postgis or postgresql caching mechanisms but I made a simple test using logged queries from tile rendering. I wrote a php script issuing this queries with the option of closing and reopening the connection in between.
During querying the database a postgresql process increased memory consumption. The memory consumtion of the php script did not change at all. After closing the connection the postgresql process vanished and all memory was freed.
I did not see any memory increase on client side which is in contradiction to the comments here and to what I experienced after starting the rendering session. But after some time of rendering, where low zoom levels with large query result sets are completely rendered, there seems to be low memory consumption increase on client (renderd) side but on server side.
For the record some rough data:
In my case restarting only postgres doesn't free up memory. Renderd still holds a lot of RAM (the more I render the more it takes with no apparent limit visible). Did anyone have any progress on this issue?
I'm facing similar issue while rendering tiles for zoom-level(Z)>14
render_list -n 64 -s /var/run/renderd/renderd.sock -z 15 -Z 15 -m ajt -a
I have allocated 180GB RAM for PostGIS but all of it gets consumed and the db server restarts.
I went inside the db shell and checked all the processes, not sure if this is normal :/
Maybe if you reduced the -n 64 to a (much?) smaller number? 64 renders in parallel seems like a large number to me, even with 180GB RAM.
Lynn (D) - Running my own planet-wide tile server on very modest hardware
On 1/29/2021 1:28 AM, suneet-nokia via Tile-serving wrote:
I'm facing similar issue while rendering tiles for zoom-level(Z)>14 |render_list -n 64 -s /var/run/renderd/renderd.sock -z 15 -Z 15 -m ajt -a| I have allocated 180GB RAM for PostGIS but all of it gets consumed and the db server restarts.
I went inside the db shell and checked all the processes, not sure if this is normal :/ image https://user-images.githubusercontent.com/71066412/106239642-f85d2580-6228-11eb-8905-e3490f13bef9.png
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openstreetmap/mod_tile/issues/181#issuecomment-769606982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6353SVJDIFX5JT3JFJ2GLS4JIP5ANCNFSM4FDRE4HQ.
Tile-serving mailing list Tile-serving@openstreetmap.org https://lists.openstreetmap.org/listinfo/tile-serving
Maybe if you reduced the -n 64 to a (much?) smaller number? 64 renders in parallel seems like a large number to me, even with 180GB RAM. Lynn (D) - Running my own planet-wide tile server on very modest hardware
Thanks Lynn for responding. I've tried with -n 16 and -n 8 as well. It's the same story. The beauty here is when the Postgres restarts, render_list continues to render from where it left off. (Thanks to the file /var/lib/mod_tile/planet-import-complete) @duerk-de @pnorman : Maybe you can help
Hello.
I'm having issues also. In my case renderd is eating all it cans until it crashes.
It's now running with following command: render_list -n 2 -s /var/run/renderd/renderd.sock -z 7 -Z 8 -m ajt -a
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 38773 maprend+ 20 0 22.337g 0.021t 3700 S 30.0 68.4 8:23.98 /usr/local/bin/renderd -c /usr/local/etc/renderd.conf 39413 postgres 20 0 9009004 8.340g 8.151g R 95.0 26.6 5:21.39 postgres: 10/main: maprender gis [local] SELECT 39412 postgres 20 0 8925180 8.241g 8.153g R 72.0 26.3 9:42.41 postgres: 10/main: maprender gis [local] SELECT
It's consuming already 22G RAM. I've 1G left only, sometimes goes to less. The system as 32Gram, the remaining is allowed for the database.
Would be good to have a way to limit the amount of memory it tries to use.
Let's see if I can manage to have this running with only 2 threads.
It crashed again.
After that I set renderd.conf to 4 threads (was 8) and kept the same 2 threads in render_list command. With this it's being stable with the renderd consuming around 13G ram, what is more or less half that it was with 8 threads in renderd.conf.
So it seems that it needs about 3.2G ram per thread. Is this what to be expected?
It's rendering now for almost 24h, going on zoom 11 already.
I think in this issue two different issues are mixed, both around memory not freed. The initial report was about the RSS of renderd growing.
I have with my recent setup (using tirex) a similar problem like described by @duerk-de. In that case the RssAnon of the postgres backends is growing. A single backend grow with a rendering queue fully busy during this time leads to a growth of roughly 3GB within 7 hours for each rendering instance:
RssAnon: from 352.532 to 3.343.852, increase by 2.991.320
resetting the database connection frees this memory. I also have the suspicion that this is related to mapnik using persistent connections to postgis. By looking at the source code I think this is used in the same way in renderd and tirex.
A workaround with tirex is to send SIGHUP to the rendering backend manager. I would feel better if the root cause is understood.
Postgresql developer documentation suggests dumping the allocation structure with gdb, but I fear I won't understand the output without deeply digging into Postgresql.
Examining backend memory use PostgreSQL's palloc is a hierarchical memory allocator that wraps the platform allocator. See #Why do we use palloc() and pfree() to allocate memory?.
Memory allocated with palloc is assigned to a memory context that's part of a hierarchy rooted at TopMemoryContext. Each context has a name.
You can dump stats about a memory context and its children using the MemoryContextStats(MemoryContext*) function. In the most common usage, that's:
gdb -p $the_backend_pid (gdb) p MemoryContextStats(TopMemoryContext) The output is written to stderr.
This may appear in the main server log file, a secondary log used by the init system for before PostgreSQL's logging collector starts, journald, or on your screen if you are running a backend directly without a postmaster.
I figured out the source for my memory leak in PostgreSQL. It is a problem with JIT leaking memory. Upstream in Bug 16707: https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org
So if you have the symptom, that RssAnon of your postgres backend is continuously increasing check whether tuning off JIT helps.
use ps to figure out one or mire PIDs of the backends in use by user osm:
# ps auxw | grep -E "postgres: osm"
postgres 1340 34.4 0.0 3884840 100620 ? Rs 07:01 12:22 postgres: osm gis 172.20.0.3(36596) SELECT
postgres 1341 31.8 0.0 3874076 95448 ? Rs 07:01 11:26 postgres: osm gis 172.20.0.3(36594) SELECT
postgres 1342 34.5 0.0 3875604 97188 ? Ss 07:01 12:24 postgres: osm gis 172.20.0.3(36598) idle
postgres 1343 35.3 0.0 3873320 94616 ? Rs 07:01 12:40 postgres: osm gis 172.20.0.3(36600) SELECT
postgres 1344 34.4 0.0 3876880 98176 ? Ss 07:01 12:21 postgres: osm gis 172.20.0.3(36602) idle
postgres 1391 32.4 0.0 3882716 98844 ? Rs 07:01 11:34 postgres: osm gis 172.20.0.3(36640) SELECT
postgres 1396 32.8 0.0 3874436 96100 ? Rs 07:01 11:42 postgres: osm gis 172.20.0.3(36642) SELECT
postgres 1444 34.1 0.0 3874480 96336 ? Ss 07:01 12:05 postgres: osm gis 172.20.0.3(36674) idle
Then check out RssAnon memory allocation:
# grep RssAnon /proc/1340/status
RssAnon: 48316 kB
This value should stay relatively constant during rendering activity. Mine did increase with a rate of roughly 250MB/hour.
If affected, check whether you have JIT enabled.
# echo "SHOW jit;" | psql -U postgres
jit
-----
on
(1 row)
If this shows on, you can likely fix the leak by turning it off. Either in the configuration file or by alter system.
# echo "ALTER SYSTEM SET jit=off; SELECT pg_reload_conf();" | psql -U postgres
According to upstream reports, the leak exists since PostgreSQL 12. So at least the leak within PostgreSQL mentioned in this bug report is addressed. The potential leak inside renderd might still be there.
If this shows on, you can likely fix the leak by turning it off. Either in the configuration file or by alter system.
I'd add that with most stylesheets you may want to turn JIT off anyways.
I know the thread is a few months old, but thought I'd add my findings:
I'm using the overv/openstreetmap-tile-server
Docker image (just like @msschl) (supposedly the result of the switch2osm guide) and found that I hit the aforementioned RAM issues - RAM climbs until it runs out, but never releases.
Before reading this thread (and similar threads, like openstreetmap-tile-server/issues/27) I was convinced renderd was the issue. However, after doing some further digging and reading @stephankn post I concluded that actually what I'm seeing is the same as him - the value for RssAnon always climbed along with the RAM usage.
Following reading up on it, I disabled the JIT feature as described. This has definitely resolved the issue - previously I could exhaust my RAM is <48 hours, whereas now it ticks over with an almost identical memory usage and RssAnon value 5 days later. Genuinely, many thanks for your write-up stephankn.
I just thought I'd mention my findings on this issue:
I ran a leak detection and found a tiny, insignificant leak during the configuration parsing. It's 32 bytes and happens once during process startup.
I switched out the memory allocator (from glib to jemalloc) and that shows promise...
Background: my setup has renderd
running on a 32GB node with the psqldb sitting on a different node. The database has only a recent North America extract loaded and renders OpenStreetMap-styled tiles with scale=2 (... so they look good on a retina display). Pre-rendering with 8 threads will have the renderd
processes /threads grow to 31.5GB as soon as zoom levels 7 or higher are reached.
Recompiling with jemalloc yields the following: 0.5GB - 4.7GB per process size at zoom levels 0..11. Right now they are hovering at 1.7GB and churning "ocean tiles" (so little to no features to render)...
My hypothesis is as follows: To render a "busy" tile (one that has a lot of detail at a medium/high zoom level), renderd.mapnik
requests a lot of data from the database. This data is stored in fine-grained allocations which lead to heavy fragmentation of the memory. Although all memory seems to be freed once a tile has been rendered (mapnik
seems to be releasing everything it allocates), it leaves behind a rather "coarse landscape" and the memory allocations required for the next tile may not fit nicely inside these holes. This causes the standard allocator to just grab some more "fresh" memory from the system (by increasing the data segment ceiling), which results in a bigger and bigger process memory usage. If the memory allocator were to coalesce adjacent holes more aggressively, it might be able to find entire "pages" to be returned to the system, but I don't know if the glib allocator has this feature at all. However, jemalloc
can do that.
Fragmentation is still an issue (and will always be with a non-garbage collecting system/language), but switching to this new allocator has shown a much nicer memory consumption pattern.
PS: My leak detection findings are very rudimentary and I don't claim that I found all leaks. But to further investigate, renderd
should be able to terminate gracefully and shut down all threads. As it is coded right now, it just runs "forever"...
@rolandbosa, great work! Thank you very much for working on this!
I just thought I'd mention my findings on this issue:
* I ran a leak detection and found a tiny, insignificant leak during the configuration parsing. It's 32 bytes and happens once during process startup.
Yes, that was me, I did try and clean everything up and did also check for memory leaks, but I added that modification later and must have missed it. I will take a look at your commit and test it out, feel free to open a pull request so that others may also do so.
* I switched out the memory allocator (from glib to jemalloc) and that shows promise...
Good idea, it might be good to also try out https://github.com/google/tcmalloc, which is what openstreetmap.org's tile servers have been using for a while now (since this commit).
We should document that JIT should be avoided on tileservers because of memory issues (and more general reasons) and close this issue and open a new issue for any new memory issues.
We should document that JIT should be avoided on tileservers because of memory issues
The patches in the linked bug were applied and the issue is fixed in PG 17 and backported to all branches. I still recommend turning off JIT for the slowness issues with most stylesheets that happen with JIT in some conditions, but that's nothing to do with mod_tile and should be documented by the stylesheets.
Thanks @pnorman, I will soon be closing this issue unless there are any objections. Then, I will open up two new issues to track any memory leaks (including the aforementioned one) and JIT-related documentation updates.
@hummeltech I'll pull out the commit for the strntok_r
code into a separate branch and will mention it/create a PR in the new issue.
Regarding the memory consumption in general: I've been experimenting with tcmalloc
a little and it seems to have slightly smaller footprint than jemalloc
. But for some reason yet unknown, both have deteriorated into glib
-similar consumption issues... I was expecting ~3GB for jemalloc
/tcmalloc
compared to ~17GB for glib
. I'm seeing >10GB for either allocator. Something is different than my setup of a few days ago, hopefully I can figure out what it is.
I've committed a few traces and hope to visualize them to get a better idea of what's going on.
Here's a quick and dirty visualization of the respective memory footprints. All tests are done consecutively, but plotted on the same (relative) time axis.
Conclusions so far:
jemalloc
seems the only one that returns substantial memory to the systemtcmalloc
seems to not ever return anything, maybe additional configuration is required?glib
does have a dip here and there - are those just "lucky" allocations that reduced the top-of-heap?Timeline:
0s: start renderd
- it loads the mapnik style file within ~10-15 seconds
30s: issue render request for all tiles at zoom level 6
45s-100s: database is churning like crazy, renderd
threads are mostly idle
100-220s: renderd
threads are using up CPU and memory
220s-250s: render_list exits, all rendering is done
300s: benchmark stops
Closing this issue in favor of new replacement issue #446 for adding documentation recommending the disabling of JIT (even though the previously existing memory leak in PostgreSQL should have been resolved).
Another issue #445 has also been created in order to track potential memory leaks in this project.
Have done a whole Europe import and am now trying to prerender the tiles using reder_list. This works fine until ZoomLevel 10. I have 6 threads running and renderd consumes about 10-15GB. around tile 528 memory consumption jumps to more than 48GB which causes oom_killer (renderd runs in a proxmox CT) to kill renderd. Jun 5 08:38:17 renderd renderd[1882]: Rendering projected coordinates 10 528 368 -> 626172.135713|5322463.153556 939258.203569|5635549.221413 to a 8 x 8 tile Jun 5 08:40:29 renderd systemd[1]: renderd.service: Main process exited, code=killed, status=9/KILL Is this a mod_tile/renderd or mapnik issue. Can I do something about this ? thx