Crashes most likely related to GDAL limits

[georgbachmann]

Describe the bug

In my project I have a a lot of raster data that I want to render. Those are some vrts where some of them have thousands of files behind them. I recently now start having a lot of crashes and as a little hint I got the following message before my docker-container exited.

Too many threads are running for the current value of the dataset pool size (100). or too many proxy datasets are opened in a cascaded way. Try increasing GDAL_MAX_DATASET_POOL_SIZE.

So I guess my main question would be where I can set that parameter? Setting it as environment variable did not help.

Environment: (Note: We are not setup to test on Linux or Windows so help may be slower on those OSs)

• Docker (ubuntu:18.04 image) using the 1.1.0 tag from this repo here

Installation Method Docker

Screenshots and logfile Here are also some logs from the latest crash. It happens after a while when I pan around in my map.

tilemill_1              | [tilemill] Starting in server mode.
tilemill_1              | [tilemill] Plugin [carto] loaded.
tilemill_1              | [tilemill] Plugin [editor] loaded.
tilemill_1              | [tilemill] Plugin [fonts] loaded.
tilemill_1              | [tilemill] Plugin [templates] loaded.
tilemill_1              | [tilemill] Started [Server Core:20009].
tilemill_1              | [tilemill] Started [Server Tile:20008].
tilemill_1              | [tilemill] [millstone] processing style 'style.mss'
tilemill_1              | [tilemill] [millstone] processing style 'colors.mss'
tilemill_1              | [tilemill] [millstone] processing style 'landuse.mss'
tilemill_1              | [tilemill] [millstone] processing style 'slope.mss'
tilemill_1              | [tilemill] [millstone] processing layer 'global_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_20m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_20m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_10m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_10m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_5m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_5m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_2m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_2m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_1m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_1m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'slope_global'
tilemill_1              | [tilemill] [millstone] processing layer 'countries'
tilemill_1              | [tilemill] [millstone] processing layer 'slope'
tilemill_1              | [tilemill] [millstone] found previously unzipped file: /root/Documents/MapBox/cache/82945364-10m-admin-0-countries/82945364-10m-admin-0-countries.shp
tilemill_1              | [tilemill] [millstone] finished processing '/root/Documents/MapBox/project/hillshading'
tilemill_1              | [tilemill] [millstone] processing style 'style.mss'
tilemill_1              | [tilemill] [millstone] processing style 'colors.mss'
tilemill_1              | [tilemill] [millstone] processing style 'landuse.mss'
tilemill_1              | [tilemill] [millstone] processing style 'slope.mss'
tilemill_1              | [tilemill] [millstone] processing layer 'global_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_20m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_20m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_10m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_10m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_5m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_5m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_2m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_2m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'global_1m_hs_az315_alt60'
tilemill_1              | [tilemill] [millstone] processing layer 'global_1m_hs_az45_alt80'
tilemill_1              | [tilemill] [millstone] processing layer 'slope_global'
tilemill_1              | [tilemill] [millstone] processing layer 'countries'
tilemill_1              | [tilemill] [millstone] processing layer 'slope'
tilemill_1              | [tilemill] [millstone] found previously unzipped file: /root/Documents/MapBox/cache/82945364-10m-admin-0-countries/82945364-10m-admin-0-countries.shp
tilemill_1              | [tilemill] [millstone] finished processing '/root/Documents/MapBox/project/hillshading'
tilemill_1              | [tilemill] Error: child process: "tile" failed with signal "SIGSEGV"
tilemill_1              | [tilemill] Closing child process: core (pid:14)
tilemill_1              | [tilemill] Closing child process: tile (pid:15)
tilemill_1              | Exiting [tilemill]

[csytsma]

@georgbachmann Did you make any progress on this? Do you think it's a Docker or Millstone issue?

[georgbachmann]

@csytsma I change my datasource. So not those super large vrts, but more smaller ones and that 'solved' the problem for me...