search

t-rex-tileserver / t-rex

t-rex is a vector tile server specialized on publishing MVT tiles from your own data
https://t-rex.tileserver.ch/
MIT License
545 stars 68 forks source link

limit number of tiles generated async due to spatialite limitation #235

Closed arbakker closed 3 years ago

arbakker commented 3 years ago

Fixes https://github.com/t-rex-tileserver/t-rex/issues/234

The libspatialite max connections limitation is solved with libspatialite v4.4.0 (when compiled against GEOS 3.5.0), see https://groups.google.com/g/spatialite-users/c/xrV7CA_GlwM .

Commented the fix, so it can be removed at some later point in time.

pka commented 3 years ago

Thanks a lot for the investigation! The task queue size should be greater or equal than the number of available CPU cores. Instead of detecting the available cores, I've set the size to the highest number of cores I could imagine. But when all these tasks use a DB connection, then we should definetively reduce them to the minimum. I will check whether we loose performance with setting the queue size to the number of cores.

pka commented 3 years ago

The build failure on Mac comes from an incompatibility of the DummyDatasource replacing the optional GDALDatasource. But I'll try to avoid the datasource check anyway.

arbakker commented 3 years ago

Aah, that makes sense. I already thought the task queue was quite big, but that explains. I suppose that the queue size should be limited to 64 for spatialite formats, if number of available cores > 64, in case the queue size is set to the number of cores.

pka commented 3 years ago

In a quick test, num_cpus * 2 was faster than num_cpus and 128. See f4d0665