systemed / tilemaker

Make OpenStreetMap vector tiles without the stack
https://tilemaker.org/
Other
1.5k stars 232 forks source link

use libdeflate rather than zlib #769

Closed cldellow closed 1 month ago

cldellow commented 1 month ago

The zlib implementation that ships with most distributions is fairly slow, even when you allow for the zlib algorithm itself being quite slow (vs lz4 and zstd).

There are faster implementations. libdeflate 1 is one such example. According to benchmarks 2, it's maybe 2-3x faster than zlib.

This PR updates helper.cpp's compression routines to use libdeflate.

It saves ~2-3% of total execution time for me:

Creating a GB mbtiles (zlib):

real 2m1.706s
user 28m24.186s
sys 0m41.886s

Creating a GB mbtiles (libdeflate):

real 1m58.450s
user 27m32.579s
sys 0m51.848s

Snapshotting an external dependency is sorta distasteful - I worry about how much is too much from a maintenance POV. To mitigate that, the snapshot is a direct copy of the upstream folders, so it should be easy to update in the future if needed.

This also lets us drop the zlib1g-devel and boost-iostreams dependencies, which is nice.

systemed commented 1 month ago

Thanks - useful improvement! Tested on Ubuntu, Intel Mac and ARM Mac and happily compiles and runs on all three.