The zlib implementation that ships with most distributions is fairly slow, even when you allow for the zlib algorithm itself being quite slow (vs lz4 and zstd).
There are faster implementations. libdeflate1 is one such example. According to benchmarks 2, it's maybe 2-3x faster than zlib.
This PR updates helper.cpp's compression routines to use libdeflate.
It saves ~2-3% of total execution time for me:
Creating a GB mbtiles (zlib):
real 2m1.706s
user 28m24.186s
sys 0m41.886s
Creating a GB mbtiles (libdeflate):
real 1m58.450s
user 27m32.579s
sys 0m51.848s
Snapshotting an external dependency is sorta distasteful - I worry about how much is too much from a maintenance POV. To mitigate that, the snapshot is a direct copy of the upstream folders, so it should be easy to update in the future if needed.
This also lets us drop the zlib1g-devel and boost-iostreams dependencies, which is nice.
The zlib implementation that ships with most distributions is fairly slow, even when you allow for the zlib algorithm itself being quite slow (vs lz4 and zstd).
There are faster implementations.
libdeflate
1 is one such example. According to benchmarks 2, it's maybe 2-3x faster than zlib.This PR updates helper.cpp's compression routines to use libdeflate.
It saves ~2-3% of total execution time for me:
Creating a GB mbtiles (zlib):
Creating a GB mbtiles (libdeflate):
Snapshotting an external dependency is sorta distasteful - I worry about how much is too much from a maintenance POV. To mitigate that, the snapshot is a direct copy of the upstream folders, so it should be easy to update in the future if needed.
This also lets us drop the zlib1g-devel and boost-iostreams dependencies, which is nice.