In #960, when switching compression libraries, the default compression settings for LZ4 were also changed. In python-lz4, the default level was 0. Now it is 9.
We have seen an over 100% increase in producer latency in a production application due to this change. It happens to publish a relatively large binary payload. We are faced with a decision to either downgrade back to 0.10.0 or explicitly monkeypatch codec.lz4_compress.
Generally speaking, folks who are deliberately choosing to install lz4 when Standard Library natively supports others are opting for faster/weaker compression. From the link included in the function documentation:
the choice of going from 0/1 -> 9 yields a 45% latency increase for a 1.2% improvement in compression ratio. I don't think folks are signing up for this trade-off.
Expected behaviour
Our "expected" behavior is that, since one cannot explicitly control compression settings and only compression.type, that settings are identical to they were when aiokafka directly used python-lz4. Could we consider reverting the default level to 0 to match the original behavior pre-0.11.0?
Describe the bug
In #960, when switching compression libraries, the default compression settings for LZ4 were also changed. In
python-lz4, the default level was 0. Now it is 9.We have seen an over 100% increase in producer latency in a production application due to this change. It happens to publish a relatively large binary payload. We are faced with a decision to either downgrade back to 0.10.0 or explicitly monkeypatch
codec.lz4_compress.Generally speaking, folks who are deliberately choosing to install lz4 when Standard Library natively supports others are opting for faster/weaker compression. From the link included in the function documentation:
the choice of going from 0/1 -> 9 yields a 45% latency increase for a 1.2% improvement in compression ratio. I don't think folks are signing up for this trade-off.
Expected behaviour
Our "expected" behavior is that, since one cannot explicitly control compression settings and only
compression.type, that settings are identical to they were when aiokafka directly usedpython-lz4. Could we consider reverting the default level to 0 to match the original behavior pre-0.11.0?