search

airlift / aircompressor

A port of Snappy, LZO, LZ4, and Zstandard to Java
Apache License 2.0
562 stars 111 forks source link

Fix failure when data compressing incompressible data #135

Closed martint closed 2 years ago

martint commented 2 years ago

Incompressible data that requires more than maxCompressedLength(...) would cause the Huffman compressor to fail with an error due to lack of space in the output buffer.

With this change, when that happens, the compressor will fall back to emitting the raw literals instead.

Fixes https://github.com/airlift/aircompressor/issues/122

dongjoon-hyun commented 2 years ago

It's great to have this. May I ask when is the next release with this?

martint commented 2 years ago

The release is in progress. It should be available in maven central soon.

dongjoon-hyun commented 2 years ago

Thank you so much!

martint commented 2 years ago

@dongjoon-hyun, release 0.21 is now available in maven central. Please let me know if you encounter any issues.

dongjoon-hyun commented 2 years ago

Thanks. Yeps, for Apache ORC, it's tested and merged via https://issues.apache.org/jira/browse/ORC-965 . I guess @melin can test it more on his Spark jobs (the initial reported one), too.