search

airlift / aircompressor

A port of Snappy, LZO, LZ4, and Zstandard to Java
Apache License 2.0
568 stars 113 forks source link

Support lz4 framing decompression #188

Open guptashailesh92 opened 6 months ago

guptashailesh92 commented 6 months ago

Is there any plan to support lz4 framing support while decompression. https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md

Currently Trino queries are failing if such compressed files are provided.

dain commented 6 months ago

Reading the code over it doesn't look too difficult to implement is needed (we do this for Snappy)

That said, Hadoop has it's own framing format for LZ4 which isn't compatible with anything else. Does Hive or Spark support these files directly? If so, what compresson codec are you specifying? In the past Hadoop would introduce a second codec when there is a conflict between the built in framing and one provided by compression project (normally happens when the framing comes out after hadoop has created its own).