Frame format

[matreyes]

Hi!

I have stumbled upon this library because I need to consume an lz4 compressed topic in Kafka, and I'm having an issues with lz4b compilation (C++ nif).

Then, trying it out I've realized that Kafka uses the lz4 frame format https://docs.rs/lz4_flex/latest/lz4_flex/#example-decompress-data-on-stdin-with-frame-format

Sadly I have zero experience with Rust to offer a PR.

Is there anyone interested in adding this feature?

Thanks!

[whatyouhide]

Hey @matreyes thanks for the issue. Do you want to give the code in #10 a try to see if it works with the use case you're facing?

You'd have to depend on the GH branch itself:

{:nimble_lz4, github: "whatyouhide/nimble_lz4", branch: "al/frame"}

and make sure to run every Mix command prepended by NIMBLELZ4_FORCE_BUILD=true.

[matreyes]

It works ❤️!

For the record: Using brod 4.0.0 (kafka library), you have to add to your config file:

config :kafka_protocol, provide_compression: [lz4: KafkaLZ4]

and then create a simple interface like:

defmodule KafkaLZ4 do
    def decompress(compressed) do
       {:ok, decompressed} = NimbleLZ4.decompress_frame(compressed)
       decompressed
    end

    def compress(decompressed) do
       NimbleLZ4.compress_frame(decompressed)
    end
end

And finally, wherever you start the producer, make sure to pass the configuration [compression: :lz4]

Thanks @whatyouhide Se fai un giro a Bologna, invito la cena!

[whatyouhide]

Released v1.1.0 :)

[nestor-sk]

Believe it or not, I was preparing an issue asking for the frame format, too. I started this morning, and now I was going to look for some code to paste into the issue, and I found it changed. Thank you!