search

SpringMT / zstd-ruby

Ruby binding for zstd(Zstandard - Fast real-time compression algorithm)
https://github.com/facebook/zstd
BSD 3-Clause "New" or "Revised" License
67 stars 16 forks source link

Streaming decompression questions #87

Open gucki opened 4 months ago

gucki commented 4 months ago

I have a tcp connection that receives the zstd compressed data. Let's assume I do a data = socket.read(16*1024) and this fills my data with a zstd frame and another partial zstd frame. If I use decompressor.decompress(buf) (decompressor being an instance of Zstd::StreamingDecompress) I only get back the data of the first frame. How can I know how much data from data has not been processed and needs to be reused after receiving more data from the socket? Afaik the original zstd library returns this information, but the ruby abstraction discards it?

SpringMT commented 4 months ago

Thank you for the issues. I would like to understand more about what you mean by "the original zstd library returns this information." Could you please provide more details or an example of how the original zstd library provides this information?

gucki commented 4 months ago

Hey @SpringMT , I just implemented a new method that returns the needed details to the caller and opened a PR. If the example/ use case in the PR is unclear, please let me know and I'll explain further. I tested my implementation quite a bit and it works without any issues so far.