search

SpringMT / zstd-ruby

Ruby binding for zstd(Zstandard - Fast real-time compression algorithm)
https://github.com/facebook/zstd
BSD 3-Clause "New" or "Revised" License
69 stars 16 forks source link

Add streaming decompression #15

Closed Fonsan closed 4 years ago

Fonsan commented 6 years ago

Adds streaming decompression with by passing enumerator yielding parts of the input, see specs for usage.

TODO add Usage to readme

SpringMT commented 6 years ago

Great! 1 commented.

Fonsan commented 6 years ago

While testing this in a private project I came across a memory leak scenario:


enumerator = Zstd.enum_for(:decompress_streaming, buffer_enumerator)

loop do
  enumerator.first(50)
end

As each execution is run in its own fiber and because the execution is paused before the decompression fully terminates zstd streams are left open and not freed.

This is fundamental in how Fibers work in ruby and I do not see a way to support the drafted syntax without leaving a memory leak.

I have started to formulate a different syntax which would ensure proper cleanup even in scenarios when decompression ends prematurely

andrew-aladev commented 4 years ago

Hello, I've released new version of ruby-zstds with streaming and multi threaded support (:gvl => true/false). You may be interested in reviewing extension code. It uses different approach to implement streaming, this code is about 95% covered with tests, it may help you with finishing this pull request. Thank you.

Fonsan commented 4 years ago

@andrew-aladev excellent thank you for letting me know, your approach seems to be more robust, I am abandoning this pr for now