Non-blocking IO

[seanmonstar]

I've using libflate in reqwest for a little while now, and quite pleased with, thanks!

I'm now updating reqwest to use non-blocking IO, find that the reader decoders are not able to handle it. This is due to several read_exact and read_u32 etc calls that require read_exact. It's not really a recoverable operation, since there's no defined way to determine how many bytes were actually available.

A solution would be to create the Decoder immediately, and keep an internal buffer, and an state enum. That way if in a state of needing to 'read exact', but that many bytes aren't available yet, the internal buffer holds the partial bytes, and the state keeps a position, and you can try again when a user calls encoder.read() again.

[sile]

Thanks for using libflate. Supporting non-blocking I/O is an interesting feature, so I would like to make libflate can handle it.

---

A solution would be to create the Decoder immediately, and keep an internal buffer, and an state enum.

I think that this solution will work, but I'm concerned about the possibility that the performance may decline. I am going to try implementing the feature in this (or the next) weekend and it's feedback will be written here.

[sile]

At the version 0.1.9, I added the non_blocking module that includes decoders which support non-blocking I/O.

Below is a benchmark result: (The non-blocking version decoder is somewhat inferior in performance to others)

$ cd libflate/flate_bench/
$ cargo run --release -- enwiki-latest-all-titles-in-ns0
# ENCODE (input_size=279095302)
- libflate: elapsed=5.496754s, size=96761378
-   flate2: elapsed=9.369781s, size=74692153

# DECODE (input_size=74692153)
-                libflate: elapsed=1.095406s, size=279095302
- libflate (non-blocking): elapsed=1.715038s, size=279095302
-                  flate2: elapsed=0.789984s, size=279095302
-                 inflate: elapsed=1.643498s, size=279095302

[seanmonstar]

Thanks! I'll try it out in async reqwest.