zLib decoding fails with failed to fill whole buffer

[Phill030]

I'm trying to decode a zLib Vec<u8> which works fine in C# & TypeScript but I'm trying it in Rust and it instantly fails with that error message. This is the code I use:

let mut buffer = Vec::new();
buffer = binary_reader.read_bytes_at(file_record.size as usize, file_record.offset as usize).unwrap(); // works fine
println!("Buffer: {:?}", buffer); // Works! Prints: Buffer: [120, 218, 251, 255, 207, 144, 193, 138, 193, 151, 161, 146, 33, 143, 33, 149, 161, 156, 161, 24, 72, 38, 51, 148, 48, 100, 50, 228, 3, 69, 120, 25, 184, 24]
if file_record.is_compressed {
    let mut decoder = Decoder::new(&buffer[..]).unwrap();
    match decoder.read_to_end(&mut buffer.clone()) {
        Ok(_) => {
            println!("{:?}", buffer);
        }
        Err(e) => {
            println!("Unexpected end of file\n[{}]",e)
        }
    }
}

Buffer: [120, 218, 251, 255, 207, 144, 193, 138, 193, 151, 161, 146, 33, 143, 33, 149, 161, 156, 161, 24, 72, 38, 51, 148, 48, 100, 50, 228, 3, 69, 120, 25, 184, 24]

[sile]

Could you share your TypeScript code that was able to decode the data?

I tried decoding the data using Node.js but it failed as follows:

> require('zlib')
> zlib.inflateSync(new Uint8Array( [120, 218, 251, 255, 207, 144, 193, 138, 193, 151, 161, 146, 33, 143, 33, 149, 161, 156, 161, 24, 72, 38, 51, 148, 48, 100, 50, 228, 3, 69, 120, 25, 184, 24]))
Uncaught Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (node:zlib:189:17)
    at Zlib.callbackTrampoline (node:internal/async_hooks:130:17)
    at processChunkSync (node:zlib:457:12)
    at zlibBufferSync (node:zlib:178:12)
    at Object.syncBufferWrapper [as inflateSync] (node:zlib:792:14)
    at REPL41:1:6
    at Script.runInThisContext (node:vm:128:12)
    at REPLServer.defaultEval (node:repl:570:29)
    at bound (node:domain:433:15) {
  errno: -5,
  code: 'Z_BUF_ERROR'
}

I tried with Python3 too, but it got the same error:

> import zlib
> >>> x = [120, 218, 251, 255, 207, 144, 193, 138, 193, 151, 161, 146, 33, 143, 33, 149, 161, 156, 161, 24, 72, 38, 51, 148, 48, 100, 50, 228, 3, 69, 120, 25, 184, 24]
>>> y = b''.join(x.to_bytes(1, byteorder='big') for x in x)
>>> zlib.decompress(y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

[Phill030]

@sile Sure: this is my TypeScript code:

import * as zlib from "zlib";
const inflatedBuffer = zlib.inflateSync(buffer, { finishFlush: zlib.constants.Z_FULL_FLUSH });

This is how I was able to solve your unexpected end of file error

[Phill030]

It should return

ff fe 31 00 3a 00 4d 00 79 00 6e 00 65 00 77 00 73 00 65 00 63 00 74 00 69 00 6f 00 6e 00 0d 00 0a

or

1:Mynewsection

if done right

[Phill030]

Using a 4 year old crate called inflate works, using the inflate_bytes_zlib method so you may want to have a look in that :D I hope this can be implemented/fixed

[sile]

Thank you.

After some investigation, it seems that the input data is somewhat corrupted, so unexpected EOF error is raised without Z_FULL_FLUSH option ( Z_FULL_FLUSH option seems to force to return decoded data even when the decoding process isn't completed). I'm not sure that libflate should fully support this case. But, anyway, I'm going to consider if there is a reasonable way to retrieve incomplete decoding data from the decoder instance.

[sile]

Added Decoder::unread_decoded_data() in #72. You can retrieve partial decoded data using the method even when the decoding process has failed as follows:

        let encoded_data = [
             120, 218, 251, 255, 207, 144, 193, 138, 193, 151, 161, 146, 33, 143, 33, 149, 161, 156,
             161, 24, 72, 38, 51, 148, 48, 100, 50, 228, 3, 69, 120, 25, 184, 24,
         ];

         let mut decoder = Decoder::new(&encoded_data[..]).unwrap();
         let mut buf = Vec::new();
         let result = decoder.read_to_end(&mut buf);
         assert!(result.is_err());
         buf.extend_from_slice(decoder.unread_decoded_data());

         let decoded_data = [
             255, 254, 49, 0, 58, 0, 77, 0, 121, 0, 110, 0, 101, 0, 119, 0, 115, 0, 101, 0, 99, 0,
             116, 0, 105, 0, 111, 0, 110, 0, 13, 0, 10,
         ];
         assert_eq!(buf, decoded_data);