Streaming interface never signals completion

[anna-is-cute]

Hi!

It was my understanding from reading the docs that the ondata callback would set the second parameter to true for the last chunk of decompressed data. From my own usage and examination of the code, it appears this is not true. This makes properly using the streaming interface impossible.

I'd really love to use this since I need streaming and don't want the bloat of wasm. Thanks c:

Edit: I also have a fair number of files that give an "unexpected EOF" when provided in small chunks to the streaming interface but work fine when provided as one large chunk (or when provided to the non-streaming function). Would you like a separate issue for that?

[101arrowz]

Could you send some data to reproduce the issues you're referring to? I haven't encountered either of them myself. No need to make a separate issue for the EOF bug yet.

[anna-is-cute]

The zstd-compressed file is in this zip archive: zstd-file.zip (github wouldn't let me upload it directly). The zstd cli tool decompresses it fine for me, and so does the fzstd non-streaming API.

const url = new URL(hash, base);
const resp = await fetch(url);
if (resp.status < 200 || resp.status > 299) {
    throw new Error(`${resp.status} error downloading ${hash}`);
}

const compressed = await resp.blob();

// this works, but the uncommented code does not
// const decompressed = new Blob([fzstd.decompress(new Uint8Array(await compressed.arrayBuffer()))]);

let decompressed = new Blob();
let resolver: (value: undefined) => void;
const promise = new Promise(resolve => resolver = resolve);
const stream = new fzstd.Decompress((chunk, isLast) => {
    decompressed = new Blob([decompressed, chunk]);
    if (isLast) {
        // this never happens for any file
        console.log('is last');
        resolver(undefined);
    }
});

const CHUNK_SIZE = 16 * 1024;
const chunks = Math.ceil(compressed.size / CHUNK_SIZE);
for (let i = 0; i < chunks; i++) {
    const chunk = compressed.slice(i * CHUNK_SIZE, (i + 1) * CHUNK_SIZE);
    const array = await chunk.arrayBuffer();

    // the final chunk will throw the unexpected EOF error
    // unless you provide the entire compressed data in one chunk
    stream.push(new Uint8Array(array), i === chunks - 1);
}

// wait for the final chunk
await promise;

files[hash] = decompressed;

This is the code I'm running. I've done just about every test I can think of to verify that I am indeed providing the chunks correctly. Hopefully there's just a simple thing I'm doing wrong!

[101arrowz]

Thanks, I'll take a look at this in a few hours!

[101arrowz]

A few hours later, this is fixed!