search

101arrowz / fzstd

High performance Zstandard decompression in a pure JavaScript, 8kB package
MIT License
218 stars 11 forks source link

Fails on this input #9

Closed photopea closed 2 years ago

photopea commented 2 years ago

The library fails at this input. Is it a bug in a library or in the input data.txt ?

101arrowz commented 2 years ago

That file yields a 4.2 GB output. Obviously this will fail to decompress in memory. I tried to decompress with the streaming decompressor and it yielded the same output as zstd -d (and it was actually faster, surprisingly - probably due to I/O bandwidth limitations). What code did you use to decompress?

photopea commented 2 years ago

I used fzstd.decompress(uint8array) . It seems like it is because of a bug in Google Chrome, it works fine in Firefox.

https://bugs.chromium.org/p/chromium/issues/detail?id=1201109

photopea commented 1 year ago

What about this array? Is it a valid zstd stream? I can not decode it with any decoder. It is just 1.52 MB.

data.txt

101arrowz commented 1 year ago

It's valid and decompresses fine with the original Zstandard decompressor as well as fzstd when done streaming. The decompressed size of that file is 2.66GB, which is still too large to decompress in memory. I think JavaScript engines in general support up to 2^31 bytes (2GiB or about 2.14GB), though Firefox should support more.