oven-sh / bun

Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one
https://bun.sh
Other
73.01k stars 2.66k forks source link

Zipping files is extremely slow with archiver or jszip #12397

Open mnapoli opened 2 months ago

mnapoli commented 2 months ago

What version of Bun is running?

1.1.18+5a0b93523

What platform is your computer?

Darwin 23.4.0 arm64 arm

What steps can reproduce the bug?

Rough script:

var fs = require("fs");
var JSZip = require("jszip");

var zip = new JSZip();
// Call this 10k times on 10k files
// zip.file(path, fs.readFileAsync(path));

zip
.generateNodeStream({type:'nodebuffer',streamFiles:true})
.pipe(fs.createWriteStream('out.zip'))
.on('finish', function () {
    console.log("out.zip written.");
});

What is the expected behavior?

The script should complete in a couple of seconds. That's what happens with Node.

What do you see instead?

The script is extremely slow: if I log progress, I see 0.01% to 0.1% progress every second. Needless to say the script would complete in several minutes or possibly hours.

Additional information

I tried both archiver and jszip, hoping the problem was specific to an implementation. No difference.

I also tried replacing fs.readFileAsync(path) with Bun.file(path).blob() but it made no difference. Loading all files in memory and adding them to the zip instance takes a second at most (including with Bun).

The time is then spent by the zipping method.

whekin commented 2 months ago

I experienced a similar issue. I had to switch the library to PIzzip, which is a synchronous fork of the library and it worked okay. I'd also add that I worked with small docx documents, and while Node took a couple of dozens milliseconds to generate a buffer, it took 30x more time in Bun (around 1-1.5 seconds)

ghiscoding commented 1 month ago

see old PR #11770