[koa-compress] Using this library adds 200+ ms to content download time

[kingjerod]

I've noticed that when I use this library, the "Content Download" time in Chrome dev tools is increased by 200+ ms. I see the same thing in Firefox for the "Receiving" time. I've created a small test script to isolate this issue from anything else:

const Koa = require('koa');
const app = new Koa();

app.keys = ['key1'];
app.use(require('koa-compress')({
  flush: require('zlib').Z_SYNC_FLUSH
}));

let data = [];
for(let i = 0; i < 50000; i++) {
  data.push('hello');
}
app.use(async (ctx, next) => {
  ctx.body = data;
});
app.listen(3000);
console.log('Listening on port 3000.');

This is using Koa v2, koa-compress v2.0, Node v7.2.1 along with the harmony flag.

With compression on, the page loads between 200-230 ms. Transferred amount is 637B. With compression off, the page loads between 8-30 ms. Transferred amount is 390.63KB.

It looks like compression is working fine, but why is it adding 200 ms?

I'm running these tests inside a local Ubuntu vagrant environment.

[jonathanong]

what happens if you take out the flush parameter?

[kingjerod]

Still takes the same amount of time 200+ ms.

[jonathanong]

it's probably your data payload being gigantic. if you streamed it, it should be a lot faster, at least for first byte

[kingjerod]

If I change that loop to only be 500 iterations, I still have the same problem. Only 4.1KB of data.

[gugadev]

Same here. There is no difference.

[fl0w]

I can't reproduce this. Tried using both node 7.2.1 and 8.1.0, on macOS 10.12.5.

$ time curl localhost:3000 &> /dev/null

real    0m0.018s
user    0m0.004s
sys 0m0.004s

// copy-pasted from above.
const Koa = require('koa');
const app = new Koa();

app.keys = ['key1'];
app.use(require('koa-compress')({
  flush: require('zlib').Z_SYNC_FLUSH
}));

let data = [];
for(let i = 0; i < 50000; i++) {
  data.push('hello');
}
app.use(async (ctx, next) => {
  ctx.body = data;
});
app.listen(3000);
console.log('Listening on port 3000.');

Either this has been fixed or there's some OS variable not provided here.

[greim]

I found similar behavior in Chrome devtools and also curl, using a 15MB json file full of enron emails. ~10s to download 10 files compressed, ~2.5s to download 10 files uncompressed. All downloads in serial order.

I guess it doesn't surprise me that gzip would be the bottleneck over localhost, but I worry parallel requests would saturate the core. Or maybe this is normal gzip penalty? At an old job we used to upload pre-gzipped files to our CDN to avoid such slowdowns.

// server code
const Koa = require('koa');
const app = new Koa();

app.keys = ['key1'];
app.use(require('koa-compress')({
  flush: require('zlib').Z_SYNC_FLUSH
}));

app.use(async (ctx, next) => {
  ctx.set('content-type', 'application/json; charset=UTF-8')
  ctx.body = require('fs').createReadStream(__dirname + '/enron.json', 'utf8');
});
app.listen(3000);
console.log('Listening on port 3000.');

// devtools console
(async () => {
  const start = Date.now();
  for (let i=0; i<10; i++) {
    await fetch('/').then(r => r.text());
  }
  const end = Date.now();
  console.log(end - start);
})()

# curl command
curl -v -H "Accept-Encoding: gzip" "http://localhost:3000/?foo=[0-9]" > /dev/null

koa-compress version: 3.0.0 Node version: 8.x and 10.x OS: Mac OS 10.11.6

[tracker1]

Try setting the compression level a bit lower {level:7}, should help. This should probably be the default, iirc zlib defaults to 9, which is quite a bit more expensive.

[greim]

@tracker1 - I don't think 9 is the default. zlib.Z_DEFAULT_COMPRESSION reports as being -1 for me. Setting to Z_BEST_SPEED (1) definitely speeds things up. Setting to 7 slows it down considerably from the default.

The above were for a 50mb json file.