I want to transfer a project to the fastify organisation - a more performant encoding negotiator

[Uzlopak]

Hi Fastify Team, especially @jsumners @mcollina @climba03003 @kibertoad

I rewrote the encoding-negotiator package.You can see it my proposal here: https://github.com/Uzlopak/encoding-negotiator

It is a drop-in-replacement. It passes all the unit tests of the original. I also added some unit test from the negotiator package.

Basically I wrote my own encoding parser and wrote a more simple implementation avoiding creating too many intermediary Objects/Arrays and nos Array-helper-functions.

Looking forward for your feedback. If you give me positive feedback I would transfer the projet to you. :)

And here the benchmarks:

encoding-negotiator

identity;q=1 and gzip,identity x 2,395,449 ops/sec ±1.31% (85 runs sampled)
gzip;q=1, identity;q=0.5 and gzip,deflate x 906,691 ops/sec ±0.98% (87 runs sampled)
deflate;q=0.5,identity; q=0.5 and gzip,deflate x 978,052 ops/sec ±1.24% (90 runs sampled)
deflate;q=0.5, gzip;q=0.5 and gzip,deflate x 798,266 ops/sec ±0.77% (92 runs sampled)
deflate;q=0.5, gzip;q=0.5 and deflate,gzip x 793,106 ops/sec ±1.03% (89 runs sampled)
* and gzip,deflate x 3,114,850 ops/sec ±1.08% (85 runs sampled)
deflate;q=1.0, * and gzip x 1,375,655 ops/sec ±0.93% (90 runs sampled)
test,br and br x 1,690,280 ops/sec ±0.89% (86 runs sampled)
gzip;q=0 and  x 2,889,740 ops/sec ±0.86% (87 runs sampled)
gzip;q=0 and gzip,identity x 2,466,312 ops/sec ±1.02% (86 runs sampled)
white rabbit and gzip,identity x 3,090,214 ops/sec ±1.07% (87 runs sampled)
undefined and gzip,identity x 112,126,214 ops/sec ±1.06% (89 runs sampled)
compress;q=0.5, gzip;q=1.0 and gzip,compress x 807,871 ops/sec ±0.89% (88 runs sampled)
gzip;q=1.0, compress;q=0.5 and compress,gzip x 867,816 ops/sec ±1.02% (86 runs sampled)
compress;q=0.5, gzip;q=1.0 and compress x 803,591 ops/sec ±0.81% (90 runs sampled)
gzip, deflate, br and br,gzip,deflate x 853,248 ops/sec ±0.93% (90 runs sampled)
* and br,gzip,deflate x 2,958,058 ops/sec ±1.05% (86 runs sampled)
*;q=0, identity;q=1 and gzip,identity x 911,999 ops/sec ±0.98% (90 runs sampled)
identity;q=0 and identity x 2,760,307 ops/sec ±1.02% (90 runs sampled)
gzip, compress;q=0 and compress,gzip x 1,062,974 ops/sec ±1.00% (88 runs sampled)
gzip;q=0.8, deflate and gzip,deflate x 1,097,123 ops/sec ±0.82% (89 runs sampled)
gzip;q=0.8, identity;q=0.5, *;q=0.3 and deflate,gzip,br x 575,336 ops/sec ±0.80% (91 runs sampled)

Performance of my proposed @fastify/encoding-negotiator

identity;q=1 and gzip,identity x 7,392,507 ops/sec ±1.08% (87 runs sampled)
gzip;q=1, identity;q=0.5 and gzip,deflate x 3,809,129 ops/sec ±1.36% (87 runs sampled)
deflate;q=0.5,identity; q=0.5 and gzip,deflate x 2,679,748 ops/sec ±1.14% (90 runs sampled)
deflate;q=0.5, gzip;q=0.5 and gzip,deflate x 3,118,989 ops/sec ±0.97% (89 runs sampled)
deflate;q=0.5, gzip;q=0.5 and deflate,gzip x 3,216,800 ops/sec ±0.96% (86 runs sampled)
* and gzip,deflate x 1,171,889,836 ops/sec ±0.91% (87 runs sampled)
deflate;q=1.0, * and gzip x 5,283,509 ops/sec ±1.08% (90 runs sampled)
test,br and br x 10,228,472 ops/sec ±1.18% (86 runs sampled)
gzip;q=0 and  x 215,294,057 ops/sec ±0.66% (89 runs sampled)
gzip;q=0 and gzip,identity x 10,953,354 ops/sec ±0.87% (90 runs sampled)
white rabbit and gzip,identity x 7,254,849 ops/sec ±1.74% (84 runs sampled)
undefined and gzip,identity x 215,927,839 ops/sec ±0.90% (92 runs sampled)
compress;q=0.5, gzip;q=1.0 and gzip,compress x 3,050,411 ops/sec ±0.85% (83 runs sampled)
gzip;q=1.0, compress;q=0.5 and compress,gzip x 3,037,218 ops/sec ±0.87% (87 runs sampled)
compress;q=0.5, gzip;q=1.0 and compress x 3,110,427 ops/sec ±1.08% (87 runs sampled)
gzip, deflate, br and br,gzip,deflate x 4,878,550 ops/sec ±1.19% (89 runs sampled)
* and br,gzip,deflate x 382,854,995 ops/sec ±32.82% (32 runs sampled)
*;q=0, identity;q=1 and gzip,identity x 5,360,172 ops/sec ±1.21% (90 runs sampled)
identity;q=0 and identity x 6,917,270 ops/sec ±1.09% (89 runs sampled)
gzip, compress;q=0 and compress,gzip x 4,785,504 ops/sec ±1.00% (90 runs sampled)
gzip;q=0.8, deflate and gzip,deflate x 3,797,374 ops/sec ±1.05% (92 runs sampled)
gzip;q=0.8, identity;q=0.5, *;q=0.3 and deflate,gzip,br x 2,193,807 ops/sec ±0.90% (88 runs sampled)

[climba03003]

Given that it is a drop-in-replacement for something, the plugin that using it is fastify-static

I think it follow the same track of fast-uri or busboy. A new name should be raised as it is a version writing from scratch.

[Uzlopak]

@climba03003 @fastify/compress is using it too.

I am open for name suggestions.

[Uzlopak]

I thought about the solution. I would like to restrict the lengrh of accept encodung to about 64 characters and maximum size of the result of parse to about 7. Or else somebody can send huge headers and can potentially do harm.

Wdyt?

[Uzlopak]

I implemented it now in a way, which should use less memory and could potentially also handle very large strings.

I also think I balanced the performance slightly better now:

identity;q=1 and gzip,identity x 7,197,232 ops/sec ±1.16% (88 runs sampled)
gzip;q=1, identity;q=0.5 and gzip,deflate x 11,189,322 ops/sec ±0.93% (92 runs sampled)
deflate;q=0.5,identity; q=0.5 and gzip,deflate x 2,665,658 ops/sec ±0.93% (90 runs sampled)
deflate;q=0.5, gzip;q=0.5 and gzip,deflate x 3,171,891 ops/sec ±0.87% (90 runs sampled)
deflate;q=0.5, gzip;q=0.5 and deflate,gzip x 3,149,777 ops/sec ±1.29% (88 runs sampled)
* and gzip,deflate x 725,137,702 ops/sec ±1.71% (87 runs sampled)
deflate;q=1.0, * and gzip x 5,071,701 ops/sec ±0.91% (91 runs sampled)
test,br and br x 9,902,619 ops/sec ±1.49% (88 runs sampled)
gzip;q=0 and  x 88,254,473 ops/sec ±0.89% (90 runs sampled)
gzip;q=0 and gzip,identity x 12,530,714 ops/sec ±1.58% (87 runs sampled)
white rabbit and gzip,identity x 7,068,896 ops/sec ±0.86% (92 runs sampled)
undefined and gzip,identity x 85,644,043 ops/sec ±0.85% (92 runs sampled)
compress;q=0.5, gzip;q=1.0 and gzip,compress x 2,890,048 ops/sec ±0.97% (92 runs sampled)
gzip;q=1.0, compress;q=0.5 and compress,gzip x 2,984,300 ops/sec ±1.11% (88 runs sampled)
compress;q=0.5, gzip;q=1.0 and compress x 2,887,101 ops/sec ±0.99% (87 runs sampled)
gzip, deflate, br and br,gzip,deflate x 4,671,692 ops/sec ±1.35% (88 runs sampled)
* and br,gzip,deflate x 496,582,442 ops/sec ±16.78% (68 runs sampled)
*;q=0, identity;q=1 and gzip,identity x 5,614,068 ops/sec ±1.02% (91 runs sampled)
identity;q=0 and identity x 7,489,868 ops/sec ±1.27% (87 runs sampled)
gzip, compress;q=0 and compress,gzip x 5,010,295 ops/sec ±1.15% (88 runs sampled)
gzip;q=0.8, deflate and gzip,deflate x 3,913,630 ops/sec ±0.92% (90 runs sampled)
gzip;q=0.8, identity;q=0.5, *;q=0.3 and deflate,gzip,br x 2,369,812 ops/sec ±0.93% (90 runs sampled)

[Uzlopak]

I will later test if it works also for Accept-Language header and Accept-Charset. If so, we could name it more common. Like @fastity/header-negotiator

[jsumners]

Can you explain what this is for? I don't know what this solves/replaces.

[mcollina]

I've created https://github.com/fastify/accept-negotiator, feel free to send a PR there!

[kibertoad]

I thought about the solution. I would like to restrict the length of accept encodung to about 64 characters and maximum size of the result of parse to about 7. Or else somebody can send huge headers and can potentially do harm.

It should be OK, as long as it would be possible to pass an override.

[kibertoad]

@Uzlopak

• Typings are incorrect, it can also return undefined and null (why both, btw?)
• There is an inline function inside index.js. Is it unavoidable? I think this might cause deoptimization.
• for (let i = 0, il = header.length; i < il; ++i) { you can use var for performance or move let outside of that block, to avoid redefining it.

[Uzlopak]

@kibertoad I am currently preparing the PR for the package. For this I am currently investigating, how it will work with fastify-static and fastify-compress. Also checking if it makes a difference on returning undefined or null. etc.etc. Will provide the PR asap.