Open cakedan opened 1 year ago
This is unfortunately an issue with using webstreams vs. node streams. Node streams perform much better than webstream currently.
If we don't decompress the response, it takes 4.4 seconds. If we do (using zlib.createBrotliDecompress()) it takes ~70 seconds.
... and if we re-assign this.body
to the result of pipeline
, it fixes the issue, but causes issues with invalid gzipped/brotli compressed bodies.
@ronag you know much more about streams than I do, do you have any idea?
... and if we re-assign this.body to the result of pipeline, it fixes the issue, but causes issues with invalid gzipped/brotli compressed bodies.
Not sure I follow.
So for example, this diff fixes the issue:
diff --git a/lib/fetch/index.js b/lib/fetch/index.js
index 0b2e3394..14e84b29 100644
--- a/lib/fetch/index.js
+++ b/lib/fetch/index.js
@@ -2023,7 +2023,7 @@ async function httpNetworkFetch (
status,
statusText,
headersList: headers[kHeadersList],
- body: decoders.length
+ body: this.body = decoders.length
? pipeline(this.body, ...decoders, () => { })
: this.body.on('error', () => {})
})
but then zlib/brotli tests fail (only tested on the node-fetch tests but it's safe to assume others will fail too)
I think it just removes the decompression...?
yeah it does, but what I don't understand is why it's causing an issue here, but not with node-fetch. Node-fetch uses pipeline & zlib too.
Webstreams...
I thought so too (made an issue in the performance repo), but considering that removing the decompression fixes the issue...?
Can be that web streams have smaller chunks which makes the decompression much less efficient... or something
I don't think this is something we can solve in undici.
web streams have smaller chunks
No, I don't think so? In the OP node-fetch has 10k more chunks than undici.fetch does. Adjusting the highwatermark/size didn't make much difference if I'm remembering correctly
🤷♂️
@cakedan do you have a repro that runs locally, without the external server? I can't seem to replicate the issue
Bug Description
I've noticed a bug where if the body of a fetch response is large (let us say above 50mb), grabbing the body contents takes a long time to load. I compared it to
undici
's normal request function andnode-fetch
's fetch response. In comparison,node-fetch
took about 8.5 to 9.4 seconds to grab a 130mb file,undici
's request function took 2.3 to 2.6 seconds, andundici
's fetch took 95.5 to 104.2 seconds.Reproducible By
undici
fetch, 95.5 seconds, 18,605 buffersundici
request, 2.6 seconds, 9794 buffersnode-fetch
fetch, 9.4 seconds, 28909 buffers(Also, doing
.arrayBuffer()
yields the same results)Expected Behavior
Similar timing to
undici
'srequest
functionLogs & Screenshots
Environment
Ubuntu 22.10, Node v19.8.1 and v18.7.0, undici v5.2.0 and v5.21.0
Additional context