tomas / needle

Nimble, streamable HTTP client for Node.js. With proxy, iconv, cookie, deflate & multipart support.
https://www.npmjs.com/package/needle
MIT License
1.63k stars 236 forks source link

zlib error with brotli #314

Closed JiPaix closed 4 years ago

JiPaix commented 4 years ago

I use node v12.16.1 and the code below throws me an error: unexpected end of file at BrotliDecoder.zlibOnError [as onerror] (zlib.js:180:17)

var needle = require("needle")

needle("get", 'https://www.cloudflare.com/', {compressed: true}).then((r) => {
    console.log(r.headers)
}).catch((e) => {
    throw e
})

It works if i comment out this line: https://github.com/tomas/needle/blob/0ef49f6d3613391b10be88531824861f6167414d/lib/needle.js#L62 from: https://github.com/tomas/needle/blob/0ef49f6d3613391b10be88531824861f6167414d/lib/needle.js#L60-L64 I couldn't reproduce issues from https://github.com/tomas/needle/issues/202 but the code from above still works on all version mentioned

v5.12.0 (which is the latest v5), v6.9.5 (latest v6), and v7.5.0 (latest v7)

Phygon commented 4 years ago

I ran into a similar error as the OP on the current nodejs 12.x:

Error: unexpected end of file
    at BrotliDecoder.zlibOnError [as onerror] (zlib.js:180:17)

The error does not happen on every request.

The zlib_options mentioned above appear to be wrong in case we decompress a brotli stream.

 // Enable Z_SYNC_FLUSH to avoid Z_BUF_ERROR errors (Node PR #2595) 
 var zlib_options = { 
   flush: zlib.Z_SYNC_FLUSH, 
   finishFlush: zlib.Z_SYNC_FLUSH 
 } 

According to the nodejs docs,

  // For Brotli, the equivalent is zlib.constants.BROTLI_OPERATION_FLUSH.
  { finishFlush: zlib.constants.Z_SYNC_FLUSH },

the values have to be BROTLI_OPERATION_FLUSH instead of Z_SYNC_FLUSH if decompressing a brotli stream. I tried with both flush and finishFlush set to BROTLI_OPERATION_FLUSH, and I could decompress br encoded responses without an error.

tomas commented 4 years ago

Fixed by #326, available in npm as needle@2.5.2