sindresorhus / got

🌐 Human-friendly and powerful HTTP request library for Node.js
MIT License
14.27k stars 935 forks source link

content-length header not set correctly when streaming small files. #1279

Closed quinw68 closed 3 years ago

quinw68 commented 4 years ago

Describe the bug

When piping a got http GET readable stream to an express "response" writable stream, the content-length header is automatically getting set incorrectly for small files that only have one chunk to receive in the stream. It seems to be using the value from the "downloadProgress" event instead of how many bytes were actually read.

Actual behavior

The content-length header that is set by the got stream is not set to the actual number of bytes received for small files (files that only require reading one chunk from the stream).

Expected behavior

If the number of bytes received in a stream is small, it should be able to correctly set the content-length header so anything the stream is piped into can reference it.

Code to reproduce

import { Response } from 'express';

function streamFile(response: Response) {
  let readable = got.stream(`http://some-url/with/small/file`, {
      headers: { Authorization: `Bearer ${this.token}`},
      searchParams: querystring.stringify(queryParams)
    });

  // For a 98 byte text file, the browser receives a content-length header of "33"
  readableStream.pipe(response);
}

Checklist

quinw68 commented 4 years ago

FYI, a temporary workaround to allow the browser to handle the content-length on its own is to just remove the content-length header after receiving the response:

readableStream.on('response', () => { res.removeHeader('content-length'); })

szmarczak commented 4 years ago

What happens if you set options.decompress to false?

quinw68 commented 4 years ago

@szmarczak That actually works and makes sense as to why the header had the wrong size. I guess this is a non-issue. Thanks for the tip sir