kevva / download

Download and extract files
MIT License
1.28k stars 200 forks source link

MaxRedirectsError when downloading a http-to-https url #159

Closed frankshaka closed 6 years ago

frankshaka commented 6 years ago

When downloading a file over a http: proxy from an http: URL that redirects to https: URL, download reuses the httpOverHttp tunneling agent created by caw, which then causes new requests always sent with http protocol. That would generate a redirect loop until got finally throws a MaxRedirectsError.

Sample:

const download = require('download')
download('http://pngquant.org/pngquant-2.10.1-src.tar.gz', {
  proxy: 'http://localhost:8080'
}).then(res => console.log(res))
  .catch(err => console.error(err))

Result:

{ MaxRedirectsError: Redirected 10 times. Aborting.
    at ClientRequest.fn.request.res (/Users/frankshaka/Temp/test-download/node_modules/got/index.js:69:23)
    at Object.onceWrapper (events.js:254:19)
    at ClientRequest.emit (events.js:159:13)
    at HTTPParser.parserOnIncomingClient (_http_client.js:550:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:117:23)
    at Socket.socketOnData (_http_client.js:439:20)
    at Socket.emit (events.js:159:13)
    at addChunk (_stream_readable.js:265:12)
    at readableAddChunk (_stream_readable.js:252:11)
    at Socket.Readable.push (_stream_readable.js:209:10)
  name: 'MaxRedirectsError',
  host: 'pngquant.org',
  hostname: 'pngquant.org',
  method: 'GET',
  path: '/pngquant-2.10.1-src.tar.gz',
  protocol: 'https:',
  url: 'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
  statusCode: 301,
  statusMessage: 'Moved Permanently',
  redirectUrls: 
   [ 'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz',
     'https://pngquant.org/pngquant-2.10.1-src.tar.gz' ] }

It should be considered that the proxy agent should be re-created in case of protocol-changing redirects.

n0v1 commented 6 years ago

We had exactly the same problem and I recently wrote a pull request to fix this. See #158.