Downloading through proxy with authentication

[khkassem]

Hi

I am using the latest version with Electron 9 - node 12

I am trying to pass through a proxy with authentication when downloading and i get an error 400

   const newDownloadHelper = new DownloaderHelper( url, folderPath, {
      fileName: filename,
      forceResume: true,

      retry: { maxRetries: 20, delay: 5000 },
      override: { skip: true, skipSmaller: true },
      removeOnStop: false,
      httpsRequestOptions: {
        agent: this.proxyAgent,
      },
    })

and this.proxyAgent is :

     const HttpsProxyAgent = require('https-proxy-agent')
     const urlLib = require('url')
      let proxyOpts = urlLib.parse(host)
      proxyOpts.auth = `${user}:${password}`
      proxyAgent = new HttpsProxyAgent(proxyOpts)

the Error stack :

Error: Response status was 400
    at ClientRequest.<anonymous> (c:\devz\workspace\ezad\Adtoad\node_modules\node-downloader-helper\dist\index.js:1:7280)
    at Object.onceWrapper (events.js:313:26)
    at ClientRequest.emit (events.js:223:5)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:583:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:115:17)
    at Socket.socketOnData (_http_client.js:456:22)
    at Socket.emit (events.js:223:5)
    at addChunk (_stream_readable.js:309:12)
    at readableAddChunk (_stream_readable.js:290:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:181:23) {status: 400, body: '', stack: 'Error: Response status was 400
    at ClientR…Read (internal/stream_base_commons.js:181:23)', message: 'Response status was 400'}

I use the same proxyAgent with axios without any problem in the same application

Is proxy not supported in node-downloader-helper or I am doing it the wrong way?

Thank you

[hgouveia]

Hello @khkassem , looking the code it should work, since the library just pass this information directly to https protocol when making the request, i am wondering if probably you are using a http url instead of https ? just to make sure, add the option to both

  httpRequestOptions: { agent: this.proxyAgent },
  httpsRequestOptions: { agent: this.proxyAgent },

[khkassem]

Hello

My destination URL is https, but I tried the two cases, I get the same error (400), if i put only httpRequestOptions (not https) I get an error 403.

the authentication is not the problem, i tested on a squid proxy without authentication and i get the same errors.

I tried global-agent (https://www.npmjs.com/package/global-agent) and i get the same results.

I looked to the code of node-downloader-helper, and I debugged it, the proxy options seems set in the request object (node-downloader-helper seems using http.request or https.request depending on the protocol.

In my application I am using axios to call api, and node-downloader-helper to download file, using the same proxy settings, the restfull api requests are working well through the proxy but not the downloads. I was using axios to download file before and it was working but i changed it to support some actions like resume.

Thank you

[khkassem]

Hello

I continued my investigation, the query is arriving to the proxy - from the logs of the proxy squid : 1646871838.114 335 127.0.0.1 TCP_TUNNEL/200 722 CONNECT zzz.mydomain.fr:443 - HIER_DIRECT/xxx.xxx.xxx.xxx -

and its arriving to my apache server, i get this line in the access logs of my apache server (400) : xxx.xxx.xxx.xxx - - [10/Mar/2022:01:31:46 +0100] "GET / HTTP/1.0" 400 476

when I download my file without proxy from browser (with or without proxy) or from node-downloader-helper without any proxy I get in the apache logs like this: xxx.xxx.xxx.xxx - - [10/Mar/2022:01:38:09 +0100] "GET /filename.zzz HTTP/1.1" 200 1436685

So the query is redirected from the proxy to the http server, but apache respond with 400, and the url seems wrong from the apache logs when using node downloader with a proxy. may be I have to add some headers to my options?

Thank you

[hgouveia]

@khkassem it seems this issue is related with #82 , i will try to release a fix soon and probably should fix yours aswell

[hgouveia]

Implemented on v2.1.0, let me know if this solved this issue as well

[khkassem]

Hello

It still not working, the same error 400, when I debugged it last time, I have a doubt that the query parameters were not taken, I tested with an url without any query params and it still not working after the update. I am trying to move to axios, its ok and passing through the proxy but i have many Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close in Node Pipeline stream]

node-downloader-helper was more stable, but this proxy problem is blocking me from using it.

Thank you

[hgouveia]

Hello, could you try by adding a User-Agent also you could try to add an additional Proxy-Authorization header with the credentials to see if make any difference

    const proxyAuth = Buffer.from(`${username}:${password}`, 'utf8').toString('base64');   
    const dl = new DownloaderHelper( url, folderPath, {
       //... others settings
       httpRequestOptions: { agent: this.proxyAgent },
       httpsRequestOptions: { agent: this.proxyAgent },
       headers: {
        'user-agent': 'ndh/2.1.0', // could be anything
        'Proxy-Authorization': `Basic ${proxyAuth}`
      },
    });

but at this point not sure what else we could do, i will eventually go back to this and implement a native proxy support to the library but it might time some time

[khkassem]

Hello

I am a litte more advanced, its Electron js which cause the problem. I tried the sample by adding the proxy to the file example/index.js and its working I can download through the proxy, I added the same file to my Electron js project and its not working i get the error 400.

I am using electron 9, I am going to try with a newer version.

/*eslint no-console: ["error", { allow: ["log", "warn", "error"] }] */
const { byteHelper } = require('../bin/helpers');
const { DownloaderHelper, DH_STATES } = require('../dist');
const url = 'https://www.google.fr'; // https://proof.ovh.net/files/
const pkg = require('../package.json');
const zlib = require('zlib');
const urlLib = require('url')

const pauseResumeTimer = (_dl, wait) => {
    setTimeout(() => {
        if (_dl.state === DH_STATES.FINISHED ||
            _dl.state === DH_STATES.FAILED) {
            return;
        }

        _dl.pause()
            .then(() => console.log(`Paused for ${wait / 1000} seconds`))
            .then(() => setTimeout(() => _dl.resume(), wait));

    }, wait);
};

var host = 'http://localhost:3128'
var user = 'test'
var password = 'test'

const HttpsProxyAgent = require('https-proxy-agent')
let proxyOpts = urlLib.parse(host)
//proxyOpts.auth = `${user}:${password}`
const proxyAgent = new HttpsProxyAgent(proxyOpts)

// these are the default options
const options = {
    method: 'GET', // Request Method Verb
    // Custom HTTP Header ex: Authorization, User-Agent
    headers: {
        'user-agent': pkg.name + '@' + pkg.version
    },
    retry: { maxRetries: 3, delay: 3000 }, // { maxRetries: number, delay: number in ms } or false to disable (default)
    fileName: filename => `${filename}`, // Custom filename when saved
    /* override
    object: { skip: skip if already exists, skipSmaller: skip if smaller }
    boolean: true to override file, false to append '(number)' to new file name
    */
    override: { skip: true, skipSmaller: true },
    forceResume: true, // If the server does not return the "accept-ranges" header but it does support it
    removeOnStop: true, // remove the file when is stopped (default:true)
    removeOnFail: true, // remove the file when fail (default:true)    
    httpRequestOptions: {}, // Override the http request options  
    httpsRequestOptions: {
        agent: proxyAgent
    } // Override the https request options, ex: to add SSL Certs
};

let startTime = new Date();
const dl = new DownloaderHelper(url, __dirname, options);

dl
    .once('download', () => pauseResumeTimer(dl, 5000))
    .on('download', downloadInfo => console.log('Download Begins: ',
        {
            name: downloadInfo.fileName,
            total: downloadInfo.totalSize
        }))
    .on('end', downloadInfo => console.log('Download Completed: ', downloadInfo))
    .on('skip', skipInfo =>
        console.log('Download skipped. File already exists: ', skipInfo))
    .on('error', err => console.error('Something happened', err))
    .on('retry', (attempt, opts, err) => {
        console.log({
            RetryAttempt: `${attempt}/${opts.maxRetries}`,
            StartsOn: `${opts.delay / 1000} secs`,
            Reason: err ? err.message : 'unknown'
        });
    })
    .on('resume', isResumed => {
        // is resume is not supported,  
        // a new pipe instance needs to be attached
        if (!isResumed) {
            dl.unpipe();
            dl.pipe(zlib.createGzip());
            console.warn("This URL doesn't support resume, it will start from the beginning");
        }
    })
    .on('stateChanged', state => console.log('State: ', state))
    .on('renamed', filePaths => console.log('File Renamed to: ', filePaths.fileName))
    .on('progress', stats => {
        const progress = stats.progress.toFixed(1);
        const speed = byteHelper(stats.speed);
        const downloaded = byteHelper(stats.downloaded);
        const total = byteHelper(stats.total);

        // print every one second (`progress.throttled` can be used instead)
        const currentTime = new Date();
        const elaspsedTime = currentTime - startTime;
        if (elaspsedTime > 1000) {
            startTime = currentTime;
            console.log(`${speed}/s - ${progress}% [${downloaded}/${total}]`);
        }
    });

console.log('Downloading: ', url);
//dl.pipe(zlib.createGzip()); // Adding example of pipe to compress the file while downloading
dl.start().catch(err => { /* already listening on 'error' event but catch can be used too */ });

[khkassem]

Hello

I tried with a fresh installed Electron 9 project and its working, there is something in my project! I tried on Electron 17 and its working. I'll try to figure it out on my project but it will be difficult, if you have any hint which can help me Thank you

[hgouveia]

@khkassem mm interesting, this library uses the native http and https modules, make sure to use them in the main process and communicated with it with ipc and not in the renderer process unless you have nodeIntegration: true , but if that is not the case, i wonder if you are using any other library that might be modifying the default behavior of http or https modules ? you could try to disabled those if that the case, for example node-fetch could be one? but i am not completely sure

[khkassem]

Hello

I just did a test by removing all my code from my main.js of electron, and added only code from the clean electron project, and it still not working (I didn't change anything in package or npm install) but the new main.js is loading nothing but the the node-downloader and electron stuff.

for me, electron must load nothing if its not imported or required, npm install itself can change things in electron?

I am not using node-fetch, just https-proxy-agent and axios wich can touch the https module.

Thank you for helping

[hgouveia]

Hello @khkassem , moved the conversation into the discussion #85 , lets continue there, i will close this for the moment since this issue seems to be related more about electron itself, but lets try to figure out there what we could do