schickling / chromeless

🖥 Chrome automation made simple. Runs locally or headless on AWS Lambda.
https://chromeless.netlify.com
MIT License
13.25k stars 575 forks source link

--proxy-server setting on local Chromeless does not work for authenticated proxies #457

Open wflanagan opened 6 years ago

wflanagan commented 6 years ago

When I attempt to use a proxy server on the local configuration, and that proxy has authentication, Chromeless returns an empty body without an error.

I'm using the most recent Chromeless as of 14 July 2018. Here is my script.

I've obviously removed my private proxies usernames, passwords, and host names. If you message me 1:1 I'm happy to give them to you for testing purposes.

FYI, my proxies are provided by a commercial service where I pay for usage. I'm highly confident they are configured correctly. They have run millions of URLs without problems and continue to do so in our production environment.

const chromeLauncher = require('chrome-launcher');
const Chromeless = require('chromeless').Chromeless;

if (process.env.NODE_ENV !== 'production') {
  require('dotenv').config()
}
const argv = require('minimist')(process.argv.slice(2));
const url = argv['u'];

async function run(url) {
  const chrome = await chromeLauncher.launch({
    port: 9222,
    chromeFlags: [
      '--disable-gpu',
      '--headless',
      //'--proxy-server=http://username:password@domain_name.com:80' // doesn't work (with DNS resolution to get IP)
      //'--proxy-server=http://username:password@1.2.3.4:80' // doesn't work (checking DNS resolution as problem)
      //'--proxy-server=http://201.49.66.97:92' // works (get one from https://www.proxynova.com/proxy-server-list/)
    ]
  });

  console.log('Chrome debuggable on port: ' + chrome.port);
  const chromeless = new Chromeless({
    launchChrome: false,
  })

  const html = await chromeless
    .goto(url)
    .wait(15000) // set arbitrarily long to ensure that this isn't a thing that its not finished rendering
    .html()

  const response = { html: html, url: url }
  console.log(JSON.stringify(response))
  await chromeless.end()
}

run(url).catch(console.error.bind(console))

Ideally, I can do this using Lambda as well. But, given its lack of "on demand changing proxy servers", I have opted to run a local Chrome for now.

Chrome Version: Version 67.0.3396.99 (Official Build) (64-bit) Node Version: v10.5.0 Development Environment: MacOS High Sierra 10.13.5 (MBRt 16gb RAM)

wflanagan commented 6 years ago

As an update to this, this isn't Chromeless--it's Chrome itself that's the problem.

See https://superuser.com/questions/902620/google-chrome-proxy-settings-with-username-and-password for more information.