Proxy

[Nullum-CC]

I am having issues with the following line;

options.add_argument(f'--proxy-server=https://X:X')

I need to add my authentication in the proxy. Proxy is IP:PORT:USER:PASS. Any explanation how to fill that in?

Thanks!

[steathy]

to use proxy with auth in selenium chromedrive needs some work either use selenium-wire or load an extension with chrome driver

[chemeng]

There are a couple of ways you can do it. One is to use a plugin (see code below). The other one is to do IP based authentication (if you can) in which case you don't need the user/pass.

def set_authenticated_proxy_through_plugin(proxy):
    '''
    Creates plugin for proxy auth
    '''
    pluginfile = 'proxy_auth_plugin_{}.zip'.format(proxy_to_url(proxy))
    pluginfile = os.path.join(PLUGIN_DIR, pluginfile)
    if glob.glob(pluginfile):
        return pluginfile
    manifest_json = """
                        {
                            "version": "1.0.0",
                            "manifest_version": 2,
                            "name": "Chrome Proxy",
                            "permissions": [
                                "proxy",
                                "tabs",
                                "unlimitedStorage",
                                "storage",
                                "<all_urls>",
                                "webRequest",
                                "webRequestBlocking"
                            ],
                            "background": {
                                "scripts": ["background.js"]
                            },
                            "minimum_chrome_version":"22.0.0"
                        }
                        """
    background_js = """
                        var config = {
                                mode: "fixed_servers",
                                rules: {
                                singleProxy: {
                                    scheme: "http",
                                    host: "%s",
                                    port: parseInt(%s)
                                },
                                bypassList: ["localhost"]
                                }
                            };

                        chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                        function callbackFn(details) {
                            return {
                                authCredentials: {
                                    username: "%s",
                                    password: "%s"
                                }
                            };
                        }

                        chrome.webRequest.onAuthRequired.addListener(
                                    callbackFn,
                                    {urls: ["<all_urls>"]},
                                    ['blocking']
                        );
                        """ % (proxy['proxy_address'], str(proxy['ports']['http']), proxy['username'], proxy['password'])
    with zipfile.ZipFile(pluginfile, 'w') as zp:
        zp.writestr("manifest.json", manifest_json)
        zp.writestr("background.js", background_js)
    return pluginfile

[JAManfredi]

This only works for headful right?

[chemeng]

Right, I combined this with Pyvirtualdisplay so make it "headless". Very painful but it worked..IP authentication was much easier

[ultrafunkamsterdam]

the correct url format would be options.add_argument(f'--proxy-server=https://{user}:{pass}@{host}:{port}')

however, sending plaintext password is never a good idea.

[KhushC-03]

There are a couple of ways you can do it. One is to use a plugin (see code below). The other one is to do IP based authentication (if you can) in which case you don't need the user/pass.

How did you load this extension in selenium using ucdriver?

[Avnsx]

@JAManfredi @Nullum-CC @chemeng @KhushC-03 I do strongly recommend y'all to just go with the easiest option, which is undetected-chromdriver together with selenium-wire. Here's an example of how easy it is: https://github.com/wkeeling/selenium-wire/issues/256#issue-849660821 (don't mind the issue, that got fixed by now so the code snippet works) For the other proxy types it's the same and it also supports auth proxies and headless mode since there's no extensions. Here's the docs just in case: https://github.com/wkeeling/selenium-wire#proxies The only ongoing issue with it, is the missing support for ipv6 proxies.

Edit 2022: The code snippet from before doesn't work anymore now you've to do it like this:

from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)

[ghost]

This is closed now but neither passing it as options or using the plugin works on V2 for me

[taytanqbpass]

anybody got a workaround for this? both plugin and seleniumwire didnt work for me.

[Avnsx]

anybody got a workaround for this? both plugin and seleniumwire didnt work for me.

are you sure this doesn't work?

import seleniumwire.undetected_chromedriver as uc
from seleniumwire import webdriver

chrome_options = webdriver.ChromeOptions()
chrome_options.headless = False
seleniumwire_options = {'proxy': {'https': 'type://host:port',}} 
driver = uc.Chrome(options=chrome_options, seleniumwire_options=seleniumwire_options)

driver.get('https://api.myip.com/')

[taytanqbpass]

@Avnsx sorry, i mean authenticated proxies.

[Avnsx]

@Avnsx sorry, i mean authenticated proxies.

Did you try seleniumwire_options = {'proxy': {'https': 'type://username:password@host:port',}}?

for type https, socks4 or socks5

Also have you tried to do a loop on multiple proxies, to see if any of them worked? If I remember correctly some proxies seleniumwire did not allow me to connect to, while I could connect with requests.

[taytanqbpass]

i had to use undetected_chromedriver.v2 here (for google login) so the seleniumwire_options option is not really possible. are there other ways for this?

[Avnsx]

i had to use undetected_chromedriver.v2 here (for google login) so the seleniumwire_options option is not really possible. are there other ways for this?

Not sure, I didn't ever try using proxies with v2. But for me that code snippet I sent works for v1.

[taytanqbpass]

still no luck with v2 here, anybody got any ideas? tried the zip plugin, it didn't work on my side.

[caerry]

f.e:

proxy = dict(
        proxy_address="127.0.0.1",
        port=5000,
        username="admin",
        password="admin"
)
proxy_plugin_name = set_authenticated_proxy_through_plugin(proxy)
options = uc.ChromeOptions()
options.add_extension(proxy_plugin_name)
chrome = uc.Chrome(chrome_options=options)

[caerry]

Installed undetected-chromedriver 2.2.1

Try it yourself

[muhy06]

Installed undetected-chromedriver 2.2.1

Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

[caerry]

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

[muhy06]

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

that would be great help, been trying it for last couple of days now.

https://drive.google.com/file/d/1NKTg0uXqJ7eEVo7F8ke2suNby_FrMrwL/view?usp=sharing

[muhy06]

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

Using the same method on selenium and it works, but not on undetected-chromedriver. Plugins does not load up.

[caerry]

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

that would be great help, been trying it for last couple of days now.

https://drive.google.com/file/d/1NKTg0uXqJ7eEVo7F8ke2suNby_FrMrwL/view?usp=sharing

you use options, not chrome_options

Read carefully what I wrote earlier, please.

[muhy06]

it worked, but it opened up two browsers simultaneously. Second one has plugin but i guess it is not very much like undetected-chromedriver.

[caerry]

it worked, but it opened up two browsers simultaneously. Second one has plugin but i guess it is not very much like undetected-chromedriver.

solved?

[caerry]

here's the solution.

selenium-wire           4.5.6
selenium                4.1.0
undetected-chromedriver 2.2.1

from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)

[woodcowper]

Yo! is that possible to handle proxy auth with latest version of undetected-chromedriver 3.1.3 ?

[Ahmad-Ishaq]

is there a way we can add a chrome profile because using this snippet of code doesn't allows us to work with oprions to add profile !!!

here's the solution.

selenium-wire           4.5.6
selenium                4.1.0
undetected-chromedriver 2.2.1

from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)

[rezonasyu]

options=chrome_options, seleniumwire_options=seleniumwire_options

This works definately... But cant use it with google login. https certificate error.

What could be the work around to work with V2?

[realdronos]

nvm

[moein99]

@rezonasyu forget about selenium wire (because it still has a problem with HTTPS certificate) and use undetected chrome with this Then you will be able to use proxy with credentials + Gmail login.

[SolimanAhmed25]

@muhy06 @Avnsx

Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/"

The code:

import zipfile import undetected_chromedriver.v2 as uc

def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000'

options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

[basvdl97]

@muhy06 @Avnsx

Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/"

The code:

import zipfile import undetected_chromedriver.v2 as uc

def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000'

options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

You need to downgrade ther undetected chrome version: pip install undetected-chromedriver==2.2.1 And I also ended up using webdriver.ChromeOptions(), then u should be fine.

[muhy06]

@muhy06 @Avnsx Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/" The code: import zipfile import undetected_chromedriver.v2 as uc def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000' options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

You need to downgrade ther undetected chrome version: pip install undetected-chromedriver==2.2.1 And I also ended up using webdriver.ChromeOptions(), then u should be fine.

Hi, I finally found the solution. Im using windows 10. Not sure if other OS have the same issue. Basically, zip file for pluginfile is not working instead create a folder "pluginfile" and place two files: manifest.json and background.js. then use following code snippet.:

`

    import undetected_chromedriver as uc

    opts = uc.ChromeOptions()
    pluginfile = set_authenticated_proxy_through_plugin(proxies)
    opts.add_argument(f'--proxy-server=http://{proxies["proxy"]}:{proxies["port"]}')
    opts.add_argument('--load-extension={}'.format(pluginfile))
    driver = uc.Chrome(options=opts,use_subprocess=True)

`

pluginfile is the folder path where you have saved manifest.json and background.js.

Note if you use selenium-wire for undetected, it is no longer fully undetected. I have tested it.

[SafeerAbbas624]

@muhy06 @Avnsx Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/" The code: import zipfile import undetected_chromedriver.v2 as uc def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000' options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

You need to downgrade ther undetected chrome version: pip install undetected-chromedriver==2.2.1 And I also ended up using webdriver.ChromeOptions(), then u should be fine.

Hi, I finally found the solution. Im using windows 10. Not sure if other OS have the same issue. Basically, zip file for pluginfile is not working instead create a folder "pluginfile" and place two files: manifest.json and background.js. then use following code snippet.:

`

    import undetected_chromedriver as uc

    opts = uc.ChromeOptions()
    pluginfile = set_authenticated_proxy_through_plugin(proxies)
    opts.add_argument(f'--proxy-server=http://{proxies["proxy"]}:{proxies["port"]}')
    opts.add_argument('--load-extension={}'.format(pluginfile))
    driver = uc.Chrome(options=opts,use_subprocess=True)

`

pluginfile is the folder path where you have saved manifest.json and background.js.

Note if you use selenium-wire for undetected, it is no longer fully undetected. I have tested it.

Yeesssss its working. Do not make zip plugin or extension. make folder and it will work. even I didn't downgraded any library. all are the latest. thanks mate