ultrafunkamsterdam / undetected-chromedriver

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
https://github.com/UltrafunkAmsterdam/undetected-chromedriver
GNU General Public License v3.0
9.89k stars 1.16k forks source link

Proxy #82

Closed Nullum-CC closed 3 years ago

Nullum-CC commented 3 years ago

I am having issues with the following line;

options.add_argument(f'--proxy-server=https://X:X')

I need to add my authentication in the proxy. Proxy is IP:PORT:USER:PASS. Any explanation how to fill that in?

Thanks!

steathy commented 3 years ago

to use proxy with auth in selenium chromedrive needs some work either use selenium-wire or load an extension with chrome driver

chemeng commented 3 years ago

There are a couple of ways you can do it. One is to use a plugin (see code below). The other one is to do IP based authentication (if you can) in which case you don't need the user/pass.

def set_authenticated_proxy_through_plugin(proxy):
    '''
    Creates plugin for proxy auth
    '''
    pluginfile = 'proxy_auth_plugin_{}.zip'.format(proxy_to_url(proxy))
    pluginfile = os.path.join(PLUGIN_DIR, pluginfile)
    if glob.glob(pluginfile):
        return pluginfile
    manifest_json = """
                        {
                            "version": "1.0.0",
                            "manifest_version": 2,
                            "name": "Chrome Proxy",
                            "permissions": [
                                "proxy",
                                "tabs",
                                "unlimitedStorage",
                                "storage",
                                "<all_urls>",
                                "webRequest",
                                "webRequestBlocking"
                            ],
                            "background": {
                                "scripts": ["background.js"]
                            },
                            "minimum_chrome_version":"22.0.0"
                        }
                        """
    background_js = """
                        var config = {
                                mode: "fixed_servers",
                                rules: {
                                singleProxy: {
                                    scheme: "http",
                                    host: "%s",
                                    port: parseInt(%s)
                                },
                                bypassList: ["localhost"]
                                }
                            };

                        chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                        function callbackFn(details) {
                            return {
                                authCredentials: {
                                    username: "%s",
                                    password: "%s"
                                }
                            };
                        }

                        chrome.webRequest.onAuthRequired.addListener(
                                    callbackFn,
                                    {urls: ["<all_urls>"]},
                                    ['blocking']
                        );
                        """ % (proxy['proxy_address'], str(proxy['ports']['http']), proxy['username'], proxy['password'])
    with zipfile.ZipFile(pluginfile, 'w') as zp:
        zp.writestr("manifest.json", manifest_json)
        zp.writestr("background.js", background_js)
    return pluginfile
JAManfredi commented 3 years ago

This only works for headful right?

chemeng commented 3 years ago

Right, I combined this with Pyvirtualdisplay so make it "headless". Very painful but it worked..IP authentication was much easier

ultrafunkamsterdam commented 3 years ago

the correct url format would be options.add_argument(f'--proxy-server=https://{user}:{pass}@{host}:{port}')

however, sending plaintext password is never a good idea.

KhushC-03 commented 3 years ago

There are a couple of ways you can do it. One is to use a plugin (see code below). The other one is to do IP based authentication (if you can) in which case you don't need the user/pass.

How did you load this extension in selenium using ucdriver?

Avnsx commented 3 years ago

@JAManfredi @Nullum-CC @chemeng @KhushC-03 I do strongly recommend y'all to just go with the easiest option, which is undetected-chromdriver together with selenium-wire. Here's an example of how easy it is: https://github.com/wkeeling/selenium-wire/issues/256#issue-849660821 (don't mind the issue, that got fixed by now so the code snippet works) For the other proxy types it's the same and it also supports auth proxies and headless mode since there's no extensions. Here's the docs just in case: https://github.com/wkeeling/selenium-wire#proxies The only ongoing issue with it, is the missing support for ipv6 proxies.

Edit 2022: The code snippet from before doesn't work anymore now you've to do it like this:

from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)
ghost commented 3 years ago

This is closed now but neither passing it as options or using the plugin works on V2 for me

taytanqbpass commented 2 years ago

anybody got a workaround for this? both plugin and seleniumwire didnt work for me.

Avnsx commented 2 years ago

anybody got a workaround for this? both plugin and seleniumwire didnt work for me.

are you sure this doesn't work?

import seleniumwire.undetected_chromedriver as uc
from seleniumwire import webdriver

chrome_options = webdriver.ChromeOptions()
chrome_options.headless = False
seleniumwire_options = {'proxy': {'https': 'type://host:port',}} 
driver = uc.Chrome(options=chrome_options, seleniumwire_options=seleniumwire_options)

driver.get('https://api.myip.com/')
taytanqbpass commented 2 years ago

@Avnsx sorry, i mean authenticated proxies.

Avnsx commented 2 years ago

@Avnsx sorry, i mean authenticated proxies.

Did you try seleniumwire_options = {'proxy': {'https': 'type://username:password@host:port',}}?

for type https, socks4 or socks5

Also have you tried to do a loop on multiple proxies, to see if any of them worked? If I remember correctly some proxies seleniumwire did not allow me to connect to, while I could connect with requests.

taytanqbpass commented 2 years ago

i had to use undetected_chromedriver.v2 here (for google login) so the seleniumwire_options option is not really possible. are there other ways for this?

Avnsx commented 2 years ago

i had to use undetected_chromedriver.v2 here (for google login) so the seleniumwire_options option is not really possible. are there other ways for this?

Not sure, I didn't ever try using proxies with v2. But for me that code snippet I sent works for v1.

taytanqbpass commented 2 years ago

still no luck with v2 here, anybody got any ideas? tried the zip plugin, it didn't work on my side.

caerry commented 2 years ago
image

f.e:

proxy = dict(
        proxy_address="127.0.0.1",
        port=5000,
        username="admin",
        password="admin"
)
proxy_plugin_name = set_authenticated_proxy_through_plugin(proxy)
options = uc.ChromeOptions()
options.add_extension(proxy_plugin_name)
chrome = uc.Chrome(chrome_options=options)
caerry commented 2 years ago

Installed undetected-chromedriver 2.2.1

Try it yourself

muhy06 commented 2 years ago

Installed undetected-chromedriver 2.2.1

Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

caerry commented 2 years ago

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

muhy06 commented 2 years ago

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

that would be great help, been trying it for last couple of days now.

https://drive.google.com/file/d/1NKTg0uXqJ7eEVo7F8ke2suNby_FrMrwL/view?usp=sharing

muhy06 commented 2 years ago

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

Using the same method on selenium and it works, but not on undetected-chromedriver. Plugins does not load up.

caerry commented 2 years ago

Installed undetected-chromedriver 2.2.1 Try it yourself

No, not working, it wont add plugin. for some reason, have same undetected-chromedriver version.

give me a minimally reproducible example of what you are doing

that would be great help, been trying it for last couple of days now.

https://drive.google.com/file/d/1NKTg0uXqJ7eEVo7F8ke2suNby_FrMrwL/view?usp=sharing

you use options, not chrome_options

Read carefully what I wrote earlier, please.

muhy06 commented 2 years ago

it worked, but it opened up two browsers simultaneously. Second one has plugin but i guess it is not very much like undetected-chromedriver.

WhatsApp Image 2021-11-26 at 20 34 35

caerry commented 2 years ago

it worked, but it opened up two browsers simultaneously. Second one has plugin but i guess it is not very much like undetected-chromedriver.

WhatsApp Image 2021-11-26 at 20 34 35

solved?

caerry commented 2 years ago

here's the solution.

selenium-wire           4.5.6
selenium                4.1.0
undetected-chromedriver 2.2.1
from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)
woodcowper commented 2 years ago

Yo! is that possible to handle proxy auth with latest version of undetected-chromedriver 3.1.3 ?

Ahmad-Ishaq commented 2 years ago

is there a way we can add a chrome profile because using this snippet of code doesn't allows us to work with oprions to add profile !!!

here's the solution.

selenium-wire           4.5.6
selenium                4.1.0
undetected-chromedriver 2.2.1
from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)
rezonasyu commented 2 years ago
options=chrome_options, seleniumwire_options=seleniumwire_options

This works definately... But cant use it with google login. https certificate error.

What could be the work around to work with V2?

realdronos commented 2 years ago

nvm

moein99 commented 2 years ago

@rezonasyu forget about selenium wire (because it still has a problem with HTTPS certificate) and use undetected chrome with this Then you will be able to use proxy with credentials + Gmail login.

SolimanAhmed25 commented 2 years ago

@muhy06 @Avnsx

Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/"

The code:

import zipfile import undetected_chromedriver.v2 as uc

def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000'

options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

basvdl97 commented 1 year ago

@muhy06 @Avnsx

Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/"

The code:

import zipfile import undetected_chromedriver.v2 as uc

def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000'

options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

You need to downgrade ther undetected chrome version: pip install undetected-chromedriver==2.2.1 And I also ended up using webdriver.ChromeOptions(), then u should be fine.

muhy06 commented 1 year ago

@muhy06 @Avnsx Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/" The code: import zipfile import undetected_chromedriver.v2 as uc def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000' options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

You need to downgrade ther undetected chrome version: pip install undetected-chromedriver==2.2.1 And I also ended up using webdriver.ChromeOptions(), then u should be fine.

Hi, I finally found the solution. Im using windows 10. Not sure if other OS have the same issue. Basically, zip file for pluginfile is not working instead create a folder "pluginfile" and place two files: manifest.json and background.js. then use following code snippet.:

`

    import undetected_chromedriver as uc

    opts = uc.ChromeOptions()
    pluginfile = set_authenticated_proxy_through_plugin(proxies)
    opts.add_argument(f'--proxy-server=http://{proxies["proxy"]}:{proxies["port"]}')
    opts.add_argument('--load-extension={}'.format(pluginfile))
    driver = uc.Chrome(options=opts,use_subprocess=True)

`

pluginfile is the folder path where you have saved manifest.json and background.js.

Note if you use selenium-wire for undetected, it is no longer fully undetected. I have tested it.

SafeerAbbas624 commented 1 year ago

@muhy06 @Avnsx Guys I tried this code, but it did not work. can some one help me. I am using proxies form "Smart Proxy" this is there link "https://smartproxy.com/" The code: import zipfile import undetected_chromedriver.v2 as uc def set_authenticated_proxy_through_plugin(endpoint, port, username, password): ''' Creates plugin for proxy auth '''

manifest_json = """
                    {
                        "version": "1.0.0",
                        "manifest_version": 2,
                        "name": "Chrome Proxy",
                        "permissions": [
                            "proxy",
                            "tabs",
                            "unlimitedStorage",
                            "storage",
                            "<all_urls>",
                            "webRequest",
                            "webRequestBlocking"
                        ],
                        "background": {
                            "scripts": ["background.js"]
                        },
                        "minimum_chrome_version":"22.0.0"
                    }
                    """
background_js = """
                    var config = {
                            mode: "fixed_servers",
                            rules: {
                            singleProxy: {
                                scheme: "http",
                                host: "%s",
                                port: parseInt(%s)
                            },
                            bypassList: ["localhost"]
                            }
                        };

                    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

                    function callbackFn(details) {
                        return {
                            authCredentials: {
                                username: "%s",
                                password: "%s"
                            }
                        };
                    }

                    chrome.webRequest.onAuthRequired.addListener(
                                callbackFn,
                                {urls: ["<all_urls>"]},
                                ['blocking']
                    );
                    """ % (endpoint, port, username, password)

pluginfile = 'proxies_extension.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
return pluginfile

username = 'user' password = 'pass' endpoint = 'gate.dc.smartproxy.com' port = '20000' options = uc.ChromeOptions() proxies_extension = set_authenticated_proxy_through_plugin(username, password, endpoint, port) Browser = uc.Chrome(chrome_options=options) Browser.get('https://www.myexternalip.com/raw')

You need to downgrade ther undetected chrome version: pip install undetected-chromedriver==2.2.1 And I also ended up using webdriver.ChromeOptions(), then u should be fine.

Hi, I finally found the solution. Im using windows 10. Not sure if other OS have the same issue. Basically, zip file for pluginfile is not working instead create a folder "pluginfile" and place two files: manifest.json and background.js. then use following code snippet.:

`

    import undetected_chromedriver as uc

    opts = uc.ChromeOptions()
    pluginfile = set_authenticated_proxy_through_plugin(proxies)
    opts.add_argument(f'--proxy-server=http://{proxies["proxy"]}:{proxies["port"]}')
    opts.add_argument('--load-extension={}'.format(pluginfile))
    driver = uc.Chrome(options=opts,use_subprocess=True)

`

pluginfile is the folder path where you have saved manifest.json and background.js.

Note if you use selenium-wire for undetected, it is no longer fully undetected. I have tested it.

Yeesssss its working. Do not make zip plugin or extension. make folder and it will work. even I didn't downgraded any library. all are the latest. thanks mate