search

kaliiiiiiiiii / Selenium-Profiles

undetected Selenium using chromedriver and emulation / device profiles
Other
255 stars 29 forks source link

Getting detected by cloudfare #55

Closed desis123 closed 11 months ago

desis123 commented 1 year ago

First of all thanks for this nice profile based selenium package. The purpose of undetected Chrome-driver is to bypass bot detection, While I am using latest version of pure undetected chrome-driver which is version 4.4.7 , it is able to visit https://nowsecure.nl , that means it can bypass cloudflare . but while I am using selenium profiles with uc_driver=True, it can not bypass https://nowsecure.nl , thats why I think selenium profiles needs to include new version undetected chrome driver.

I am using following code 

from selenium_profiles.webdriver import Chrome
from selenium_profiles.profiles import profiles
from selenium.webdriver.common.by import By  # locate elements
from selenium.webdriver import ChromeOptions

profile = profiles.Windows()
options = ChromeOptions()
mydriver = Chrome(profile, options=options, uc_driver=True)

driver = mydriver.start()  # or .Android

driver.get('https://nowsecure.nl')  # test fingerprint

input("Press ENTER to exit: ")
driver.quit()  # Execute on the End!

But with following code, I mean with pure undetected chromedriver 

import undetected_chromedriver as uc
import time
driver = uc.Chrome()
driver.get('https://nowsecure.nl')
input("Press ENTER to exit: ")

Its able to bypass cloudflare

kaliiiiiiiiii commented 1 year ago

I think selenium profiles needs to include new version undetected chrome driver.

Selenium-Profiles always uses the version of undetected-chromedriver, which is installed on your env. => same version

UPDATE:

Installation with pip install selenium-driverless

from selenium_driverless import webdriver
import asyncio

async def main():
    options = webdriver.Options()
    async with webdriver.Chrome(options=options) as driver:
        await driver.get('http://nowsecure.nl#relax')
        await driver.implicitly_wait(3)

        title = await driver.title
        url = await driver.current_url
        source = await driver.page_source
        print(title)

asyncio.run(main())

synchronous version is available as well (documentation)

kaliiiiiiiiii commented 1 year ago

Looks to me like cloudfare upgraded their bot-protection:)

found the following scripts in service workers which might be related

(function() {
    var workerData = {
        p: navigator.platform,
        l: navigator.languages,
        hwC: navigator.hardwareConcurrency,
        dM: navigator.deviceMemory,
        wd: navigator.webdriver,
        uA: navigator.userAgent
    };
    postMessage(workerData);
}
)()
onmessage = function(e) {var gsb = function(){
    var sizeA = 0;
    var sizeB = 0;
    var counter = 0;
    try {
        var fn_1 = function () {
            counter += 1;
            fn_1();
        };
        fn_1();
    }
    catch (_a) {
        sizeA = counter;
        try {
            counter = 0;
            var fn_2 = function () {
                var local = 1;
                counter += local;
                fn_2();
            };
            fn_2();
        }
        catch (_b) {
            sizeB = counter;
        }
    }
    var bytes = (sizeB * 8) / (sizeA - sizeB);
    return [sizeA, sizeB, bytes];
}; postMessage({a3: gsb()})};
function getTimingResolution() {
    var runs = 5000;
    var valA = 1;
    var valB = 1;
    var res;
    for (var i = 0; i < runs; i++) {
        var a = performance.now();
        var b = performance.now();
        if (a < b) {
            res = b - a;
            if (res > valA && res < valB) {
                valB = res;
            } else if (res < valA) {
                valB = valA;
                valA = res;
            }
        }
    }
    return valA;
};

onmessage = function(){ postMessage({ a: getTimingResolution() })}
self.onmessage = function(m) {
    self.postMessage({
        o: true
    });
    eval("debugger");
    self.postMessage({
        o: false
    })
}

that basically exposes emulation, since those values don't get emulated in workers. Currently, there's no way to fix that.

kaliiiiiiiiii commented 1 year ago

@desis123

  1. what platform are you running on?
  2. What version of Chrome do you have (chrome://version as url) ?
desis123 commented 1 year ago

@desis123

  1. what platform are you running on?
  2. What version of Chrome do you have (chrome://version as url) ?

Using Ubuntu 22.04 Chrome 114 , Latest version of Undetected Chromedriver which is 4.5.7 only works with chrome 114 , With chrome 113 , latest version does not work and To work with chrome 113 if you downgrade the undetected Chromedriver , then that standalone UC can not bypass cloudflare .

kaliiiiiiiiii commented 1 year ago

Using Ubuntu 22.04 Chrome 114 , Latest version of Undetected Chromedriver which is 4.5.7 only works with chrome 114 , With chrome 113 , latest version does not work and To work with chrome 113 if you downgrade the undetected Chromedriver , then that standalone UC can not bypass cloudflare . That's good to know.

  1. Can you try if profile={} passes? In that case, emulation gets detected.
  2. Couldn't find out what changed at undetected-chromedriver. Any ideas//commits found there?

Note: I think you meant undetected-chromedriver==3.4.7

desis123 commented 1 year ago

Yes 3.4.7 .. my bad , Yes with profile={} , it can bypass cloudflare

desis123 commented 1 year ago

Latest Version of Undetected Chrome-driver only works with chrome 114, it does not work with chrome 113 or any other version. and Only latest version Undetected Chrome-driver can bypass cloud flare.

kaliiiiiiiiii commented 1 year ago
  1. Couldn't find out what changed at undetected-chromedriver. Any ideas//commits found there?

actually, maybe its chrome==114 it's "fault". did you try on other versions?

I meant if selenium-profiles works with other chrome versions

desis123 commented 1 year ago

Yes selenium profiles works with other chrome versions , but that does not bypass cloudflare

desis123 commented 1 year ago

Just a temporary fixed , If anyone like to use Undetected Chromedriver with Mobile Emulation 

 set_device_metrics_override = dict({
              "width": 375,
              "height": 812,
              "deviceScaleFactor": 50,
              "mobile": True
          })
 browser.execute_cdp_cmd('Emulation.setDeviceMetricsOverride', set_device_metrics_override)
kaliiiiiiiiii commented 1 year ago

Just a temporary fixed , If anyone like to use Undetected Chromedriver with Mobile Emulation

 set_device_metrics_override = dict({
              "width": 375,
              "height": 812,
              "deviceScaleFactor": 50,
              "mobile": True
          })
 browser.execute_cdp_cmd('Emulation.setDeviceMetricsOverride', set_device_metrics_override)

Althought that might make it look like you're using emulation, this is basically the same thing as Selenium-Profiles does. Only difference: In your script, for example the platform isn't going to be emulated.

kaliiiiiiiiii commented 11 months ago

This is bypassable by starting chrome without chromedriver.

Here's a short POC using chromewhip (using chrome-developer-protocol only):

import asyncio
import logging
import subprocess

from chromewhip import Chrome
from chromewhip.protocol import browser, page

# see logging from chromewhip
logging.basicConfig(level=logging.DEBUG)

HOST = '127.0.0.1'
PORT = 9222
subprocess.Popen(["C:/Program Files/Google/Chrome/Application/chrome.exe", f"--remote-debugging-port={PORT}"])

loop = asyncio.get_event_loop()
c = Chrome(host=HOST, port=PORT)

loop.run_until_complete(c.connect())
tab = c.tabs[0]
loop.run_until_complete(tab.enable_page_events())

def sync_cmd(*args, **kwargs):
    return loop.run_until_complete(tab.send_command(*args, **kwargs))

result = sync_cmd(page.Page.navigate(url='http://nowsecure.nl#relax'),
                  await_on_event_type=page.FrameStartedLoadingEvent)

sync_cmd(browser.Browser.close())

unfortanely, this library seems deprecated and would need to be re-written (issue)

seems like @ultrafunkamsterdam is allready working on something like that. (resource)

@desis123 @xAffan might consider creating a selenium-driverless package, but not sure yet. Question would be how far @ultrafunkamsterdam allready is?

xAffan commented 11 months ago

Wow. This is amazing progress. @ultrafunkamsterdam Please update us about driverless UC.

On Fri, 21 Jul 2023, 1:24 pm kaliiiiiiiiiii, @.***> wrote:

This is bypassable by starting chrome without chromedriver.

Here's a short POC using chromewhip https://github.com/chazkii/chromewhip (using chrome-developer-protocol only):

import asyncioimport loggingimport subprocess from chromewhip import Chromefrom chromewhip.protocol import browser, page

see logging from chromewhiplogging.basicConfig(level=logging.DEBUG)

HOST = '127.0.0.1'PORT = 9222subprocess.Popen(["C:/Program Files/Google/Chrome/Application/chrome.exe", f"--remote-debugging-port={PORT}"]) loop = asyncio.get_event_loop()c = Chrome(host=HOST, port=PORT) loop.run_until_complete(c.connect())tab = c.tabs[0]loop.run_until_complete(tab.enable_page_events())

def sync_cmd(*args, *kwargs): return loop.run_until_complete(tab.send_command(args, **kwargs))

result = sync_cmd(page.Page.navigate(url='http://nowsecure.nl#relax'), await_on_event_type=page.FrameStartedLoadingEvent) sync_cmd(browser.Browser.close())

unfortanely, this library seems deprecated and would need to be re-written

seems like @ultrafunkamsterdam https://github.com/ultrafunkamsterdam is allready working on something like that. (resource https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1321#issuecomment-1582143871 )

@desis123 https://github.com/desis123 @xAffan https://github.com/xAffan might consider creating a selenium-driverless package, but not sure yet. Question would be how far @ultrafunkamsterdam https://github.com/ultrafunkamsterdam allready is

— Reply to this email directly, view it on GitHub https://github.com/kaliiiiiiiiii/Selenium-Profiles/issues/55#issuecomment-1645201698, or unsubscribe https://github.com/notifications/unsubscribe-auth/AODGAV2U3ZQCZPIUQ2B56PTXRI4CZANCNFSM6AAAAAAY5GZOGE . You are receiving this because you were mentioned.Message ID: @.***>

kaliiiiiiiiii commented 11 months ago

unfortanely, this library seems deprecated and would need to be re-written (issue)

Maybe, I could patch the non-IO part easely with PyCDP

Update: working on at https://github.com/kaliiiiiiiiii/Selenium-Driverless

xAffan commented 11 months ago

though your repository is private. someone has forked https://github.com/HMaker/python-cdp not sure if its useful

kaliiiiiiiiii commented 11 months ago

though your repository is private. someone has forked https://github.com/HMaker/python-cdp not sure if its useful

yep forgot, now it should be public:)

NCLnclNCL commented 11 months ago

though your repository is private. someone has forked https://github.com/HMaker/python-cdp not sure if its useful

yep forgot, now it should be public:)

is there no other way to bypass it, I see playwright or puppeteer not using the driver but it is also detected

kaliiiiiiiiii commented 11 months ago

is there no other way to bypass it, I see playwright or puppeteer not using the driver but it is also detected

huh tbh, not sure why playwright, with some modifications shouldnt work

xAffan commented 11 months ago

@NCLnclNCL

there are some solution linked here which already works (driverless-selenium) so please no spam. If you have any suggestions please give.

kaliiiiiiiiii commented 11 months ago

with selenium-driverless

(requires selenium-profiles>=2.2.8) warning:

profile = profiles.Windows() # or .Android options = ChromeOptions()

options.add_argument("--headless=new")

driver = Chrome(profile, options=options, driverless_options=True)

get url

driver.get('https://nowsecure.nl#relax') # test fingerprint

driver.quit() # Execute on the End!


see [documentation](https://github.com/kaliiiiiiiiii/Selenium-Driverless) for usage

@desis123 feel free to test if that resolves this issue for you & then close it
kaliiiiiiiiii commented 11 months ago

Closing as this issue seems to be resolved