ultrafunkamsterdam / undetected-chromedriver

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
https://github.com/UltrafunkAmsterdam/undetected-chromedriver
GNU General Public License v3.0
9.08k stars 1.09k forks source link

Can't run multiple instances #1051

Open Mikayel0 opened 1 year ago

Mikayel0 commented 1 year ago

Can not run more than one browser

import undetected_chromedriver as uc
import os
from concurrent import futures

def browser(thread):
    uc.Chrome()
    print(thread)
    input()

def main():
    os.system('cls')
    threads = input("How much threads you want: ")
    with futures.ThreadPoolExecutor() as executor: 
        future_test_results = [ executor.submit(browser, i)  
        for i in range(int(threads)) ]
        for future_test_result in future_test_results: 
            try:        
                test_result = future_test_result.result()          
            except Exception as exc:
                print('thread generated an exception: {:0}'.format(exc))

if __name__ == "__main__":
    main()

When second browser is opening

FileExistsError: [WinError 183] Cannot create a file when that file already exists:
'C:\\Users\\User\\appdata\\roaming\\undetected_chromedriver\\undetected\\chromedriver.exe' ->
'C:\\Users\\User\\appdata\\roaming\\undetected_chromedriver\\undetected_chromedriver.exe'
Dragur611 commented 1 year ago

Have the same issue. I use this configuration: def init_driver(): chrome_options = uc.ChromeOptions() chrome_options.add_argument("--incognito") chrome_options.add_argument('--headless') chrome_options.add_argument("--log-level=3") chrome_options.add_argument("--start-maximized") driver = uc.Chrome(options=chrome_options, use_subprocess=True) driver.wait = WebDriverWait(driver, 2) return driver

driver = init_driver()

When i use only 1 browser is ok. But when i try to open another

driver_2_example = init_driver()

i get this exception:

  File "C:\Users\Dragur\AppData\Local\Programs\Python\Python39\lib\site-packages\undetected_chromedriver\__init__.py", line 374, in __init__
    if self.patcher.version_main < 108:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

This happens since V110 of Chrome I'm using

undetected-chromedriver   3.4.4
selenium                  4.8.0
Python 3.9.2
Windows 10
Dragur611 commented 1 year ago

Have the same issue. I use this configuration: def init_driver(): chrome_options = uc.ChromeOptions() chrome_options.add_argument("--incognito") chrome_options.add_argument('--headless') chrome_options.add_argument("--log-level=3") chrome_options.add_argument("--start-maximized") driver = uc.Chrome(options=chrome_options, use_subprocess=True) driver.wait = WebDriverWait(driver, 2) return driver

driver = init_driver()

When i use only 1 browser is ok. But when i try to open another

driver_2_example = init_driver()

i get this exception:

  File "C:\Users\Dragur\AppData\Local\Programs\Python\Python39\lib\site-packages\undetected_chromedriver\__init__.py", line 374, in __init__
    if self.patcher.version_main < 108:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

This happens since V110 of Chrome I'm using

undetected-chromedriver   3.4.4
selenium                  4.8.0
Python 3.9.2
Windows 10

(Yes i'm the same who posted the --headless issue) sorry to bother you so much, I really appreciate your job @ultrafunkamsterdam

ultrafunkamsterdam commented 1 year ago

i really shouldnt be doing this.... here we go:

pip install -U https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

please check if it is really installed, or just run this twice. report back appreciated..

Mikayel0 commented 1 year ago

When i tried to start few browsers they started well, after first time i was getting same exception

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process:
'C:\\Users\\User\\appdata\\roaming\\undetected_chromedriver\\undetected\\chromedriver.exe' ->
'C:\\Users\\User\\appdata\\roaming\\undetected_chromedriver\\undetected_chromedriver.exe'
Dragur611 commented 1 year ago

https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

404 error

ERROR: Could not install requirement https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance because of HTTP error 404 Client Error: Not Found for url: https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance for URL https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

labodj commented 1 year ago

pip install -U https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

404 error

ERROR: Could not install requirement https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance because of HTTP error 404 Client Error: Not Found for url: https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance for URL https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

please try

pip install -U git+https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

Dragur611 commented 1 year ago

pip install -U https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

404 error ERROR: Could not install requirement https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance because of HTTP error 404 Client Error: Not Found for url: https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance for URL https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

please try

pip install -U git+https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

Collecting git+https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance
  Cloning https://github.com/ultrafunkamsterdam/undetected-chromedriver (to revision fix-multiple-instance) to c:\users\dragur\appdata\local\temp\pip-req-build-mbcpnq1o
  Running command git clone --filter=blob:none --quiet https://github.com/ultrafunkamsterdam/undetected-chromedriver 'C:\Users\Dragur\AppData\Local\Temp\pip-req-build-mbcpnq1o'
  Running command git checkout -b fix-multiple-instance --track origin/fix-multiple-instance
  branch 'fix-multiple-instance' set up to track 'origin/fix-multiple-instance'.
  Switched to a new branch 'fix-multiple-instance'
  Resolved https://github.com/ultrafunkamsterdam/undetected-chromedriver to commit e146610b7c53beaecf31a31f907c06ef976d1ed5
  Preparing metadata (setup.py) ... done
Requirement already satisfied: selenium>=4.0.0 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from undetected-chromedriver==3.4.5) (4.8.0)
Requirement already satisfied: requests in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from undetected-chromedriver==3.4.5) (2.28.2)      
Requirement already satisfied: websockets in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from undetected-chromedriver==3.4.5) (10.1)      
Requirement already satisfied: trio-websocket~=0.9 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from selenium>=4.0.0->undetected-chromedriver==3.4.5) (0.9.2)   
Requirement already satisfied: trio~=0.17 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from selenium>=4.0.0->undetected-chromedriver==3.4.5) (0.19.0)
Requirement already satisfied: urllib3[socks]~=1.26 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from selenium>=4.0.0->undetected-chromedriver==3.4.5) (1.26.7) 
Requirement already satisfied: certifi>=2021.10.8 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from selenium>=4.0.0->undetected-chromedriver==3.4.5) (2022.12.7)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from requests->undetected-chromedriver==3.4.5) (2.0.9)     
Requirement already satisfied: idna<4,>=2.5 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from requests->undetected-chromedriver==3.4.5) (2.10)
Requirement already satisfied: cffi>=1.14 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from trio~=0.17->selenium>=4.0.0->undetected-chromedriver==3.4.5) (1.15.0)
Requirement already satisfied: outcome in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from trio~=0.17->selenium>=4.0.0->undetected-chromedriver==3.4.5) (1.1.0)
Requirement already satisfied: sniffio in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from trio~=0.17->selenium>=4.0.0->undetected-chromedriver==3.4.5) (1.2.0)
Requirement already satisfied: attrs>=19.2.0 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from trio~=0.17->selenium>=4.0.0->undetected-chromedriver==3.4.5) (21.2.0)
Requirement already satisfied: async-generator>=1.9 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from trio~=0.17->selenium>=4.0.0->undetected-chromedriver==3.4.5) (1.10)
Requirement already satisfied: sortedcontainers in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from trio~=0.17->selenium>=4.0.0->undetected-chromedriver==3.4.5) (2.4.0)
Requirement already satisfied: wsproto>=0.14 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from trio-websocket~=0.9->selenium>=4.0.0->undetected-chromedriver==3.4.5) (1.0.0)
Requirement already satisfied: PySocks!=1.5.7,<2.0,>=1.5.6 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from urllib3[socks]~=1.26->selenium>=4.0.0->undetected-chromedriver==3.4.5) (1.7.1)
Requirement already satisfied: pycparser in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from cffi>=1.14->trio~=0.17->selenium>=4.0.0->undetected-chromedriver==3.4.5) (2.21)
Requirement already satisfied: h11<1,>=0.9.0 in c:\users\dragur\appdata\local\programs\python\python39\lib\site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium>=4.0.0->undetected-chromedriver==3.4.5) (0.12.0)

I used the command 3 times. But i get same error File "C:\Users\Dragur\AppData\Local\Programs\Python\Python39\lib\site-packages\undetected_chromedriver\__init__.py", line 374, in __init__ if self.patcher.version_main < 108: TypeError: '<' not supported between instances of 'NoneType' and 'int'

Dragur611 commented 1 year ago

i really shouldnt be doing this.... here we go:

pip install -U https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

please check if it is really installed, or just run this twice. report back appreciated..

tried that, but get same error

tomslarkin commented 1 year ago

I have the same issue, any solution would be appreciated!

tomslarkin commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

Dragur611 commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

Worked for me! Thanks!

vsuyvkpzica46088 commented 1 year ago

do this: chrome_driver = uc.Chrome(version_main=CHROME_VERSION, options=chrome_options)

will fix the issue

labodj commented 1 year ago

do this: chrome_driver = uc.Chrome(version_main=CHROME_VERSION, options=chrome_options)

will fix the issue

No, it doesn't

Anyway... I just retested the code updated to 57a2bd3f436832ffe8b127efa1d358a64f32be6f and it seems to be working again using chrome 110.

You can try it like so

python -m pip uninstall undetected-chromedriver
python -m pip install git+https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance
python -m pip install --upgrade selenium
abd0014 commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

Thaaaaaaaank you. it's worked for me.

antooro commented 1 year ago

My two browsers are opening but I see this issue: Only the first one is making the requests and what i have programmed. The second one stays only in the initial chrome screen.

I'm using different profiles in each driver!!!! So i set --profile-directory and --user-data-dir

selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:51381

FIXED=

Split your profile data under diferent folders, dont use same one, one per eaceh

abd0014 commented 1 year ago

My two browsers are opening but I see this issue: Only the first one is making the requests and what i have programmed. The second one stays only in the initial chrome screen.

selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:51381

try to update chrome from settings to the latest version

antooro commented 1 year ago

Everything is updated

MaverickAlonso commented 1 year ago

Hi antooro, did you fix the issue mentioned? are you able to manage every single instance correctly?

PatrickVdWillik commented 1 year ago

I tried running this branch with Chrome 111 and Incapsula will trip on it. When I switch back to the regular version it works just fine.

Spykerwolf commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

Worked for me too, thanks!

simplexx commented 1 year ago

I noticed this issue as well. Latest changes added prevent running multiple instances of the driver at the same time because the naming was changed from random 8 numbers to fixed string "undetected". The result of this is that only 1 file can exist and an error is thrown if a second file is created. I did not dig further into the code, but do we even need multiple executable at all? If not then it looks like detection if the executable already exists is broken, which would make this a bug.

ultrafunkamsterdam commented 1 year ago

Instead of making it yourselves hard, mind you there is a branch for this ....

simplexx commented 1 year ago

EDIT: Please ignore below. Seems like I f'ed up something during testing (probably used unmodified driver in the test by mistake). Currently trying to run it on my server (ARM architecture on AWS). But it seems like ARM is not supported at the moment. Driver won't start throwing a wrong executable type error :(

Now I am sure it's a bug, because it happened on my win 10 machine, but when i tried it on ubuntu 20 on the server there was no such issue. So windows (10?) specific bug. In both cases I run selenium headless btw. In any case nice project! Thanks for maintaining it!

simplexx commented 1 year ago

So I have created a new server with a supported architecture, and these are the results: New branch: Does not work, throws the following error: /usr/bin/chromium-browser: 12: xdg-settings: not found /usr/bin/chromium-browser: 12: xdg-settings: not found [9126:9300:0326/154912.698727:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.DBus.ListActivatableNames: object_path= /org/freedesktop/DBus: org.freedesktop.DBus.Error.AccessDenied: An AppArmor policy prevents this sender from sending this message to this recipient; type="method_call", sender=":1.43" (uid=1000 pid=9126 comm="/snap/chromium/2406/usr/lib/chromium-browser/chrom" label="snap.chromium.chromium (enforce)") interface="org.freedesktop.DBus" member="ListActivatableNames" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus) [9126:9126:0326/154912.705819:ERROR:process_singleton_posix.cc(334)] Failed to create /home/ubuntu/.local/share/undetected_chromedriver/tmpwymqvrgk/SingletonLock: Permission denied (13) [9120:9308:0326/154912.724564:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.DBus.ListActivatableNames: object_path= /org/freedesktop/DBus: org.freedesktop.DBus.Error.AccessDenied: An AppArmor policy prevents this sender from sending this message to this recipient; type="method_call", sender=":1.44" (uid=1000 pid=9120 comm="/snap/chromium/2406/usr/lib/chromium-browser/chrom" label="snap.chromium.chromium (enforce)") interface="org.freedesktop.DBus" member="ListActivatableNames" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus) [9120:9120:0326/154912.747078:ERROR:process_singleton_posix.cc(334)] Failed to create /home/ubuntu/.local/share/undetected_chromedriver/tmp5d0tx2fo/SingletonLock: Permission denied (13) [9325:9325:0326/154912.760216:ERROR:broker_posix.cc(43)] Invalid node channel message [9324:9324:0100/000000.761303:ERROR:broker_posix.cc(43)] Invalid node channel message [9341:9341:0100/000000.797590:ERROR:broker_posix.cc(43)] Invalid node channel message [9354:9354:0326/154912.812891:ERROR:broker_posix.cc(43)] Invalid node channel message

This solution provided by @tomslarkin works:

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

rft0 commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

Thanks, it worked!

mohamed2003-kill commented 1 year ago

"I seem to be having trouble locating the 'patcher.py' file. Would someone be able to assist me in finding it? Any help would be greatly appreciated. Thank you!

tatakaetatakaetatakae commented 1 year ago

"I seem to be having trouble locating the 'patcher.py' file. Would someone be able to assist me in finding it? Any help would be greatly appreciated. Thank you!

same problem , we need a solution asap

GinVlad commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected' to: prefix = secrets.token_hex(8) I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

It worked. Thank you

kker4m commented 1 year ago

"I seem to be having trouble locating the 'patcher.py' file. Would someone be able to assist me in finding it? Any help would be greatly appreciated. Thank you!

same problem , we need a solution asap

First of all, find location of site-packages folder, you can find it with python -m site After that in site-packages folder you need to see the undetected-chromedriver folder.

kker4m commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

If you still get errors after this solution, you need to set your chrome version in the version_main variable in __init__.py where patcher.py is located. You can do this by using the @vsuyvkpzica46088 method or by giving your Google version to the version_main variable in the __init__.py file manually. As it says in the documentation, you need to give the value ignoring the imperfection (etc: 77.13.0 = 77). The second method works for me.

codes-onthe-side commented 1 year ago

My two browsers are opening but I see this issue: Only the first one is making the requests and what i have programmed. The second one stays only in the initial chrome screen.

I'm using different profiles in each driver!!!! So i set --profile-directory and --user-data-dir

selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:51381

FIXED=

Split your profile data under diferent folders, dont use same one, one per eaceh

@antooro I'm running into a similar issue. Could you provide the code you are using for splitting the profile data under different folders? Are you creating a new folder as you are creating a new instance of the driver?

hl0r0f0s commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected' to: prefix = secrets.token_hex(8) I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

If you still get errors after this solution, you need to set your chrome version in the version_main variable in __init__.py where patcher.py is located. You can do this by using the @vsuyvkpzica46088 method or by giving your Google version to the version_main variable in the __init__.py file manually. As it says in the documentation, you need to give the value ignoring the imperfection (etc: 77.13.0 = 77). The second method works for me.

It worked, thanks!

waysiPire0 commented 1 year ago

I may have figured out a workaround just now. in the "patcher.py" file I added "import secrets" at the top and then changed: from: prefix = 'undetected'

to: prefix = secrets.token_hex(8)

I figured this out by looking back at version 3.2.1 of undetected-chromedriver, and that's how it was working before.

It worked on windows, thanks

huyttdn commented 5 months ago

with undetected-chromedriver ver 3.5.4 (28-01-2024)

i edit C:\Users\Administrator\AppData\Local\Programs\Python\Python312\Lib\site-packages\undetected_chromedriver\patcher.py

change prefix = "undetected" to prefix = secrets.token_hex(8), i get this error: name 'secrets' is not defined

how to fix it ?

lMonshor commented 5 months ago

with undetected-chromedriver ver 3.5.4 (28-01-2024)

i edit C:\Users\Administrator\AppData\Local\Programs\Python\Python312\Lib\site-packages\undetected_chromedriver\patcher.py

change prefix = "undetected" to prefix = secrets.token_hex(8), i get this error: name 'secrets' is not defined

how to fix it ?

Just import secrets on the top. It worked for me.

Mike-Ro commented 1 week ago

Temporary fix:

pip install git+https://github.com/Mike-Ro/undetected-chromedriver-fix.git

or

poetry add git+https://github.com/Mike-Ro/undetected-chromedriver-fix.git