ultrafunkamsterdam / undetected-chromedriver

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
https://github.com/UltrafunkAmsterdam/undetected-chromedriver
GNU General Public License v3.0
9.42k stars 1.12k forks source link

Why does undetected chromedriver leave so many instances of chrome.exe open in memory? #1270

Open disposablecarcass opened 1 year ago

disposablecarcass commented 1 year ago

My script opens a new chrome every few minutes and exits with driver.quit(). With a normal chrome driver, driver.quit() kills everything. With undetected chrome driver, for some reason, I see each insteance of Google Chrome stay open in the task manager despite having the window disappeared from the screen. I have to end task manually in task manager or they will max out my cpu.

Does anyone know why this is happening or how to rectify?

ManiMozaffar commented 1 year ago

Because __del__ is very bad implemented you may use #1258 , or you can use seleniumbase, which doesn't have this bug in the uc module.

disposablecarcass commented 1 year ago

Because __del__ is very bad implemented you may use #1258 , or you can use seleniumbase, which doesn't have this bug in the uc module.

Is there any reason why this hasn't been merged into the main branch? Is there anyway I can specify a branch installing directly from pycharm? Or do I have to do dl from github and point to it manually?

disposablecarcass commented 1 year ago

@ManiMozaffar So I'm reading conflicting information in these threads. Does your commit itself fix this issue? Do I need to include some type of implicit wait after driver.quit() as well?

Thanks for the help

ManiMozaffar commented 1 year ago

Because __del__ is very bad implemented you may use #1258 , or you can use seleniumbase, which doesn't have this bug in the uc module.

Is there any reason why this hasn't been merged into the main branch? Is there anyway I can specify a branch installing directly from pycharm? Or do I have to do dl from github and point to it manually?

This isn't merged, because in both PRs, the maintainers haven't checked and provided feedback. Yes you can, pip install git+.....

@ManiMozaffar So I'm reading conflicting information in these threads. Does your commit itself fix this issue? Do I need to include some type of implicit wait after driver.quit() as well?

Thanks for the help

no you don't need to, just use my code. it's the issue with thread, in each thread we create new event loop, and each event loop is creating 2 new sockets with different ports, so the more chrome you open, the more thread it will have, and the more event loops it will have in parallel, causing port exhaustion.

disposablecarcass commented 1 year ago

@ManiMozaffar Thanks for the reply. I did try your code, but still have the same issue of instances of chrome.exe being left open when using driver.close()

I'm not sure what the best solution is to this, but I think for the time being, I may have to use the os package to kill all chrome instances between each iteration of my script.

I'll keep my eyes open here tho to see if any solution is found.

Thanks again

ultrafunkamsterdam commented 1 year ago

Because i will only fix cricital issues. Im rewriting the whole package. (Spoiler: this performs like crazy, and no more Chrome driver

disposablecarcass commented 1 year ago

Because i will only fix cricital issues. Im rewriting the whole package. (Spoiler: this performs like crazy, and no more Chrome driver

Interesting, thanks for the update. Any rough eta on the new package? In the meantime, I'll just use os to kill the chrome that stays open in memory

sSuHao commented 1 year ago

Did anyone try to run in incognito mode?

pm85 commented 1 year ago

Because i will only fix cricital issues. Im rewriting the whole package. (Spoiler: this performs like crazy, and no more Chrome driver

Whats the timeline on the new implementation?

Mister-Stein commented 11 months ago

Unset the --no-sandbox Chrome argument, setting no_sandbox argument of uc.Chrome to False, because it is set to True by default. Example:

uc.Chrome(driver_executable_path=driver_executable_path, options=options, user_data_dir=user_data_dir, no_sandbox=False, user_multi_procs=True, use_subprocess=False)

But some things to consider: Users say that undetected-chromedriver doesn't work without --no-sandbox Chrome argument on Windows 7 and lower, but works on Windows 8.1 and above. I didn't tested that for myself, but I think it's true. Also, undetected-chromedriver most of the times doesn't work on Linux without --no-sandbox Chrome argument(when running test script as root user for example). So, that's because no_sandbox argument of uc.Chrome is set to True by default. Concluding that, set no_sandbox argument of uc.Chrome to False for Windows, and set to True for Linux. But I warn that undetected-chromedriver has breached no_sandbox functionality, so --no-sandbox is always set, whether or not you set it to True or False. I've made pull request to fix it, so we need to wait until it is applied(or download my fork, and use as custom Python module). https://github.com/ultrafunkamsterdam/undetected-chromedriver/pull/1542

tromotixc commented 9 months ago

is there any solution for quit() problem? thank you in advance!

iamumairayub commented 8 months ago

Because i will only fix cricital issues. Im rewriting the whole package. (Spoiler: this performs like crazy, and no more Chrome driver

Eagerly waiting for this.

disposablecarcass commented 8 months ago

Because i will only fix cricital issues. Im rewriting the whole package. (Spoiler: this performs like crazy, and no more Chrome driver

Eagerly waiting for this.

do you know if there is a development blog for this? would love to follow along and try to learn. feeling pretty useless at the moment trying to fix the old version of uc. anyway hopefully this comes out soon - looking forward to it

Sleeeepy7 commented 8 months ago

My script opens a new chrome every few minutes and exits with driver.quit(). With a normal chrome driver, driver.quit() kills everything. With undetected chrome driver, for some reason, I see each insteance of Google Chrome stay open in the task manager despite having the window disappeared from the screen. I have to end task manually in task manager or they will max out my cpu.

Does anyone know why this is happening or how to rectify?

Have you found a solution?

11AnJo commented 7 months ago

@Sleeeepy7 @disposablecarcass I really don't know why but disabling gpu worked for me. I've tested it on multiple machines and multiple times. options.add_argument("--disable-gpu")

cmdlinebeep commented 6 months ago

@Sleeeepy7 @disposablecarcass I really don't know why but disabling gpu worked for me. I've tested it on multiple machines and multiple times. options.add_argument("--disable-gpu")

I get this error and I use that flag.

11AnJo commented 6 months ago

@cmdlinebeep This is so weird that its working on my computer and my friend and few vms putted for tests. Can you share version of undetected-chromedriver and your chrome version?

Darren80 commented 6 months ago

Is there still no solution for this, I am using the pip package, is the issue fixed on this repo?

11AnJo commented 6 months ago

Is there still no solution for this, I am using the pip package, is the issue fixed on this repo?

For me and my friend adding option to disable gpu helped, but its not working for everybody

options.add_argument("--disable-gpu")
ifeldshteyn commented 4 months ago

Because i will only fix cricital issues. Im rewriting the whole package. (Spoiler: this performs like crazy, and no more Chrome driver

Just checking if the fix to the open socket problem is available?

xerobia-test commented 4 months ago

I've found using an older version of Chrome like V113 works. But V113 is also less stable than the current 124

ifeldshteyn commented 4 months ago

I've found using an older version of Chrome like V113 works. But V113 is also less stable than the current 124

I don't mind downgrading but can you confirm that the socket issue really goes away with it? I had to add a hack that reboots the service after it has 1000 open sockets.

josetilin37 commented 4 months ago

hello, I had the same problem about the phantom processes and I managed to solve it by using pyautogui to close the browser just before applying the driver.quit(), this way the browser is completely closed just before the driver.quit() and leaves no trace in the task manager, this solution was effective for me. I hope this helps you.

pyautogui.moveTo(x, y, duration=0.35) #coords for closing the browser pyautogui.click() driver.quit()

You can adjust the duration of this action to make it more instantaneous

ifeldshteyn commented 4 months ago

hello, I had the same problem about the phantom processes and I managed to solve it by using pyautogui to close the browser just before applying the driver.quit(), this way the browser is completely closed just before the driver.quit() and leaves no trace in the task manager, this solution was effective for me. I hope this helps you.

pyautogui.moveTo(x, y, duration=0.35) #coords for closing the browser pyautogui.click() driver.quit()

You can adjust the duration of this action to make it more instantaneous

It's a workaround for sure but it would be nice to have a working __del__ that cleans up fds so the proc folder doesn't have a heart attack. Is this something I can do with the driver itself instead of using another dependency? Thanks!

cvtkir commented 3 months ago

For me using both driver.close() and driver.quit() one by one helped a lot

noureddineerramy commented 2 months ago

after driver.quit() call destroyChromeProcess (code that close any chrome process that its parent pid is not exists) like this :

import psutil
import os
import signal

def get_chrome_parent_pids():
    parent_pids = set()
    for proc in psutil.process_iter(['pid', 'name', 'ppid']):
        if 'chrome' in proc.name().lower():
            parent_pids.add(proc.ppid())
    return parent_pids

def kill_processes_with_parent_pid(parent_pid):
    try:
        for proc in psutil.process_iter(['pid', 'ppid']):
            if proc.info['ppid'] == parent_pid:
                try:
                    process = psutil.Process(proc.info['pid'])
                    process.terminate()
                except psutil.NoSuchProcess:
                    pass
    except psutil.NoSuchProcess:
        pass

def destroyChromeProcess():
    chrome_parent_pids = get_chrome_parent_pids()
    for parent_pid in chrome_parent_pids:
        if not psutil.pid_exists(parent_pid):
            kill_processes_with_parent_pid(parent_pid)