ultrafunkamsterdam / undetected-chromedriver

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
https://github.com/UltrafunkAmsterdam/undetected-chromedriver
GNU General Public License v3.0
9.71k stars 1.14k forks source link

Message: unknown error: cannot connect to chrome at 127.0.0.1:50276 #743

Open kSinghParth opened 2 years ago

kSinghParth commented 2 years ago

Hey,

I am running a script on a headless server via ssh. The script was working fine until about a week ago. I made some minor unrelated changes, post which undetected-chromedriver started failing. I am attaching a very simplified code below.

from undetected_chromedriver import find_chrome_executable
from selenium.webdriver.common.by import By
import undetected_chromedriver as uc
import time
from logger import logger

def scrape():
    try:
        print(find_chrome_executable())
        driver = uc.Chrome(headless=True, service_args=["--verbose", "--log-path=cd.log"])

        driver.get('https://myexternalip.com/raw')
        time.sleep(1)
        print(driver.find_elements(By.CSS_SELECTOR, "body")[0].text)
    except Exception as e:
        print(e)
        logger.exception("Exception occured: ")

scrape()

I get the following stdout/stderr

/opt/google/chrome/chrome
Message: unknown error: unable to discover open window in chrome
  (Session info: headless chrome=103.0.5060.134)
Stacktrace:
#0 0x559738e2ecd3 <unknown>
#1 0x559738c36968 <unknown>
#2 0x559738c16f49 <unknown>
#3 0x559738c91e1a <unknown>
#4 0x559738c8baa3 <unknown>
#5 0x559738c613fa <unknown>
#6 0x559738c62555 <unknown>
#7 0x559738e762bd <unknown>
#8 0x559738e7a418 <unknown>
#9 0x559738e6036e <unknown>
#10 0x559738e7b078 <unknown>
#11 0x559738e54bb0 <unknown>
#12 0x559738e97d58 <unknown>
#13 0x559738e97ed8 <unknown>
#14 0x559738eb1cfd <unknown>
#15 0x7f14f448e6ba <unknown>

The complete stack trace is

patching driver executable /home/kpsingh/.local/share/undetected_chromedriver/9169c1ae1dc52599_chromedriver
Exception occured: 
Traceback (most recent call last):
  File "a.py", line 11, in scrape
    driver = uc.Chrome(headless=True, service_args=["--verbose", "--log-path=cd.log"])
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/undetected_chromedriver/__init__.py", line 408, in __init__
    keep_alive=keep_alive,
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 72, in __init__
    service_log_path, service, keep_alive)
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/selenium/webdriver/chromium/webdriver.py", line 97, in __init__
    options=options)
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 277, in __init__
    self.start_session(capabilities, browser_profile)
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/undetected_chromedriver/__init__.py", line 590, in start_session
    capabilities, browser_profile
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 370, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 435, in execute
    self.error_handler.check_response(response)
  File "/data-2/research/expedia/Ranking-Comparator/venv/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: unable to discover open window in chrome
  (Session info: headless chrome=103.0.5060.134)
Stacktrace:
#0 0x559738e2ecd3 <unknown>
#1 0x559738c36968 <unknown>
#2 0x559738c16f49 <unknown>
#3 0x559738c91e1a <unknown>
#4 0x559738c8baa3 <unknown>
#5 0x559738c613fa <unknown>
#6 0x559738c62555 <unknown>
#7 0x559738e762bd <unknown>
#8 0x559738e7a418 <unknown>
#9 0x559738e6036e <unknown>
#10 0x559738e7b078 <unknown>
#11 0x559738e54bb0 <unknown>
#12 0x559738e97d58 <unknown>
#13 0x559738e97ed8 <unknown>
#14 0x559738eb1cfd <unknown>
#15 0x7f14f448e6ba <unknown>

Driver verbose logs.

[1658719378.800][INFO]: Starting ChromeDriver 103.0.5060.134 (8ec6fce403b3feb0869b0732eda8bd95011d333c-refs/branch-heads/5060@{#1262}) on port 44138
[1658719378.800][INFO]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
[1658719379.303][INFO]: [6c406d0e3de32079e8a9d655c7fe08c8] COMMAND InitSession {
   "capabilities": {
      "alwaysMatch": {
         "browserName": "chrome",
         "goog:chromeOptions": {
            "args": [ "--remote-debugging-host=127.0.0.1", "--remote-debugging-port=46537", "--user-data-dir=/tmp/tmpb6685gbz", "--lang=en-US", "--no-default-browser-check", "--no-first-run", "--headless", "--window-size=1920,1080", "--start-maximized", "--no-sandbox", "--log-level=0" ],
            "binary": "/opt/google/chrome/chrome",
            "debuggerAddress": "127.0.0.1:46537",
            "extensions": [  ]
         },
         "pageLoadStrategy": "normal"
      },
      "firstMatch": [ {
      } ]
   }
}
[1658719379.304][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.307][DEBUG]: DevTools HTTP Request failed
[1658719379.358][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.359][DEBUG]: DevTools HTTP Request failed
[1658719379.409][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.410][DEBUG]: DevTools HTTP Request failed
[1658719379.460][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.462][DEBUG]: DevTools HTTP Request failed
[1658719379.512][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.513][DEBUG]: DevTools HTTP Request failed
[1658719379.563][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.564][DEBUG]: DevTools HTTP Request failed
[1658719379.614][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.615][DEBUG]: DevTools HTTP Request failed
[1658719379.666][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.666][DEBUG]: DevTools HTTP Request failed
[1658719379.716][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719379.718][DEBUG]: DevTools HTTP Request failed
[1658719379.768][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/version
[1658719380.036][DEBUG]: DevTools HTTP Response: {
   "Browser": "HeadlessChrome/103.0.5060.134",
   "Protocol-Version": "1.3",
   "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/103.0.5060.134 Safari/537.36",
   "V8-Version": "10.3.174.20",
   "WebKit-Version": "537.36 (@8ec6fce403b3feb0869b0732eda8bd95011d333c)",
   "webSocketDebuggerUrl": "ws://127.0.0.1:46537/devtools/browser/2b165332-13f9-4fff-a60d-d2c1f94f8d98"
}

[1658719380.036][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/list
[1658719380.618][DEBUG]: DevTools HTTP Response: [ {
   "description": "",
   "devtoolsFrontendUrl": "/devtools/inspector.html?ws=127.0.0.1:46537/devtools/page/BB65E0E978029AB046DB51A606B084EF",
   "id": "BB65E0E978029AB046DB51A606B084EF",
   "title": "",
   "type": "page",
   "url": "about:blank",
   "webSocketDebuggerUrl": "ws://127.0.0.1:46537/devtools/page/BB65E0E978029AB046DB51A606B084EF"
} ]

[1658719380.618][DEBUG]: DevTools HTTP Request: http://127.0.0.1:46537/json/list
[1658719380.728][DEBUG]: DevTools HTTP Response: [  ]

[1658719380.732][INFO]: [6c406d0e3de32079e8a9d655c7fe08c8] RESPONSE InitSession ERROR unknown error: unable to discover open window in chrome
  (Session info: headless chrome=103.0.5060.134)
[1658719380.732][DEBUG]: Log type 'driver' lost 0 entries on destruction
[1658719380.732][DEBUG]: Log type 'browser' lost 0 entries on destruction

Operating System: Ubuntu 16.04.7 LTS Kernel: Linux 4.4.0-210-generic Architecture: x86-64

$ chrome --version Google Chrome 103.0.5060.134 unknown $ which chrome /opt/google/chrome/chrome $ python --version Python 3.7.7

undetected-chromedriver==3.1.5.post4

If I simply try to run selenium headless, I am facing no problems. Any help would be appreciated. Thanks.

HamiguaLu commented 1 year ago

Hi, in my case, delete ProfileDir\Default\Preferences file fixed the issue I suggest after it works, backup the whole profile folder so later on you can recover

alexliu0809 commented 1 year ago

I was running multiple instances of UC. Besides using some of the fixes mentioned above, adding some wait time before launching each instance also helped.

Laishiji commented 1 year ago

I think it's some file broken in the profile dir, in my case, I compared the profile dir with another working one, then found there is a file named "Preferences" took 2.3GB, I deleted the file and Chrome can start now

Thank you. It works!!!!!👏

HerreraCarlos81 commented 1 year ago

Could not find the ProfileDir folder. What am I missing here? 😑

freemedom commented 1 year ago

Could not find the ProfileDir folder. What am I missing here? 😑

--user-data-dir= must use this option can have custom ProfileDir. @HerreraCarlos81

Autolocsia commented 12 months ago

2023 ends --user-data-dir= still not working

kenniky commented 11 months ago

Having the same issue with a test script I wrote:

import undetected_chromedriver as uc

options = uc.ChromeOptions()

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'

options = uc.ChromeOptions()
options.add_argument('--headless')
options.add_argument('window-size=1400,1500')
options.add_argument('--disable-gpu')
options.add_argument('--no-sandbox')
options.add_argument('start-maximized')
options.add_argument('enable-automation')
options.add_argument('--disable-infobars')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--remote-debugging-port=9222')
options.add_argument('--user-data-dir=/home/ubuntu/chrome_profiles')

driver = uc.Chrome(options=options, version_main=110)

driver.get('https://nowsecure.nl')
driver.save_screenshot('screen.png')

a bunch of the options are there because I've found they've been needed in the past on regular chrome

diroots commented 10 months ago
2. The problem is with "SingletonLock" file, which you can find in the root of user profile folder. Just delete this file before starting Chrome and `Message: unknown error: cannot connect to chrome at 127.0.0.1:56401 from chrome not reachable` will never show again.

as of today with chrome 119 and latest 3.5.4 undetected_chromedriver, i got same problem when trying to launch chrome with options.add_argument('--user-data-dir=/path/to/my/profile') and i could see 3 Singleton*** files in the root of my profile : SingletonLock, SingletonSocket,SingletonCookie. these being symlinks, and even broken symlinks. removing them and restarting with my default options, i could reuse as expected.

hashtafak commented 9 months ago

I think it's some file broken in the profile dir, in my case, I compared the profile dir with another working one, then found there is a file named "Preferences" took 2.3GB, I deleted the file and Chrome can start now

Ha! Related to #1362 and some threads I've found on Stackoverflow. After running the same profile dir over time. Py.exe process eats the RAM from 2-5GB because of this file "Preferences".

image

anilabhadatta commented 7 months ago

@s-o-f great solution, this fixed my code... I was trying to create headless which was failing due to this but in non headless it was working. I was facing this issue in linux

anilabhadatta commented 7 months ago

https://github.com/anilabhadatta/educative.io_scraper/blob/v3-master/src/Utility/BrowserUtility.py the implementation that I used to spawn Chrome/ Chromedriver and close it is working for me To close the process: asyncio.get_event_loop().run_until_complete(browserUtil.shutdownChromeViaWebsocket())

iohansson commented 7 months ago

for those who are running headful Chrome on a server, check that you have Xvfb running. I had the same issue trying to run my script with Docker, looked into the undetected_chromedriver docker image entrypoint and found the following among the lines

#!/bin/bash

function keepUpScreen()
{
    echo "running keepUpScreen()"
    while true; do
        sleep 1
        if [ -z "$(pidof -x Xvfb)" ]; then
            Xvfb $DISPLAY -screen $DISPLAY 1920x1080x16 &
        fi
    done
}

export DISPLAY=:1
rm -f /tmp/.X1-lock &>/dev/null
keepUpScreen &
echo "running: $@"
exec $@

So I've created an entrypoint with this, and that solved my issue.

johnwhelchel commented 6 months ago

This is not documented anywhere, but I've recently found that if you don't call driver.quit(), the temp directory used for a user directory is not cleaned up. It's quite confusing, because using the driver as a context manager should implicitly clean up resources, but it doesn't (based on implementation of exit https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1614).

This seems to have fixed the issue for me.

victorbjafet commented 3 months ago

Found a solution for the problem. forget about all the extra arguments and updates/ etc.

If you are using the DIR folder that currently has a browser profile open. it will not work. you have to close everything first and then run it. That was the workaround for it for me.

@kSinghParth @wundergeu @Habidis

thank you so much, this worked for me. make sure that you check your task manager/windows tray to see if chrome is running in the background, which was my case