Open kSinghParth opened 1 year ago
I have been trying different things and the verbose driver logs seem to have changed a bit.
[1658781393.276][INFO]: Starting ChromeDriver 103.0.5060.134 (8ec6fce403b3feb0869b0732eda8bd95011d333c-refs/branch-heads/5060@{#1262}) on port 47324
[1658781393.276][INFO]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
[1658781393.779][INFO]: [c0a694c101f2727524e75d070370b885] COMMAND InitSession {
"capabilities": {
"alwaysMatch": {
"browserName": "chrome",
"goog:chromeOptions": {
"args": [ "--remote-debugging-host=127.0.0.1", "--remote-debugging-port=37090", "--user-data-dir=/tmp/tmp2mvwlfbl", "--lang=en-US", "--no-default-browser-check", "--no-first-run", "--headless", "--window-size=1920,1080", "--start-maximized", "--no-sandbox", "--log-level=0" ],
"binary": "/usr/bin/google-chrome-stable",
"debuggerAddress": "127.0.0.1:37090",
"extensions": [ ]
},
"pageLoadStrategy": "normal"
},
"firstMatch": [ {
} ]
}
}
[1658781393.779][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781393.781][DEBUG]: DevTools HTTP Request failed
[1658781393.831][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781393.833][DEBUG]: DevTools HTTP Request failed
[1658781393.883][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781393.884][DEBUG]: DevTools HTTP Request failed
[1658781393.934][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781393.935][DEBUG]: DevTools HTTP Request failed
[1658781393.985][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781393.986][DEBUG]: DevTools HTTP Request failed
.
.
.
.
.
.
[1658781453.688][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781453.689][DEBUG]: DevTools HTTP Request failed
[1658781453.739][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781453.740][DEBUG]: DevTools HTTP Request failed
[1658781453.790][DEBUG]: DevTools HTTP Request: http://127.0.0.1:37090/json/version
[1658781453.791][DEBUG]: DevTools HTTP Request failed
[1658781453.796][INFO]: [c0a694c101f2727524e75d070370b885] RESPONSE InitSession ERROR unknown error: cannot connect to chrome at 127.0.0.1:37090
from chrome not reachable
[1658781453.796][DEBUG]: Log type 'driver' lost 0 entries on destruction
[1658781453.796][DEBUG]: Log type 'browser' lost 0 entries on destruction```
try to add incognito into your driver.
options.add_argument("--incognito")
Using incognito option didn't work.
Any news? Same is happening to me. It is strangely working from PyCharm but when I start it from a crontab on Ubuntu it doesn’t work
Nope, my work has come to a complete halt because of this. Cant find any solution. I have tried almost every trick posted on any related problems thread.
I've noticed that whenever I use the parameter "--user-data-dir" at the driver options:
options.add_argument("--user-data-dir=c:\WhateverFolder")
If I delete that "c:\WhateverFolder", it works again, but of course, I must re-login on every site.
Do you solve this question?
I had the same problem so I did the following: 1- google chrome was running using a virtual monitor see this thread: https://stackoverflow.com/questions/36582594/a-virtual-display-for-ubuntu-server 2- after updating and restarting the server I added the option ('--headless') to the driver and ran it, 3- the surprise was that it ran normally, then I deleted that headless option and ran it again and it worked with no problems 4- other solution is to add the option ('--no-sandbox') and sometimes it works with this 4- so in conclusion it's a bug related to the driver options, I hope they fix it
I have the same problem if I use options.add_argument("--user-data-dir=c:\WhateverFolder")
as @Habidis mentioned.
It works fine with fresh folder some amount of time but than browser both loads too long or fails with error.
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:57455
from chrome not reachable
Stacktrace:
#0 0x5650cfd6e693 <unknown>
#1 0x5650cfb679db <unknown>
#2 0x5650cfb5781e <unknown>
#3 0x5650cfb90677 <unknown>
#4 0x5650cfb87e9f <unknown>
#5 0x5650cfbc3953 <unknown>
#6 0x5650cfbbd743 <unknown>
#7 0x5650cfb93533 <unknown>
#8 0x5650cfb94715 <unknown>
#9 0x5650cfdbe7bd <unknown>
#10 0x5650cfdc1bf9 <unknown>
#11 0x5650cfda3f2e <unknown>
#12 0x5650cfdc29b3 <unknown>
#13 0x5650cfd97e4f <unknown>
#14 0x5650cfde1ea8 <unknown>
#15 0x5650cfde2052 <unknown>
#16 0x5650cfdfc71f <unknown>
#17 0x7f923f778e2d <unknown>
Process finished with exit code 1
Does anyone knows how to fix this error?
Heys Guys, I have the same problems.
following
Heys Guys, I have the same problems.
try adding these arguments to the driver
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument("--disable-setuid-sandbox")
Is there any update on this issue, i have tried to add arguments mentioned by @smsajjadzaidi but it still gives this error
Faced the same problem, occurred using python 3.7. Solution was to update to python 3.8.
Add this all:
options.add_argument("--disable-extensions") options.add_argument('--disable-application-cache') options.add_argument('--disable-gpu') options.add_argument("--no-sandbox") options.add_argument("--disable-setuid-sandbox") options.add_argument("--disable-dev-shm-usage")
And try to running in headless: options.add_argument("--headless")
Same as update to last version and use chrome driver manager instead of path or local variable
If still not working then clean your linux and try again
Edit: the 2 dumbs that put a 👎 are idiots that even didnt tested, Just try and youll see it works
I had the same issue when running uc inside a Python Docker container (based on python:3.10
which is basically some minimal Debian). Here are the steps that got uc working inside the container:
Install Chrome
DEBIAN_FRONTEND=noninteractive && \
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub > /usr/share/keyrings/chrome.pub && \
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/chrome.pub] http://dl.google.com/linux/chrome/deb/ stable main' > /etc/apt/sources.list.d/google-chrome.list && \
apt update -y && \
apt install -y google-chrome-stable
Install the X Virtual Framebuffer display server
apt install xvfb -y
Set DISPLAY environmental variable to :1
export DISPLAY=:1
Start a Xvfb process in the background
Xvfb $DISPLAY -screen $DISPLAY 1280x1024x16 &
Run a minimal py script with uc
python3 example.py
Here's the example script I used:
import undetected_chromedriver as uc
from time import sleep
log_path = "/chromedriver.log"
url = "https://check.torproject.org/"
chrome_driver = uc.Chrome(service_log_path=log_path)
chrome_driver.get(url)
sleep(5)
scraped_page = chrome_driver.page_source
chrome_driver.quit()
print(scraped_page)
Note that I'm running the driver without the --headless
argument and it seems to work fine.
All of it is a truncated version of this Docker image (requires login to Docker hub) by @ultrafunkamsterdam.
I know that the original question was not about running uc inside a Docker container, but I hope it could give you some ideas.
Thanks martroben I've merged martroben and undetected-chromedriver into Image
So can easy to run Chrome driver in container Usage:
FROM knthony/run_chrome_driver_in_container
CMD ["python", "example.py"]
@michaeloo0 i tried your image and it throws me selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:39597 from chrome not reachable
any ideas ?
yes,in this image the version of chrome is 108, so please using follow script to recreate docker image
FROM ultrafunk/undetected-chromedriver:3.20-chrome-lateinstall
RUN DEBIAN_FRONTEND=noninteractive
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub > /usr/share/keyrings/chrome.pub
RUN echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/chrome.pub] http://dl.google.com/linux/chrome/deb/ stable main' > /etc/apt/sources.list.d/google-chrome.list
RUN apt update -y
RUN apt install -y google-chrome-stable
RUN apt install xvfb -y
RUN export DISPLAY=:1
RUN Xvfb $DISPLAY -screen $DISPLAY 1280x1024x16 &
Just like @Habidis said, deleting the profile folder works for me.
Found a solution for the problem. forget about all the extra arguments and updates/ etc.
If you are using the DIR folder that currently has a browser profile open. it will not work. you have to close everything first and then run it. That was the workaround for it for me.
@kSinghParth @wundergeu @Habidis
Thank you @Anticope12 your solution worked. Just need to close other instances of Chrome before running script !!
I've been frustrated the last couple weeks as my scripts were hanging with the following error message:
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:57930 from chrome not reachable
Tried all the different arguments, but nothing worked. Now working again, just need to close browser first!
@phl0yd , @Anticope12 hi,
Can you share your code here, please? Although I closed the Chrome app in the background, it didn't work on my end.
@asimptot Not sure if all these options are necessary, but this is the code I am currently running.
Before you change your code, you may want to open task manager and make sure every Chrome task is shut down.
import undetected_chromedriver as uc
options = uc.ChromeOptions() options.add_argument('--incognito') options.add_argument("--window-size=1920,1080") options.add_argument("--start-maximized")
options.add_argument("--disable-extensions") options.add_argument("--disable-application-cache") options.add_argument("--disable-gpu") options.add_argument("--no-sandbox") options.add_argument("--disable-setuid-sandbox") options.add_argument("--disable-dev-shm-usage")
driver = uc.Chrome(use_subprocess=True,options = options)
with driver: driver.get('https://www.yourwebsite.com')
......
Here is my code and that worked for me!
chrome_options = uc.ChromeOptions()
chrome_options.add_argument('--headless=new')
self.browser = uc.Chrome(options=chrome_options, version_main=110)
yeah congrats for repeating my solution 👏👏👏👏👏
@asimptot @phl0yd @Anticope12 this solution is not working in a docker file unfortunately
@rtrive what kind of issue did you get?
Please change 51st line in https://github.com/rtrive/f1-telegram-fantasy-bot/blob/6bb7fa08701bd3211fc102d5f4a3046d9c88182d/src/main.py#L49:
chrome_options.add_argument('--headless=new')
Please change 51st line in https://github.com/rtrive/f1-telegram-fantasy-bot/blob/6bb7fa08701bd3211fc102d5f4a3046d9c88182d/src/main.py#L49:
chrome_options.add_argument('--headless=new')
Nothing change :(
I've tried everything in this thread and I'm still getting an error on every build in Docker
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:54841
or whatever random port.
Here's the current Dockerfile I'm working with, which is my best guess at a setup given the answer by martroben
` FROM python:3.10 RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub > /usr/share/keyrings/chrome.pub RUN echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/chrome.pub] http://dl.google.com/linux/chrome/deb/ stable main' > /etc/apt/sources.list.d/google-chrome.list RUN apt-get update -y RUN apt-get install -y google-chrome-stable
RUN apt install xvfb -y RUN export DISPLAY=:1
ENV DockerHOME=/home/scraper
RUN mkdir -p $DockerHOME
WORKDIR $DockerHOME
RUN pip install --upgrade pip
COPY . $DockerHOME
RUN pip install -r requirements.txt
RUN Xvfb $DISPLAY -screen $DISPLAY 1280x1024x16 & CMD ["python3", "-u", "scrapeMain.py"] `
I’ve tried many different combos of chrome options in the script and even just used the example script above and still no luck.
I've also tried a Dockerfile that uses a build from https://hub.docker.com/r/ultrafunk/undetected-chromedriver, but with those attempts I always seem to give an error that the 'server is already active for display 1' for some reason.
A little new to this whole Docker thing, so any help would be greatly appreciated. Thanks in advance
@kaweinh Did you find a solution? Did you try to install a lower version of chrome, for instance, 108? I think undetected_chromedriver is not compatible with the latest version of Chrome (v.111)
Got same issues today.
Solved by update latest version undetected_chromedriver.
Also added these settings.
chrome_options = uc.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920,1080")
# get ua from config file
ua = get_config_by_key("UA", "user_agent")
# update user-agent
chrome_options.add_argument(f"--user-agent={ua}")
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-application-cache")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--remote-debugging-port=9222")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-setuid-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
All this work, to get it working on docker....🤣 There is an official image with all that buillt in .
Here is my code and that worked for me!
chrome_options = uc.ChromeOptions() chrome_options.add_argument('--headless=new') self.browser = uc.Chrome(options=chrome_options, version_main=110)
I've added a pull request fixing this.
All this work, to get it working on docker....🤣 There is an official image with all that buillt in .
@ultrafunkamsterdam Not only to get it to work in docker, it's also broken on for example google colab and every other platform/environment where you don't atleast have a virtual display and that's a huge cockblock
I like it so much!!! WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0 after chrome update, and amazing win11 recent update. Jeeez why that thing can't just work ???? unknown error???? In a 2023???? We need to fire all the devs connected to win11, pyCharm, and all other smart person who make our life hell! I don't believe they are dumb, I believe they just like and enjoy to torture people, just seek sociopaths (((
In this SO post they suggest that the issue is that "the versions of chrome, undetected-chrome and chromedriver where NOT aligned."
I updated @asimptot code to match the version of Chrome I'm using and it worked.
Clearing of the user profile folder worked for me. As same as incognito mode. But i need history, cookies, etc. between sessions. So i found only two ways suitable for me:
version_main=107
argument. But using old browser makes your detection easier.Message: unknown error: cannot connect to chrome at 127.0.0.1:56401 from chrome not reachable
will never show again.@s-o-f yes you are correct about that. But deleting it means not being able to reuse the custom profile sadly.
Looking at all of the above solutions, my reason should be the addition of --user-data-dir, and there is something wrong(don't know exactly why) with the userdata folder. My solution looks to @Anticope12 @s-o-f , but it's not quite the same. I've turned off all of chrome.exe. I guess because I forced close chrome by mistake, which caused some bug, I manually used chrome.exe "--user-data-dir=xxx" in cmd to reopen chrome.exe with this userdata directory. And close. And then it's normal in selenium.
I tried all of the add_argument methods above, and none of them worked.
(windows)
Try this:
options = Options()
options.add_argument('--remote-debugging-port=9230')
options.add_argument('--remote-allow-origins=*')
RHEL8, google-chrome-stable build 114.0.5735.90, python3.9, uc 3.4.7, chromedriver 114.0.5735.16
My options are minimal at the following ....
options.add_argument("--user-data-dir='Google/Chrome/user/data'")
options.add_argument('--profile-directory=Default')
options.add_argument("--incognito")
options.add_argument("--window-size=1550,1000")
options.add_argument("--disable-extensions")
options.add_argument("--disable-application-cache")
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument("--disable-setuid-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--headless=new")
Xvfb stuff is not needed for me.
chromedriver seems to be very sensitive to /tmp. Check your env
! Ensure /tmp is mounted correctly. Ensure your user can write.
Tried all but nothing
chrome_options.add_argument('--headless=new') Its work like charm thanks man
I think it's some file broken in the profile dir, in my case, I compared the profile dir with another working one, then found there is a file named "Preferences" took 2.3GB, I deleted the file and Chrome can start now
I think it's some file broken in the profile dir, in my case, I compared the profile dir with another working one, then found there is a file named "Preferences" took 2.3GB, I deleted the file and Chrome can start now
Yes!Yes, it works! Love You!
I think it's some file broken in the profile dir, in my case, I compared the profile dir with another working one, then found there is a file named "Preferences" took 2.3GB, I deleted the file and Chrome can start now
how could we find this file? Could you give some details, please?
Hey,
I am running a script on a headless server via ssh. The script was working fine until about a week ago. I made some minor unrelated changes, post which undetected-chromedriver started failing. I am attaching a very simplified code below.
I get the following stdout/stderr
The complete stack trace is
Driver verbose logs.
Operating System: Ubuntu 16.04.7 LTS Kernel: Linux 4.4.0-210-generic Architecture: x86-64
$ chrome --version Google Chrome 103.0.5060.134 unknown $ which chrome /opt/google/chrome/chrome $ python --version Python 3.7.7
undetected-chromedriver==3.1.5.post4
If I simply try to run selenium headless, I am facing no problems. Any help would be appreciated. Thanks.