search

flathunters / flathunter

A bot to help people with their rental real-estate search. 🏠🤖
GNU Affero General Public License v3.0
831 stars 180 forks source link

MacOS - Message: disconnected: not connected to DevTools (failed to check if window was closed: disconnected: not connected to DevTools) #357

Open miklyx opened 1 year ago

miklyx commented 1 year ago

HI! Today I've got fresh repo update and latest chrome update. And every 20 mins getting bot fail just after immoscout24 link Traceback (most recent call last): File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunt.py", line 118, in <module> main() File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunt.py", line 114, in main launch_flat_hunt(config, heartbeat) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunt.py", line 45, in launch_flat_hunt hunter.hunt_flats() File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 56, in hunt_flats for expose in processor_chain.process(self.crawl_for_exposes(max_pages)): File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 35, in crawl_for_exposes return chain(*[try_crawl(searcher, url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 35, in <listcomp> return chain(*[try_crawl(searcher, url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 27, in try_crawl return searcher.crawl(url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/abstract_crawler.py", line 150, in crawl return self.get_results(url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/crawler/immobilienscout.py", line 90, in get_results soup = self.get_page(search_url, self.get_driver(), page_no) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/crawler/immobilienscout.py", line 175, in get_page return self.get_soup_from_url( File "/Users/kristina/.local/share/virtualenvs/flathunter-Ytmhlxpi/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/abstract_crawler.py", line 80, in get_soup_from_url driver.get(url) File "/Users/kristina/.local/share/virtualenvs/flathunter-Ytmhlxpi/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 476, in get_wrapped if self.execute_script("return navigator.webdriver"): File "/Users/kristina/.local/share/virtualenvs/flathunter-Ytmhlxpi/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 500, in execute_script return self.execute(command, {"script": script, "args": converted_args})["value"] File "/Users/kristina/.local/share/virtualenvs/flathunter-Ytmhlxpi/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute self.error_handler.check_response(response) File "/Users/kristina/.local/share/virtualenvs/flathunter-Ytmhlxpi/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools (failed to check if window was closed: disconnected: not connected to DevTools) (Session info: chrome=112.0.5615.49)

miklyx commented 1 year ago

Updated all packages and chrome - and error still presents

codders commented 1 year ago

Hi @miklyx ,

What are you using as the driver_arguments for chromedriver in your config? Are you running inside of docker or directly from a terminal?

miklyx commented 1 year ago

Hi! I use ` driver_arguments:

also, I run it directly from terminal.

codders commented 1 year ago

Hi @miklyx ,

You can try changing "window-size=1024,768" for "--window-size=1024,768" - that might help. Also, if you're running from the console you can drop the "--headless" argument and see what happens in the browser.

miklyx commented 1 year ago

Hi! Done it. Result is - it starts to open search page at immoscout24. After some cycles it throws error again

Traceback (most recent call last): File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunt.py", line 118, in <module> main() File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunt.py", line 114, in main launch_flat_hunt(config, heartbeat) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunt.py", line 45, in launch_flat_hunt hunter.hunt_flats() File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 56, in hunt_flats for expose in processor_chain.process(self.crawl_for_exposes(max_pages)): File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 35, in crawl_for_exposes return chain(*[try_crawl(searcher, url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 35, in <listcomp> return chain(*[try_crawl(searcher, url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/hunter.py", line 27, in try_crawl return searcher.crawl(url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/abstract_crawler.py", line 150, in crawl return self.get_results(url, max_pages) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/crawler/immobilienscout.py", line 90, in get_results soup = self.get_page(search_url, self.get_driver(), page_no) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/crawler/immobilienscout.py", line 175, in get_page return self.get_soup_from_url( File "/Users/kristina/Library/Python/3.9/lib/python/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/Users/kristina/Documents/python/flathunter_bot/flathunter/flathunter/abstract_crawler.py", line 80, in get_soup_from_url driver.get(url) File "/Users/kristina/Library/Python/3.9/lib/python/site-packages/undetected_chromedriver/__init__.py", line 645, in get return super().get(url) File "/Users/kristina/Library/Python/3.9/lib/python/site-packages/selenium/webdriver/remote/webdriver.py", line 449, in get self.execute(Command.GET, {"url": url}) File "/Users/kristina/Library/Python/3.9/lib/python/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute self.error_handler.check_response(response) File "/Users/kristina/Library/Python/3.9/lib/python/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools (failed to check if window was closed: disconnected: not connected to DevTools) (Session info: chrome=112.0.5615.49)

silasburger commented 1 year ago

I'm experiencing the same

taraskatsora1998 commented 1 year ago

I have the same issue too

codders commented 1 year ago

@silasburger @taraskatsora1998 can you share more about your setup? What platform are you running on? Are you running inside docker? How much RAM is available to the processes? What browser (Chrome or Chromium) are you using and at what version? Do you have the latest code?

cibanez-teladoc commented 1 year ago

Same issue here. Platform: macOS Ventura 13.3.1 (a) (22E772610a) Chrome driver: mac-arm64/114.0.5735.90 Chrome browser version: 114.0.5735.90 (Official Build) (arm64) RAM: 16 GB Running on my local machine.

arpitpaliwaldh commented 1 year ago

Facing same issue. Is there solution for this?

codders commented 1 year ago

@arpitpaliwaldh the solution is collecting more data. Can you confirm what platform you're using?

pembertonrw commented 1 year ago

I'm facing the same issue and I've got a similar setup to @cibanez-teladoc. I encounter this issue as soon as it's trying to interact with the page i.e. it goes to the webpage OK but as soon as it tryings to type something it, it fails.

Platform: macOS Ventura 13.4 (22F66) Chrome driver: mac-arm64/114.0.5735.90 Chrome browser version: 114.0.5735.106 (Official Build) (arm64) RAM: 16 GB Java: 11.0.11 Running on my local machine.

arpitpaliwaldh commented 1 year ago

Yes i think the issue is with the latest chrome version and chrome driver not compatible. Downgraded chrome and chromedriver version on my machine now test is working.

Platform: macOS Ventura 13.3 Chrome driver: 114.0.5735.90 Chrome browser version: 114.0.5735.106

miklyx commented 1 year ago

Hi! Actually - It's floating error. Sometimes it presents, sometimes process is working well for a couple of days. Also - after some fallings it may start a phantom "google chrome helper renderer" process with >100% cpu usage.

codders commented 1 year ago

@miklyx Thanks for the info. Are you also running on Mac with Chrome?

TomerLankri commented 1 year ago

Following having same error :(

codders commented 1 year ago

@TomerLankri Please share your platform details to help us with the debugging

codders commented 1 year ago

@TomerLankri Please share your platform details to help us with the debugging

TomerLankri commented 1 year ago

Platform: macOS Ventura 13.3.1 Chrome driver: mac-arm64/114.0.5735.90 Chrome browser version: 114.0.5735.90 RAM: 16 GB Running on my local machine.

miklyx commented 1 year ago

@miklyx Thanks for the info. Are you also running on Mac with Chrome?

I'm running latest mac OS with latest chrome

hubx commented 1 year ago

Just wanted to confirm that the Stacktrace is the same as above but was cut of and continues as the referenced issue #52

selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools (failed to check if window was closed: disconnected: not connected to DevTools) (Session info: chrome=114.0.5735.133) Stacktrace: 0 undetected_chromedriver 0x00000001050e56b8 undetected_chromedriver + 4937400 1 undetected_chromedriver 0x00000001050dcb73 undetected_chromedriver + 4901747 2 undetected_chromedriver 0x0000000104c9a616 undetected_chromedriver + 435734 3 undetected_chromedriver 0x0000000104c80bc1 undetected_chromedriver + 330689 4 undetected_chromedriver 0x0000000104c80923 undetected_chromedriver + 330019 5 undetected_chromedriver 0x0000000104c9c262 undetected_chromedriver + 442978 6 undetected_chromedriver 0x0000000104d1c56e undetected_chromedriver + 968046 7 undetected_chromedriver 0x0000000104d01de3 undetected_chromedriver + 859619 8 undetected_chromedriver 0x0000000104ccfd7f undetected_chromedriver + 654719 9 undetected_chromedriver 0x0000000104cd10de undetected_chromedriver + 659678 10 undetected_chromedriver 0x00000001050a12ad undetected_chromedriver + 4657837 11 undetected_chromedriver 0x00000001050a6130 undetected_chromedriver + 4677936 12 undetected_chromedriver 0x00000001050acdef undetected_chromedriver + 4705775 13 undetected_chromedriver 0x00000001050a705a undetected_chromedriver + 4681818 14 undetected_chromedriver 0x000000010507992c undetected_chromedriver + 4495660 15 undetected_chromedriver 0x00000001050c4838 undetected_chromedriver + 4802616 16 undetected_chromedriver 0x00000001050c49b7 undetected_chromedriver + 4802999 17 undetected_chromedriver 0x00000001050d599f undetected_chromedriver + 4872607 18 libsystem_pthread.dylib 0x00007ff814af31d3 _pthread_start + 125 19 libsystem_pthread.dylib 0x00007ff814aeebd3 thread_start + 15

For now what works for me is a good old bash loop that automatically retries while true; do python flathunt.py || sleep 5; done

miklyx commented 1 year ago

At fresh repo version and updated selenium and IMDBTraktSyncer I've got

[2023/06/26 23:16:17|patcher.py |INFO ]: patching driver executable /Users/mrashkovskiy/Library/Application Support/undetected_chromedriver/undetected_chromedriver Traceback (most recent call last): File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunt.py", line 99, in main() File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunt.py", line 95, in main launch_flat_hunt(config, heartbeat) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunt.py", line 35, in launch_flat_hunt hunter.hunt_flats() File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 56, in hunt_flats for expose in processor_chain.process(self.crawl_for_exposes(max_pages)): File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 35, in crawl_for_exposes return chain([try_crawl(searcher, url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 35, in return chain([try_crawl(searcher, url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 27, in try_crawl return searcher.crawl(url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/abstract_crawler.py", line 148, in crawl return self.get_results(url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/crawler/immobilienscout.py", line 90, in get_results soup = self.get_page(search_url, self.get_driver(), page_no) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/crawler/immobilienscout.py", line 65, in get_driver self.driver = get_chrome_driver(driver_arguments) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/chrome_wrapper.py", line 47, in get_chrome_driver driver = uc.Chrome(version_main=chrome_version, options=chrome_options) # pylint: disable=no-member File "/Users/mrashkovskiy/opt/anaconda3/lib/python3.9/site-packages/undetected_chromedriver/init.py", line 441, in init super(Chrome, self).init( TypeError: init() got an unexpected keyword argument 'executable_path'

miklyx commented 1 year ago

This is my pip list Package Version

aiohttp 3.8.3 aiosignal 1.3.1 alabaster 0.7.12 anaconda-client 1.9.0 anaconda-navigator 2.1.1 anaconda-project 0.10.1 antiorm 1.2.1 anyio 2.2.0 appdirs 1.4.4 applaunchservices 0.2.1 appnope 0.1.2 apprise 1.2.1 appscript 1.1.2 argh 0.26.2 argon2-cffi 20.1.0 arrow 0.13.1 asn1crypto 1.4.0 astroid 2.6.6 astropy 4.3.1 async-generator 1.10 async-timeout 4.0.2 atomicwrites 1.4.0 attrs 21.2.0 autopep8 1.5.7 Babel 2.9.1 backcall 0.2.0 backoff 2.2.1 backports.functools-lru-cache 1.6.4 backports.shutil-get-terminal-size 1.0.0 backports.tempfile 1.0 backports.weakref 1.0.post1 beautifulsoup4 4.10.0 binaryornot 0.4.4 bitarray 2.3.0 bkcharts 0.2 black 19.10b0 bleach 4.0.0 bokeh 2.4.1 boto 2.49.0 Bottleneck 1.3.2 brotlipy 0.7.0 cached-property 1.5.2 certifi 2021.10.8 cffi 1.14.6 chardet 4.0.0 charset-normalizer 2.0.4 click 8.0.3 cloudpickle 2.0.0 clyent 1.2.2 colorama 0.4.4 conda 4.10.3 conda-build 3.21.5 conda-content-trust 0+unknown conda-pack 0.6.0 conda-package-handling 1.7.3 conda-repo-cli 1.0.4 conda-token 0.3.0 conda-verify 3.4.2 contextlib2 0.6.0.post1 cookiecutter 1.7.2 cryptography 3.4.8 cycler 0.10.0 Cython 0.29.24 cytoolz 0.11.0 daal4py 2021.3.0 dask 2021.10.0 db 0.1.1 debugpy 1.4.1 decorator 5.1.0 defusedxml 0.7.1 diff-match-patch 20200713 distributed 2021.10.0 docutils 0.17.1 entrypoints 0.3 et-xmlfile 1.1.0 exceptiongroup 1.1.0 fastcache 1.1.0 feedparser 6.0.10 filelock 3.3.1 flake8 3.9.2 Flask 1.1.2 fonttools 4.25.0 frozenlist 1.3.3 fsspec 2021.8.1 future 0.18.2 gevent 21.8.0 glob2 0.7 gmpy2 2.0.8 greenlet 1.1.1 h11 0.14.0 h5py 3.2.1 HeapDict 1.0.1 html5lib 1.1 idna 3.2 imagecodecs 2021.8.26 imageio 2.9.0 imagesize 1.2.0 IMDBTraktSyncer 1.5.2 importlib-metadata 4.8.1 inflection 0.5.1 iniconfig 1.1.1 instaloader 4.9.6 intervaltree 3.1.0 ipykernel 6.4.1 ipython 7.29.0 ipython-genutils 0.2.0 ipywidgets 7.6.5 isort 5.9.3 itsdangerous 2.0.1 jdcal 1.4.1 jedi 0.18.0 Jinja2 2.11.3 jinja2-time 0.2.0 joblib 1.1.0 json5 0.9.6 jsonpath-ng 1.5.3 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.1.12 jupyter-console 6.4.0 jupyter-core 4.8.1 jupyter-server 1.4.1 jupyterlab 3.2.1 jupyterlab-pygments 0.1.2 jupyterlab-server 2.8.2 jupyterlab-widgets 1.0.0 keyring 23.1.0 kiwisolver 1.3.1 lazy-object-proxy 1.6.0 libarchive-c 2.9 llvmlite 0.37.0 locket 0.2.1 lxml 4.6.3 Markdown 3.4.1 MarkupSafe 1.1.1 matplotlib 3.4.3 matplotlib-inline 0.1.2 mccabe 0.6.1 mistune 0.8.4 mkl-fft 1.3.1 mkl-random 1.2.2 mkl-service 2.4.0 mock 4.0.3 more-itertools 8.10.0 mpmath 1.2.1 msgpack 1.0.2 multidict 6.0.2 multipledispatch 0.6.0 munkres 1.1.4 mypy-extensions 0.4.3 navigator-updater 0.2.1 nbclassic 0.2.6 nbclient 0.5.3 nbconvert 6.1.0 nbformat 5.1.3 nest-asyncio 1.5.1 networkx 2.6.3 nltk 3.6.5 nose 1.3.7 notebook 6.4.5 numba 0.54.1 numexpr 2.7.3 numpy 1.20.3 numpydoc 1.1.0 oauthlib 3.2.2 olefile 0.46 openpyxl 3.0.9 outcome 1.2.0 packaging 21.0 pandas 1.3.4 pandocfilters 1.4.3 parso 0.8.2 partd 1.2.0 path 16.0.0 pathlib2 2.3.6 pathspec 0.7.0 patsy 0.5.2 pep8 1.7.1 pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.4.0 pip 23.1.2 pkginfo 1.7.1 pluggy 0.13.1 ply 3.11 poyo 0.5.0 prometheus-client 0.11.0 prompt-toolkit 3.0.20 psutil 5.8.0 psycopg2-binary 2.9.5 ptyprocess 0.7.0 py 1.10.0 pycodestyle 2.7.0 pycosat 0.6.3 pycparser 2.20 pycurl 7.44.1 pydocstyle 6.1.1 pyerfa 2.0.0 pyflakes 2.3.1 Pygments 2.10.0 PyJWT 2.1.0 pylint 2.9.6 pyls-spyder 0.4.0 pyodbc 4.0.0-unsupported pyOpenSSL 21.0.0 pyparsing 3.0.4 pyrsistent 0.18.0 PySocks 1.7.1 pytest 6.2.4 python-dateutil 2.8.2 python-dotenv 0.21.1 python-lsp-black 1.0.0 python-lsp-jsonrpc 1.0.0 python-lsp-server 1.2.4 python-slugify 5.0.2 pytz 2021.3 PyWavelets 1.1.1 PyYAML 6.0 pyzmq 22.2.1 QDarkStyle 3.0.2 qstylizer 0.1.10 QtAwesome 1.0.2 qtconsole 5.1.1 QtPy 1.10.0 random-user-agent 1.0.1 regex 2021.8.3 requests 2.26.0 requests-oauthlib 1.3.1 requests-random-user-agent 2023.2.15 rope 0.19.0 Rtree 0.9.7 ruamel-yaml-conda 0.15.100 scikit-image 0.18.3 scikit-learn 0.24.2 scikit-learn-intelex 2021.20210714.100439 scipy 1.7.1 seaborn 0.11.2 selenium 4.10.0 Send2Trash 1.8.0 setuptools 58.0.4 sgmllib3k 1.0.0 simplegeneric 0.8.1 singledispatch 3.7.0 sip 4.19.13 six 1.16.0 sniffio 1.2.0 snowballstemmer 2.1.0 sortedcollections 2.1.0 sortedcontainers 2.4.0 soupsieve 2.2.1 spacexpy 2.0.2 Sphinx 4.2.0 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 sphinxcontrib-websupport 1.2.4 spyder 5.1.5 spyder-kernels 2.1.3 SQLAlchemy 1.4.22 statsmodels 0.12.2 sympy 1.9 tables 3.6.1 TBB 0.2 tblib 1.7.0 terminado 0.9.4 testpath 0.5.0 text-unidecode 1.3 textdistance 4.2.1 threadpoolctl 2.2.0 three-merge 0.1.1 tifffile 2021.7.2 tinycss 0.4 toml 0.10.2 toolz 0.11.1 tornado 6.1 tqdm 4.62.3 traitlets 5.1.0 trio 0.22.0 trio-websocket 0.9.2 typed-ast 1.4.3 typing-extensions 3.10.0.2 ujson 4.0.2 undetected-chromedriver 3.2.1 unicodecsv 0.14.1 Unidecode 1.2.0 urllib3 1.26.7 watchdog 2.1.3 wcwidth 0.2.5 webencodings 0.5.1 websockets 10.4 Werkzeug 2.0.2 wheel 0.37.0 whichcraft 0.6.1 widgetsnbextension 3.5.1 wrapt 1.12.1 wsproto 1.2.0 wurlitzer 2.1.1 xlrd 2.0.1 XlsxWriter 3.0.1 xlwings 0.24.9 xlwt 1.3.0 xmltodict 0.12.0 yapf 0.31.0 yarl 1.8.1 zict 2.0.0 zipp 3.6.0 zope.event 4.5.0 zope.interface 5.4.0

codders commented 1 year ago

Hi @miklyx ,

Not sure what IMDBTraktSyncer has to do with it - that's not in the pipfile. Are you using pipenv to install the dependencies?

codders commented 1 year ago

If you're not using pipenv to run the project, there can be unexpected interactions with other libraries installed. I would strongly recommend using pipenv install to install dependencies and pipenv shell to launch the project, to ensure you're using the same versions of all the libraries as other users of the project.

miklyx commented 1 year ago

Sorry, with pipenv it still works. I'll write if it'll be an errors

miklyx commented 1 year ago

With pipenv - It worked yesterday all day and dropped today - kinda record! no falls in 24 hours. MacOs Ventura 13.4.1, Python 3.9.7, all other from pipenv. trace looks the same

`[2023/06/28 10:30:29|immobilienscout.py |DEBUG ]: Got search URL https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?haspromotion=false&price=-1100.0&livingspace=50.0-&exclusioncriteria=projectlisting,swapflat&pricetype=rentpermonth&geocodes=110000000703,110000000101,110000000201,110000000102,110000000301,110000000202,110000000401,110000000105,110000000701&sorting=2&enteredFrom=result_list&pagenumber={0} Traceback (most recent call last): File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunt.py", line 99, in main() File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunt.py", line 95, in main launch_flat_hunt(config, heartbeat) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunt.py", line 44, in launch_flat_hunt hunter.hunt_flats() File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 56, in hunt_flats for expose in processor_chain.process(self.crawl_for_exposes(max_pages)): File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 35, in crawl_for_exposes return chain([try_crawl(searcher, url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 35, in return chain([try_crawl(searcher, url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/hunter.py", line 27, in try_crawl return searcher.crawl(url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/abstract_crawler.py", line 148, in crawl return self.get_results(url, max_pages) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/crawler/immobilienscout.py", line 90, in get_results soup = self.get_page(search_url, self.get_driver(), page_no) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/crawler/immobilienscout.py", line 175, in get_page return self.get_soup_from_url( File "/Users/mrashkovskiy/.local/share/virtualenvs/flathunter-N5-Pi6pW/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/Users/mrashkovskiy/Documents/py/flathunt/flathunt_2_0/flathunter/flathunter/abstract_crawler.py", line 70, in get_soup_from_url driver.get(url) File "/Users/mrashkovskiy/.local/share/virtualenvs/flathunter-N5-Pi6pW/lib/python3.9/site-packages/undetected_chromedriver/init.py", line 473, in get_wrapped if self.execute_script("return navigator.webdriver"): File "/Users/mrashkovskiy/.local/share/virtualenvs/flathunter-N5-Pi6pW/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 406, in execute_script return self.execute(command, {"script": script, "args": converted_args})["value"] File "/Users/mrashkovskiy/.local/share/virtualenvs/flathunter-N5-Pi6pW/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 346, in execute self.error_handler.check_response(response) File "/Users/mrashkovskiy/.local/share/virtualenvs/flathunter-N5-Pi6pW/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools (failed to check if window was closed: disconnected: not connected to DevTools) (Session info: chrome=114.0.5735.133) Stacktrace: 0 undetected_chromedriver 0x000000010536f6b8 undetected_chromedriver + 4937400 1 undetected_chromedriver 0x0000000105366b73 undetected_chromedriver + 4901747 2 undetected_chromedriver 0x0000000104f24616 undetected_chromedriver + 435734 3 undetected_chromedriver 0x0000000104f0abc1 undetected_chromedriver + 330689 4 undetected_chromedriver 0x0000000104f0a923 undetected_chromedriver + 330019 5 undetected_chromedriver 0x0000000104f26262 undetected_chromedriver + 442978 6 undetected_chromedriver 0x0000000104fa656e undetected_chromedriver + 968046 7 undetected_chromedriver 0x0000000104f8bde3 undetected_chromedriver + 859619 8 undetected_chromedriver 0x0000000104f59d7f undetected_chromedriver + 654719 9 undetected_chromedriver 0x0000000104f5b0de undetected_chromedriver + 659678 10 undetected_chromedriver 0x000000010532b2ad undetected_chromedriver + 4657837 11 undetected_chromedriver 0x0000000105330130 undetected_chromedriver + 4677936 12 undetected_chromedriver 0x0000000105336def undetected_chromedriver + 4705775 13 undetected_chromedriver 0x000000010533105a undetected_chromedriver + 4681818 14 undetected_chromedriver 0x000000010530392c undetected_chromedriver + 4495660 15 undetected_chromedriver 0x000000010534e838 undetected_chromedriver + 4802616 16 undetected_chromedriver 0x000000010534e9b7 undetected_chromedriver + 4802999 17 undetected_chromedriver 0x000000010535f99f undetected_chromedriver + 4872607 18 libsystem_pthread.dylib 0x00007ff8019531d3 _pthread_start + 125 19 libsystem_pthread.dylib 0x00007ff80194ebd3 thread_start + 15

[2023/06/28 10:30:29|init.py |INFO ]: ensuring close`

Lamiamrad commented 1 year ago

Hello Guys,

I am facing the same issue as yours, I am working under MacBook pro MacOs 13.4.1.

did u get any chance to fix this issue ?

Thanks :)

rjborba commented 1 year ago

Hey guys. Same issue for M1 Pro Ventura 13.4.1

Johnny0405 commented 1 year ago

Hey Guys, i am facing the same Problem. MacBook pro MacOs 13.4.1. When I remove "--remote-debugging-port=9222" it works.

miklyx commented 1 year ago

Actually that works! No fails all the day

miklyx commented 1 year ago

Early I was delighted - it fell. But it worked for almost 2 days

ManiRenukuntla commented 1 year ago

Hi Guys,

After chrome browser updated to v115. I have faced same issue.

my system config is: MAC M1 chip selenium == 4.11.0

I'm able to execute it on my PC using "locust -f ./locust.py -u 1 -r 1 --headless" But I couldn't execute it in the Docker container on my machine.

I'm facing Devtools error. ERROR/builderUI_Test: BuilderUI_loginFlowMessage: unknown error: Chrome failed to start: crashed. performance-slave-1 | (unknown error: DevToolsActivePort file doesn't exist) performance-slave-1 | (The process started from chrome location /usr/bin/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) performance-slave-1 | Stacktrace: performance-slave-1 | #0 0x0040006a7613 performance-slave-1 | #1 0x0040003d3537 performance-slave-1 | #2 0x0040003fe588 performance-slave-1 | #3 0x0040003fb5d9 performance-slave-1 | #4 0x00400043b643 performance-slave-1 | #5 0x00400043ad8f performance-slave-1 | #6 0x004000432803 performance-slave-1 | #7 0x00400040668b performance-slave-1 | #8 0x00400040742e performance-slave-1 | #9 0x004000668c28 performance-slave-1 | #10 0x00400066caf7 performance-slave-1 | #11 0x00400067701c performance-slave-1 | #12 0x00400066d723 performance-slave-1 | #13 0x00400063c457 performance-slave-1 | #14 0x004000691628 performance-slave-1 | #15 0x0040006917b7 performance-slave-1 | #16 0x0040006a0483 performance-slave-1 | #17 0x004002550ea7 start_thread performance-slave-1 |

filipopo commented 1 year ago

Just ran into this issue, it seems that the argument --disable-gpu Was causing this DevTools error for me, removing it made Chrome work for me

codders commented 1 year ago

@filipopo - Thanks for the report! Does that work for any of the other Mac OS users? (I only have Linux to test on here).

A1exMiranda commented 1 year ago

I'm facing the same problem on a Linux server. It seems to be working when restarting the server but after a few cycles the problem reappears, I'll try removing the argument "--disable-gpu" and tell you if it works.

A1exMiranda commented 12 months ago

Error still persists after a few hours of running right, "--disable-gpu" didn't seem to fix the issue for me.

The error I'm getting is: selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools (failed to check if window was closed: disconnected: not connected to DevTools) (Session info: headless chrome=114.0.5735.90)

These are my actual arguments: chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--window-size=1920,1080') chrome_options.add_argument('--headless') chrome_options.add_argument('--disable-dev-shm-usage') chrome_options.add_argument('--remote-debugging-port=9222') driver = webdriver.Chrome("/usr/local/bin/chromedriver",options=chrome_options)

anees1221 commented 10 months ago

I have the same problem I am using windows 10 I started experiencing the same problem when updated chrome browser to 115. All the later updates have the same problem. Today I have installed chrome 118. I though it would be fixed but the same problem persists.

A1exMiranda commented 10 months ago

I've found a version that seems to work, I've been testing it for a couple of weeks.

Try with Google Chrome for Testing 117.0.5938.62 and ChromeDriver 117.0.5938.62.

These are my arguments: chrome_options = webdriver.ChromeOptions() service = Service('/usr/local/bin/chromedriver') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--headless') chrome_options.add_argument('--remote-debugging-port=9222') chrome_options.add_argument('--disable-gpu') driver = webdriver.Chrome(service=service,options=chrome_options)

anees1221 commented 10 months ago

@A1exMiranda Thank you for your reply, it definitely it works with any version of Google Chrome for Testing

anees1221 commented 9 months ago

I've found a version that seems to work, I've been testing it for a couple of weeks.

Try with Google Chrome for Testing 117.0.5938.62 and ChromeDriver 117.0.5938.62.

These are my arguments: chrome_options = webdriver.ChromeOptions() service = Service('/usr/local/bin/chromedriver') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--headless') chrome_options.add_argument('--remote-debugging-port=9222') chrome_options.add_argument('--disable-gpu') driver = webdriver.Chrome(service=service,options=chrome_options)

@A1exMiranda again facing the same issue with Google Chrome for Testing.

kothaaridarshi commented 8 months ago

@anees1221 @A1exMiranda I am facing the same issue did you able to fix it?

anees1221 commented 7 months ago

@kothaaridarshi Unfortunately no.

codders commented 7 months ago

Mandatory XKCD: https://xkcd.com/2881

A1exMiranda commented 7 months ago

Try updating selenium too. The version I have is 4.12.0. I'm getting new updates of chrome and chromedriver and I've been running with no issues for the past 3 months.

ChromeDriver 119.0.6045.105 Google Chrome for Testing 119.0.6045.105

I really hope upgrading fixes your issues.

nienienienie commented 5 months ago

same here

tsbxmw commented 4 months ago

same in win10 & chrome 124 & java 11 & selenium 4.19.1 same .jar file, at agent A, it always failed. but another agent works well.

(failed to check if window was closed: disconnected: not connected to DevTools)
  (Session info: chrome=124.0.6367.78)
Build info: version: '4.19.1', revision: 'abe0ee07dc'
System info: os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '11.0.16'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [c65f8d45f6d213834ce8053b8c32abe8, get {url=https://www.google.com}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 124.0.6367.78, chrome: {chromedriverVersion: 124.0.6367.78 (a087f2dd364d..., userDataDir: C:\Users\TP-CESHI\AppData\L...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:61009}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: windows, proxy: Proxy(), se:cdp: ws://localhost:61009/devtoo..., se:cdpVersion: 124.0.6367.78, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: c65f8d45f6d213834ce8053b8c32abe8
    at jdk.internal.reflect.GeneratedConstructorAccessor63.newInstance(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.openqa.selenium.remote.ErrorCodec.decode(ErrorCodec.java:167)
    at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:138)
    at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:50)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:190)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:216)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:174)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:518)
    at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:300)
    at com.taobao.uiauto.service.selenium.action.AbstractAction.pageGoto(AbstractAction.java:523)
    at com.taobao.uiauto.service.selenium.action.system.PageGoto.run(PageGoto.java:70)
    at com.taobao.uiauto.service.selenium.ActionsRunner.run(ActionsRunner.java:54)
    at com.taobao.uiauto.service.service.impl.BrowserImpl.executeCommandAsync(BrowserImpl.java:138)
    at jdk.internal.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
steelliberty commented 4 months ago

Help- I need this not to happen.

MacOS : Monterey M1 version 12.6 selenium==4.20.0

options.headless = True from selenium import webdriver from selenium.webdriver.chrome.service import Service as ChromeService from webdriver_manager.chrome import ChromeDriverManager driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

Error fetching page content: Message: disconnected: not connected to DevTools (failed to check if window was closed: disconnected: not connected to DevTools) (Session info: chrome=124.0.6367.119)

Stacktrace: 0 chromedriver 0x00000001024fda18 chromedriver + 5216792 1 chromedriver 0x00000001024f5653 chromedriver + 5183059 2 chromedriver 0x0000000102067527 chromedriver + 406823 3 chromedriver 0x000000010204d59e chromedriver + 300446 4 chromedriver 0x000000010204d493 chromedriver + 300179 5 chromedriver 0x00000001020698b2 chromedriver + 415922 6 chromedriver 0x00000001020f405b chromedriver + 983131 7 chromedriver 0x00000001020d54a3 chromedriver + 857251 8 chromedriver 0x00000001020a5fe3 chromedriver + 663523 9 chromedriver 0x00000001020a692e chromedriver + 665902 10 chromedriver 0x00000001024bb930 chromedriver + 4946224 11 chromedriver 0x00000001024c19e4 chromedriver + 4970980 12 chromedriver 0x000000010249c42e chromedriver + 4817966 13 chromedriver 0x00000001024c24f9 chromedriver + 4973817 14 chromedriver 0x000000010248c6b4 chromedriver + 4753076 15 chromedriver 0x00000001024e4ba8 chromedriver + 5114792 16 chromedriver 0x00000001024e4d67 chromedriver + 5115239 17 chromedriver 0x00000001024f5273 chromedriver + 5182067 18 libsystem_pthread.dylib 0x00007ff80e6814e1 _pthread_start + 125 19 libsystem_pthread.dylib 0x00007ff80e67cf6b thread_start + 15