Closed abuchmueller closed 1 year ago
Seems to be an issue with docker cache, I had to clear docker build cache by using --no-cache
as in docker build -t flathunter . --no-cache
Seems to be an issue with docker cache, I had to clear docker build cache by using
--no-cache
as indocker build -t flathunter . --no-cache
Thanks, had the same Error. Which driver arguments do you use? I still get a Chrome is not reachable error.
Seems to be an issue with docker cache, I had to clear docker build cache by using
--no-cache
as indocker build -t flathunter . --no-cache
Thanks, had the same Error. Which driver arguments do you use? I still get a Chrome is not reachable error.
driver_arguments:
- "--headless"
- "--no-sandbox"
- "--disable-dev-shm-usage"
It does crash sometimes, I run my containers with the--restart-on-failure
flag tho so this is not an issue.
After re-building the docker image with --no-cache
because of this error, I now get selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
after receiving a captcha result from 2captcha for crawling Immoscout:
flathunter | [2022/09/04 12:46:30|twocaptcha_solver.py |DEBUG ]: Got response from 2captcha/res: CAPCHA_NOT_READY
flathunter | [2022/09/04 12:46:30|twocaptcha_solver.py |INFO ]: Captcha is not ready yet, waiting...
flathunter | [2022/09/04 12:46:36|twocaptcha_solver.py |DEBUG ]: Got response from 2captcha/res: CAPCHA_NOT_READY
flathunter | [2022/09/04 12:46:36|twocaptcha_solver.py |INFO ]: Captcha is not ready yet, waiting...
flathunter | [2022/09/04 12:46:41|twocaptcha_solver.py |DEBUG ]: Got response from 2captcha/res: OK|{"geetest_challenge":"4a0b13ec8c86b3d9d9e51a31e607d809","geetest_validate":"09c594631ab2d4f55453eb324eced0cb","geetest_seccode":"09c594631ab2d4f55453eb324eced0cb|jordan"}
flathunter | Traceback (most recent call last):
flathunter | File "flathunt.py", line 110, in <module>
flathunter | main()
flathunter | File "flathunt.py", line 106, in main
flathunter | launch_flat_hunt(config, heartbeat)
flathunter | File "flathunt.py", line 30, in launch_flat_hunt
flathunter | hunter.hunt_flats()
flathunter | File "/usr/src/app/flathunter/hunter.py", line 54, in hunt_flats
flathunter | for expose in processor_chain.process(self.crawl_for_exposes(max_pages)):
flathunter | File "/usr/src/app/flathunter/hunter.py", line 34, in crawl_for_exposes
flathunter | for searcher in self.config.searchers()
flathunter | File "/usr/src/app/flathunter/hunter.py", line 35, in <listcomp>
flathunter | for url in self.config.target_urls()])
flathunter | File "/usr/src/app/flathunter/hunter.py", line 25, in try_crawl
flathunter | return searcher.crawl(url, max_pages)
flathunter | File "/usr/src/app/flathunter/abstract_crawler.py", line 159, in crawl
flathunter | return self.get_results(url, max_pages)
flathunter | File "/usr/src/app/flathunter/crawl_immobilienscout.py", line 56, in get_results
flathunter | soup = self.get_page(search_url, self.driver, page_no)
flathunter | File "/usr/src/app/flathunter/crawl_immobilienscout.py", line 138, in get_page
flathunter | afterlogin_string=self.afterlogin_string
flathunter | File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 105, in retry
flathunter | ret = target(*args, **kwargs)
flathunter | File "/usr/src/app/flathunter/abstract_crawler.py", line 94, in get_soup_from_url
flathunter | return BeautifulSoup(driver.page_source, 'html.parser')
flathunter | File "/usr/local/lib/python3.7/site-packages/undetected_chromedriver/__init__.py", line 433, in __getattribute__
flathunter | return super().__getattribute__(item)
flathunter | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 539, in page_source
flathunter | return self.execute(Command.GET_PAGE_SOURCE)['value']
flathunter | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 428, in execute
flathunter | self.error_handler.check_response(response)
flathunter | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
flathunter | raise exception_class(message, screen, stacktrace)
flathunter | selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
flathunter | from unknown error: cannot determine loading status
flathunter | from tab crashed
flathunter | (Session info: headless chrome=105.0.5195.102)
flathunter | Stacktrace:
flathunter | #0 0x55eb1095b693 <unknown>
flathunter | #1 0x55eb107549db <unknown>
flathunter | #2 0x55eb107410eb <unknown>
flathunter | #3 0x55eb1073ffd0 <unknown>
flathunter | #4 0x55eb1074050c <unknown>
flathunter | #5 0x55eb1074cdff <unknown>
flathunter | #6 0x55eb1074d992 <unknown>
flathunter | #7 0x55eb1075c452 <unknown>
flathunter | #8 0x55eb10760324 <unknown>
flathunter | #9 0x55eb10740936 <unknown>
flathunter | #10 0x55eb1075bfe3 <unknown>
flathunter | #11 0x55eb107be388 <unknown>
flathunter | #12 0x55eb107aa743 <unknown>
flathunter | #13 0x55eb10780533 <unknown>
flathunter | #14 0x55eb10781715 <unknown>
flathunter | #15 0x55eb109ab7bd <unknown>
flathunter | #16 0x55eb109aebf9 <unknown>
flathunter | #17 0x55eb10990f2e <unknown>
flathunter | #18 0x55eb109af9b3 <unknown>
flathunter | #19 0x55eb10984e4f <unknown>
flathunter | #20 0x55eb109ceea8 <unknown>
flathunter | #21 0x55eb109cf052 <unknown>
flathunter | #22 0x55eb109e971f <unknown>
flathunter | #23 0x7f1527125ea7 <unknown>
Hi there,
I don't know if you've seen the other issues about Selenium Docker crashes, but do make sure that you have enough memory allocated to the container, and do try disabling SHM. That's helped other users get docker running reliably.
Arthur
Marcel Mindemann @.***> schrieb am So., 4. Sept. 2022, 11:48:
After re-building the docker image with --no-cache because of this error, I now get the following error after receiving a captcha result from 2captcha for crawling Immoscout:
flathunter | [2022/09/04 12:46:30|twocaptcha_solver.py |DEBUG ]: Got response from 2captcha/res: CAPCHA_NOT_READY flathunter | [2022/09/04 12:46:30|twocaptcha_solver.py |INFO ]: Captcha is not ready yet, waiting... flathunter | [2022/09/04 12:46:36|twocaptcha_solver.py |DEBUG ]: Got response from 2captcha/res: CAPCHA_NOT_READY flathunter | [2022/09/04 12:46:36|twocaptcha_solver.py |INFO ]: Captcha is not ready yet, waiting... flathunter | [2022/09/04 12:46:41|twocaptcha_solver.py |DEBUG ]: Got response from 2captcha/res: OK|{"geetest_challenge":"4a0b13ec8c86b3d9d9e51a31e607d809","geetest_validate":"09c594631ab2d4f55453eb324eced0cb","geetest_seccode":"09c594631ab2d4f55453eb324eced0cb|jordan"} flathunter | Traceback (most recent call last): flathunter | File "flathunt.py", line 110, in
flathunter | main() flathunter | File "flathunt.py", line 106, in main flathunter | launch_flat_hunt(config, heartbeat) flathunter | File "flathunt.py", line 30, in launch_flat_hunt flathunter | hunter.hunt_flats() flathunter | File "/usr/src/app/flathunter/hunter.py", line 54, in hunt_flats flathunter | for expose in processor_chain.process(self.crawl_for_exposes(max_pages)): flathunter | File "/usr/src/app/flathunter/hunter.py", line 34, in crawl_for_exposes flathunter | for searcher in self.config.searchers() flathunter | File "/usr/src/app/flathunter/hunter.py", line 35, in flathunter | for url in self.config.target_urls()]) flathunter | File "/usr/src/app/flathunter/hunter.py", line 25, in try_crawl flathunter | return searcher.crawl(url, max_pages) flathunter | File "/usr/src/app/flathunter/abstract_crawler.py", line 159, in crawl flathunter | return self.get_results(url, max_pages) flathunter | File "/usr/src/app/flathunter/crawl_immobilienscout.py", line 56, in get_results flathunter | soup = self.get_page(search_url, self.driver, page_no) flathunter | File "/usr/src/app/flathunter/crawl_immobilienscout.py", line 138, in get_page flathunter | afterlogin_string=self.afterlogin_string flathunter | File "/usr/local/lib/python3.7/site-packages/backoff/_sync.py", line 105, in retry flathunter | ret = target(*args, **kwargs) flathunter | File "/usr/src/app/flathunter/abstract_crawler.py", line 94, in get_soup_from_url flathunter | return BeautifulSoup(driver.page_source, 'html.parser') flathunter | File "/usr/local/lib/python3.7/site-packages/undetected_chromedriver/init.py", line 433, in getattribute flathunter | return super().getattribute(item) flathunter | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 539, in page_source flathunter | return self.execute(Command.GET_PAGE_SOURCE)['value'] flathunter | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 428, in execute flathunter | self.error_handler.check_response(response) flathunter | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response flathunter | raise exception_class(message, screen, stacktrace) flathunter | selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash flathunter | from unknown error: cannot determine loading status flathunter | from tab crashed flathunter | (Session info: headless chrome=105.0.5195.102) flathunter | Stacktrace: flathunter | #0 0x55eb1095b693 flathunter | #1 0x55eb107549db flathunter | #2 0x55eb107410eb flathunter | #3 0x55eb1073ffd0 flathunter | #4 0x55eb1074050c flathunter | #5 0x55eb1074cdff flathunter | #6 0x55eb1074d992 flathunter | #7 0x55eb1075c452 flathunter | #8 0x55eb10760324 flathunter | #9 0x55eb10740936 flathunter | #10 0x55eb1075bfe3 flathunter | #11 0x55eb107be388 flathunter | #12 0x55eb107aa743 flathunter | #13 0x55eb10780533 flathunter | #14 0x55eb10781715 flathunter | #15 0x55eb109ab7bd flathunter | #16 0x55eb109aebf9 flathunter | #17 0x55eb10990f2e flathunter | #18 0x55eb109af9b3 flathunter | #19 0x55eb10984e4f flathunter | #20 0x55eb109ceea8 flathunter | #21 0x55eb109cf052 flathunter | #22 0x55eb109e971f flathunter | #23 0x7f1527125ea7 — Reply to this email directly, view it on GitHub https://github.com/flathunters/flathunter/issues/213#issuecomment-1236310084, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEK5USMLNMZJICXF2KSHLV4R5A7ANCNFSM6AAAAAAQEAEIEA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
It's happening again. After 27 days my container stopped working and won't start any more.
Rebuilding with --no-cache
didn't do the trick this time.
`ChromeDriver only supports Chrome version 105
Current browser version is 107.0.5304.87`
Wasn't webdriver-manager
supposed to handle this? super annoying
I have exactly the same problem and flathunter doesnt start anymore. Anyone has a fix for this? Im not running this on Docker, so the "--no-chache" isnt helpful for me.
flathunter[124052]: [#033[94m2022/10/30 17:54:12#033[0m|#033[94mconfig.py #033[0m|#033[93mINFO #033[0m]: Using config path /opt/flathunter_new/config.yaml
flathunter[124052]: [#033[94m2022/10/30 17:54:12#033[0m|#033[94mlogging.py #033[0m|#033[93mDEBUG #033[0m]: Settings from config: <flathunter.config.Config object at 0x7f498b905fd0>
flathunter[124052]: [#033[94m2022/10/30 17:54:12#033[0m|#033[94mabstract_crawler.py #033[0m|#033[93mINFO #033[0m]: Initializing Chrome WebDriver for crawler "CrawlImmobilienscout"...
flathunter[124052]: [#033[94m2022/10/30 17:54:14#033[0m|#033[94mpatcher.py #033[0m|#033[93mINFO #033[0m]: patching driver executable /home/user/.local/share/undetected_chromedriver/d30dd413ab34505e_chromedriver
flathunter[124052]: Traceback (most recent call last):
flathunter[124052]: File "flathunt.py", line 109, in <module>
flathunter[124052]: main()
flathunter[124052]: File "flathunt.py", line 75, in main
flathunter[124052]: config.init_searchers()
flathunter[124052]: File "/opt/flathunter_new/flathunter/config.py", line 87, in init_searchers
flathunter[124052]: CrawlImmobilienscout(self),
flathunter[124052]: File "/opt/flathunter_new/flathunter/crawl_immobilienscout.py", line 39, in __init__
flathunter[124052]: self.driver = self.configure_driver(driver_arguments)
flathunter[124052]: File "/opt/flathunter_new/flathunter/abstract_crawler.py", line 59, in configure_driver
flathunter[124052]: driver = uc.Chrome(options=chrome_options) # pylint: disable=no-member
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/undetected_chromedriver/__init__.py", line 401, in __init__
flathunter[124052]: super(Chrome, self).__init__(
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
flathunter[124052]: super().__init__(DesiredCapabilities.CHROME['browserName'], "goog",
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
flathunter[124052]: super().__init__(
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 272, in __init__
flathunter[124052]: self.start_session(capabilities, browser_profile)
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/undetected_chromedriver/__init__.py", line 589, in start_session
flathunter[124052]: super(selenium.webdriver.chrome.webdriver.WebDriver, self).start_session(
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 364, in start_session
flathunter[124052]: response = self.execute(Command.NEW_SESSION, parameters)
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
flathunter[124052]: self.error_handler.check_response(response)
flathunter[124052]: File "/home/user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
flathunter[124052]: raise exception_class(message, screen, stacktrace)
flathunter[124052]: selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:60163
flathunter[124052]: from session not created: This version of ChromeDriver only supports Chrome version 107
flathunter[124052]: Current browser version is 106.0.5249.91
flathunter[124052]: Stacktrace:
flathunter[124052]: #0 0x55f9041e1463 <unknown>
flathunter[124052]: #1 0x55f903fa58d8 <unknown>
flathunter[124052]: #2 0x55f903fd23c6 <unknown>
flathunter[124052]: #3 0x55f903fc882f <unknown>
flathunter[124052]: #4 0x55f90400c802 <unknown>
flathunter[124052]: #5 0x55f90400c2af <unknown>
flathunter[124052]: #6 0x55f904004443 <unknown>
flathunter[124052]: #7 0x55f903fd53c5 <unknown>
flathunter[124052]: #8 0x55f903fd6531 <unknown>
flathunter[124052]: #9 0x55f904233dce <unknown>
flathunter[124052]: #10 0x55f904237192 <unknown>
flathunter[124052]: #11 0x55f90421893e <unknown>
flathunter[124052]: #12 0x55f904238103 <unknown>
flathunter[124052]: #13 0x55f90420bd85 <unknown>
flathunter[124052]: #14 0x55f9042590a8 <unknown>
flathunter[124052]: #15 0x55f904259239 <unknown>
flathunter[124052]: #16 0x55f904274492 <unknown>
flathunter[124052]: #17 0x7ff2a4230609 start_thread
Oct 30 17:54:19 systemd[1]: flathunter_new.service: Main process exited, code=exited, status=1/FAILURE
Oct 30 17:54:19 systemd[1]: flathunter_new.service: Failed with result 'exit-code'.
For me it works when I install chrome 105 by editing the Dockerfile. Replacing the "Install Google Chrome" Block by this:
# Install Google Chrome
ARG CHROME_VERSION="105.0.5195.102-1"
RUN apt-get -y update
RUN wget --no-verbose -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION}_amd64.deb \
&& apt install -y /tmp/chrome.deb \
&& rm /tmp/chrome.deb
And then building again
For me it works when I install chrome 105 by editing the Dockerfile. Replacing the "Install Google Chrome" Block by this:
# Install Google Chrome ARG CHROME_VERSION="105.0.5195.102-1" RUN apt-get -y update RUN wget --no-verbose -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION}_amd64.deb \ && apt install -y /tmp/chrome.deb \ && rm /tmp/chrome.deb
And then building again
It seems like you are using an older version of flathunter. Nowadays there is no need to manually install chrome driver, since the library "undetected-chromedriver" and "webdriver_manager" handles the installation. The problem mentioned in this Issue is connected to this new way of installing chromedriver (at least for me).
Or am i mistaken?
I cloned the repository yesterday 😄 The snippet does not install chromedriver, but the version of chrome that the chromedriver required. I had the same error that @abuchmueller posted 4 days ago, also using docker. The installed chromedriver required chrome 105, but apt installs a newer version.
I cloned the repository yesterday 😄 The snippet does not install chromedriver, but the version of chrome that the chromedriver required. I had the same error that @abuchmueller posted 4 days ago, also using docker. The installed chromedriver required chrome 105, but apt installs a newer version.
Ok thanks for the info.
Ive looked into the code and it seems like this line (in abstract_crawyler.py) was updated (my flathunter version is maybe 2 or 3 weeks old and this line was previously without the specific version 105):
driver = uc.Chrome(version_main=105, options=chrome_options)
Also since flathunter uses automatic chromedriver handling, they removed the section in the README where its mentioned to install chrome. It wasnt clear for me. The installation happens in the dockerfile like you said, but what about users who do not use docker (like me)? It should be mentioned in the README!
After updating the code (adding "version_main=105") and installing the version 105 of chrome, everything works for me again. Thanks @Andrusowski
I had the same problem (not using docker) because my chrome version updated to 107. I replaced "version_main=105" with "version_main=107" and now it works again.
For me it works when I install chrome 105 by editing the Dockerfile. Replacing the "Install Google Chrome" Block by this:
# Install Google Chrome ARG CHROME_VERSION="105.0.5195.102-1" RUN apt-get -y update RUN wget --no-verbose -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION}_amd64.deb \ && apt install -y /tmp/chrome.deb \ && rm /tmp/chrome.deb
And then building again
Thanks, with this you can easily rebuild with the correct chrome version when you notice drift between driver and chrome. I assume this happens because the container runs a couple of weeks chrome automatically updates? Do you know of a way way to forbid automatic updates or re-sync driver and chrome without re-building?
Yeah. That's all pretty inelegant. #252 would actually fix this in a more sustainable way - so that we have always the latest version of Chrome installed, and UC always selects the right version for the version pinning. But I'm still waiting for feedback on the PR. If you want to hack with docker files in the meantime, that's all good :)
I made a PR #254 based on the work from @numb95 in #252. This should now mean that the chromedriver version will always match the installed chrome version. Please pull the latest, and let me know if that doesn't fix it for you!
Are there people still struggling with this? Or can I close this after #254 ?
Are there people still struggling with this? Or can I close this after #254 ?
Too early to tell. Need for wait for the next chrome update and see what happens. Usually this would break the container.
Have the same problem, running it NOT in the docker.
Hi @SupaSweet ,
Are you running the latest code from the 'main' branch?
@codders, yes, I've pulled it yesterday. Actually it fails occasionally. Like yesterday after numerous attempts it finally started working. Today it worked for a while, then crashed with the same error. I've restarted it and it started working ok again.
@SupaSweet and the error message is "This version of ChromeDriver only supports Chrome version 107. Current browser version is 106.0.5249.91"? Or what is exactly the error message?
@codders,
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:54836
from session not created: This version of ChromeDriver only supports Chrome version 108
Current browser version is 107.0.5304.121
Do you have chrome and chromium installed? Are they on different versions? What is the output of:
google-chrome --version
and
chromium --version
Are there people still struggling with this? Or can I close this after #254 ?
Too early to tell. Need for wait for the next chrome update and see what happens. Usually this would break the container.
Quick update: Been running smoothly since November now. I think this can be closed.
Sweet - thanks! I honestly wonder if #290 might be connected to the solution here, but there's another ticket for that :)
After pulling #208, the there seems to be an issue with the Chromedriver again. The bot crashes immediately when trying to crawl Immobilienscout.