Open rebeccacremona opened 3 years ago
Not sure if this is the same thing, but I just observed a worker fail and saw this in the syslog
Nov 9 19:05:22 ip-172-31-48-121 kernel: [73659.897341] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/perma-worker.service,task=chromium,pid=351497,uid=1001
Nov 9 19:05:22 ip-172-31-48-121 systemd[1]: perma-worker.service: A process of this unit has been killed by the OOM killer.
Nov 9 19:05:29 ip-172-31-48-121 systemd[1]: perma-worker.service: Failed with result 'oom-kill'.
Nov 9 19:05:29 ip-172-31-48-121 systemd[1]: perma-worker.service: Consumed 1h 20min 49.527s CPU time.
Possibly due to a new version of Chromium, 112.0.5615.138-1~deb11u1, we just saw a new error,
WebDriverException
Message: unknown error: unhandled inspector error: {"code":-32000,"message":"Unable to capture screenshot"}
(Session info: headless chrome=112.0.5615.138)
(Driver info: chromedriver=112.0.5615.138 (2258f2fcaeb07bc95c992b7ca47f0b43ead9865c-refs/branch-heads/5615@{#1282}),platform=Linux 5.10.0-21-cloud-amd64 x86_64)
Django Version: 3.2.18
Python Executable: /usr/local/share/.virtualenvs/perma/bin/python3
Python Version: 3.9.2
Server time: Mon, 24 Apr 2023 16:01:27 +0000
Traceback:
File "/usr/local/share/perma/perma_web/perma/celery_tasks.py" in run_next_capture
1386. screenshot = get_screenshot(link, browser)
File "/usr/local/share/perma/perma_web/perma/celery_tasks.py" in get_screenshot
738. return browser.get_screenshot_as_png()
File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py" in get_screenshot_as_png
777. return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py" in get_screenshot_as_base64
787. return self.execute(Command.SCREENSHOT)['value']
File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py" in execute
201. self.error_handler.check_response(response)
File "/usr/local/share/.virtualenvs/perma/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py" in check_response
181. raise exception_class(message, screen, stacktrace)
Exception Type: WebDriverException
Exception Value: Message: unknown error: unhandled inspector error: {"code":-32000,"message":"Unable to capture screenshot"}
(Session info: headless chrome=112.0.5615.138)
(Driver info: chromedriver=112.0.5615.138 (2258f2fcaeb07bc95c992b7ca47f0b43ead9865c-refs/branch-heads/5615@{#1282}),platform=Linux 5.10.0-21-cloud-amd64 x86_64)
Examples:
While taking screenshots (
screenshot = get_screenshot(link, browser)
)While retrieving the DOM (
dom_tree = get_dom_tree(browser)
)While launching (
get_browser
)