search

umihico / docker-selenium-lambda

The simplest demo of chrome automation by python and selenium in AWS Lambda
MIT License
535 stars 127 forks source link

WebDriverException: Message: unknown error: Chrome failed to start: crashed. #180

Closed Turkmen1Mehmet closed 1 year ago

Turkmen1Mehmet commented 1 year ago

@umihico Dockerfile:

FROM --platform=linux/amd64 public.ecr.aws/lambda/python@sha256:7f867f958f685d4321e4520e20b13900a47210ee57b3e41d05395b11f4c93c70 as build

ENV DOCKER_DEFAULT_PLATFORM=linux/amd64

RUN yum install -y unzip && \
    curl -Lo "/tmp/chromedriver.zip" "https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip" && \
    curl -Lo "/tmp/chrome-linux.zip" "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Linux_x64%2F1135561%2Fchrome-linux.zip?alt=media" && \
    unzip /tmp/chromedriver.zip -d /opt/ && \
    unzip /tmp/chrome-linux.zip -d /opt/

FROM --platform=linux/amd64 public.ecr.aws/lambda/python@sha256:7f867f958f685d4321e4520e20b13900a47210ee57b3e41d05395b11f4c93c70
RUN yum install atk cups-libs gtk3 libXcomposite alsa-lib \
    libXcursor libXdamage libXext libXi libXrandr libXScrnSaver \
    libXtst pango at-spi2-atk libXt xorg-x11-server-Xvfb \
    xorg-x11-xauth dbus-glib dbus-glib-devel -y
RUN pip install selenium==4.10.0
COPY --from=build /opt/chrome-linux /opt/chrome
COPY --from=build /opt/chromedriver /opt/
COPY main.py ./
CMD [ "main.handler" ]

serverless.yml

service: docker-selenium-lambda-memo

provider:
  name: aws
  stage: ${opt:stage, 'prod'}
  region: ${env:AWS_REGION, 'us-east-1'}
  iam:
    role: arn:aws:iam::792740507892:role/docker-selenium-lambda-memo-prod-us-east-1-lambdaRole
  ecr:
    images:
      img:
        path: ./
        platform: linux/amd64

functions:
  demo:
    timeout: 60
    memorySize: 3008
    image:
      name: img

main.py

from selenium import webdriver
from tempfile import mkdtemp
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.color import Color
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.chrome.options import Options

def handler(event=None, context=None):
    chromedriver_path = "/opt/chromedriver"
    op = webdriver.ChromeOptions()
    op.binary_location = '/opt/chrome/chrome'
    op.add_argument('--headless')
    op.add_argument('--no-sandbox')
    op.add_argument("--disable-gpu")
    op.add_argument("--window-size=1280x1696")
    op.add_argument("--single-process")
    op.add_argument("--disable-dev-shm-usage")
    op.add_argument("--disable-dev-tools")
    op.add_argument("--no-zygote")
    op.add_argument(f"--user-data-dir={mkdtemp()}")
    op.add_argument(f"--data-path={mkdtemp()}")
    op.add_argument(f"--disk-cache-dir={mkdtemp()}")
    op.add_argument("--remote-debugging-port=9222")
    op.add_argument(f"--executable-path={chromedriver_path}")

    browser = webdriver.Chrome(options=op)
    browser.get("https://www.zara.com/tr/")
    return browser.find_element(by=By.XPATH, value="//html").text

the ones i run in terminal respectively TERMİNAL:

docker build -t my-selenium-lambda .

docker run -p 8080:8080 --platform=linux/amd64 --memory=8g --cpus=2 my-selenium-lambda

curl -XPOST "http://localhost:8080/2015-03-31/functions/function/invocations" -d '{}'

mehmetturkmen@Mehmet-MacBook-Pro docker-selenium-lambda-memo % curl -XPOST "http://localhost:8080/2015-03-31/functions/function/invocations" -d '{}' {"errorMessage": "Message: unknown error: Chrome failed to start: crashed.\n (chrome not reachable)\n (The process started from chrome location /opt/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\nStacktrace:\n#0 0x0040006d74e3 \n#1 0x004000406c76 \n#2 0x00400042fd78 \n#3 0x00400042c029 \n#4 0x00400046accc \n#5 0x00400046a47f \n#6 0x004000461de3 \n#7 0x0040004372dd \n#8 0x00400043834e \n#9 0x0040006973e4 \n#10 0x00400069b3d7 \n#11 0x0040006a5b20 \n#12 0x00400069c023 \n#13 0x00400066a1aa \n#14 0x0040006c06b8 \n#15 0x0040006c0847 \n#16 0x0040006d0243 \n#17 0x004002aaf44b start_thread\n", "errorType": "WebDriverException", "requestId": "e30cb129-4e33-4000-916e-0aa0c9544f31", "stackTrace": [" File \"/var/task/main.py\", line 28, in handler\n browser = webdriver.Chrome(options=op)\n", " File \"/var/lang/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriver.py\", line 49, in init\n super().init(\n", " File \"/var/lang/lib/python3.10/site-packages/selenium/webdriver/chromium/webdriver.py\", line 54, in init\n super().init(\n", " File \"/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py\", line 206, in init\n self.start_session(capabilities)\n", " File \"/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py\", line 291, in start_session\n response = self.execute(Command.NEW_SESSION, caps)[\"value\"]\n", " File \"/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py\", line 346, in execute\n self.error_handler.check_response(response)\n", " File \"/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py\", line 245, in check_response\n raise exception_class(message, screen, stacktrace)\n"]}%

mehmetturkmen@Mehmet-MacBook-Pro docker-selenium-lambda-memo % docker run -p 8080:8080 --platform=linux/amd64 --memory=8g --cpus=2 my-selenium-lambda

04 Jul 2023 13:14:11,935 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/var/task, handler=) 04 Jul 2023 13:14:17,540 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory 04 Jul 2023 13:14:17,541 [INFO] (rapid) Configuring and starting Operator Domain 04 Jul 2023 13:14:17,541 [INFO] (rapid) Starting runtime domain 04 Jul 2023 13:14:17,542 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory 04 Jul 2023 13:14:17,546 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false START RequestId: 6ec38ac0-d9ca-45ac-a5a1-bdd927889297 Version: $LATEST [ERROR] WebDriverException: Message: unknown error: Chrome failed to start: crashed. (chrome not reachable) (The process started from chrome location /opt/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace:

0 0x0040006d74e3

1 0x004000406c76

2 0x00400042fd78

3 0x00400042c029

4 0x00400046accc

5 0x00400046a47f

6 0x004000461de3

7 0x0040004372dd

8 0x00400043834e

9 0x0040006973e4

10 0x00400069b3d7

11 0x0040006a5b20

12 0x00400069c023

13 0x00400066a1aa

14 0x0040006c06b8

15 0x0040006c0847

16 0x0040006d0243

17 0x004002aaf44b start_thread

  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/chromium/webdr  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/errorha    raise exception_class(message, screen, stacktrace) END RequestId: e30cb129-4e33-4000-916e-0aa0c9544f31 REPORT RequestId: e30cb129-4e33-4000-916e-0aa0c9544f31 Init Duration: 3.25 ms Duration: 15149.74 ms Billed Duration: 15150 ms Memory Size: 3008 MB Max Memory Used: 3008 MB

The error I get is this way, when I look at the files in docker, chromedriver and opt/chrome/chrome appear there, but I keep getting the same error.

umihico commented 1 year ago

@Turkmen1Mehmet I saw some modifications, but they does not seem to be related. Because I could simply reproduce the error by following these steps

  1. Modify URL from example.com to www.zara.com/tr
  2. Run thin command in terminal A: docker build -t my-selenium-lambda . && docker run -p 8080:8080 --platform=linux/amd64 --memory=8g --cpus=2 my-selenium-lambda
  3. Run thin command in terminal B: curl -XPOST "http://localhost:8080/2015-03-31/functions/function/invocations" -d '{}'

I could solve this error to increase memory limit, which is from 8g to 16g. But the response does not look good for your purpose. It was "Access Denied\nYou don't have permission to access \"http://www.zara.com/tr\" on this server.\nReference

Reopen this if you still have the error.

BTW you shouldn't share your AWS account ID