Superset Failing to Generate Thumbnails in Docker Compose Environment

mschaefer-gresham commented 4 days ago

Bug description

I've enabled thumbnails within the docker compose environment and I am getting this error repeating in the log on startup.

superset_worker       | [2024-09-24 09:09:40,506: INFO/ForkPoolWorker-1] Task cache_chart_thumbnail[62684bb1-a0c5-4c20-ba6a-6f522004edbf] succeeded in 1.1576069999864558s: None
superset_worker       | [2024-09-24 09:09:40,508: INFO/MainProcess] Task cache_chart_thumbnail[5859cfc0-e68a-4cc4-a9f0-2b5a30a77cc7] received
superset_worker       | [2024-09-24 09:09:40,510: INFO/ForkPoolWorker-1] Caching chart: http://superset:8088/superset/slice/1305/
superset_worker       | [2024-09-24 09:09:40,511: INFO/ForkPoolWorker-1] Processing url for thumbnail: bd0c91566fdc6a685058e6c065151327
superset_worker       | [2024-09-24 09:09:40,511: DEBUG/ForkPoolWorker-1] Init selenium driver
superset_worker       | [2024-09-24 09:09:41,552: WARNING/ForkPoolWorker-2] Failed at generating thumbnail Message: Process unexpectedly closed with status signal
superset_worker       | Traceback (most recent call last):
superset_worker       |   File "/app/superset/utils/screenshots.py", line 170, in compute_and_cache
superset_worker       |     payload = self.get_screenshot(user=user, window_size=window_size)
superset_worker       |   File "/app/superset/utils/screenshots.py", line 93, in get_screenshot
superset_worker       |     self.screenshot = driver.get_screenshot(self.url, self.element, user)
superset_worker       |   File "/app/superset/utils/webdriver.py", line 354, in get_screenshot
superset_worker       |     driver = self.auth(user)
superset_worker       |   File "/app/superset/utils/webdriver.py", line 276, in auth
superset_worker       |     driver = self.create()
superset_worker       |   File "/app/superset/utils/webdriver.py", line 273, in create
superset_worker       |     return driver_class(**kwargs)
superset_worker       |   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/firefox/webdriver.py", line 170, in __init__
superset_worker       |     RemoteWebDriver.__init__(
superset_worker       |   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
superset_worker       |     self.start_session(capabilities, browser_profile)
superset_worker       |   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
superset_worker       |     response = self.execute(Command.NEW_SESSION, parameters)
superset_worker       |   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
superset_worker       |     self.error_handler.check_response(response)
superset_worker       |   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
superset_worker       |     raise exception_class(message, screen, stacktrace)
superset_worker       | selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status signal

What am I missing?

How to reproduce the bug

Enable thumbnails feature flag.

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

18 or greater

Browser

Chrome

Additional context

No response

Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

dosubot[bot] commented 4 days ago

The error "Process unexpectedly closed with status signal" when generating thumbnails in Superset within a Docker Compose environment is likely due to insufficient memory allocation for the superset-worker service.

To resolve this, you can increase the memory limit for the superset-worker service by uncommenting and adjusting the mem_limit and mem_reservation settings in your docker-compose.yml file. Here is the relevant section with the memory settings uncommented:

  superset-worker:
    build:
      <<: *common-build
    container_name: superset_worker
    command: ["/app/docker/docker-bootstrap.sh", "worker"]
    env_file:
      - path: docker/.env # default
        required: true
      - path: docker/.env-local # optional override
        required: false
    environment:
      CELERYD_CONCURRENCY: 2
    restart: unless-stopped
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes
    extra_hosts:
      - "host.docker.internal:host-gateway"
    healthcheck:
      test: ["CMD-SHELL", "celery -A superset.tasks.celery_app:app inspect ping -d celery@$$HOSTNAME"]
    # Bump memory limit if processing selenium / thumbnails on superset-worker
    mem_limit: 2038m
    mem_reservation: 128M

By increasing the memory limit, you can help ensure that the superset-worker has enough resources to process thumbnails without running into memory-related issues ^[1].

To continue talking to Dosu, mention @dosu.

mschaefer-gresham commented 4 days ago

@dosu this did not solve the issue.

dosubot[bot] commented 4 days ago

If increasing the memory limit did not resolve the issue, there are a few other configurations to check:

Redis Configuration: Ensure that the Redis service is running and accessible by the superset-worker. Verify that the REDIS_HOST and REDIS_PORT environment variables are correctly set.
Celery Configuration: Ensure that Celery is properly configured and running. The health check command for Celery should return a successful response.
Selenium Configuration: Ensure that Selenium is properly configured and accessible. This includes setting the WEBDRIVER_BASEURL in the superset_config.py:
```
WEBDRIVER_BASEURL = "http://superset:8088/"  # When using docker compose baseurl should be http://superset_app:8088/
```

Headless Browser Installation: Ensure that a headless browser (Firefox or Chrome) is installed in the superset-worker container. If using Chrome, set the WEBDRIVER_TYPE to "chrome" in your superset_config.py:

WEBDRIVER_TYPE = "chrome"
WEBDRIVER_OPTION_ARGS = [
    "--force-device-scale-factor=2.0",
    "--high-dpi-support=2.0",
    "--headless",
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--disable-extensions",
]

Logs: Check the logs of the superset-worker container for any errors or warnings that might indicate what is going wrong.

Ensure that all these configurations are correctly set and that all dependencies are running as expected ^[1]^[2].

To continue talking to Dosu, mention @dosu.

mschaefer-gresham commented 3 days ago

@duso

I verified that redis is running and that I can ping it using the cli.
The logs indicate that celery has started successfully.
The WEBDRIVER_BASEURL is correct.

I am using the default firefox driver and using the following settings:

WEBDRIVER_OPTION_ARGS = [
    "--force-device-scale-factor=2.0",
    "--high-dpi-support=2.0",
    "--headless",
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--disable-extensions",
    ]

There are no issues in the logs other than the error reported in my original post.

mschaefer-gresham commented 3 days ago

After much digging around and trying chrome, I suspect the issue is that I'm running docker compose on a Mac M3. It appears the chromedriver and firefox executables won't run on M series.

docker exec -it superset_worker bash
root@413ee2048953:/app# cd /usr/local/bin
root@413ee2048953:/usr/local/bin# chromedriver
rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
 Trace/breakpoint trap
root@413ee2048953:/usr/local/bin# firefox
rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
 Trace/breakpoint trap

Can someone confirm? And is there a workaround?

mschaefer-gresham commented 3 days ago

This was caused due to running docker compose on Mac M3. chrome and firefix executables won't run on Apple Silicon Macs. When running on an Intel Mac it started working.

sfirke commented 3 days ago

Interesting, thank you @mschaefer-gresham for diagnosing the problem and reporting back!

apache / superset