[🐛 Bug]: S3 video upload constantly fails

[nandorpal]

What happened?

I created three selenium/video containers, one for firefox, one for edge, and one for chrome. After running some tests, randomly seemingly the video upload gets stuck and it only saves the videos in the docker container's video folder, instead of uploading it to an s3 bucket.

Command used to start Selenium Grid with Docker (or Kubernetes)

version: "3.8"

services:
  chrome:
    image: selenium/node-chrome:latest
    shm_size: 2gb
    container_name: chrome
    depends_on:
      - selenium-hub
    env_file:
      - ./.env
    ports:
      - "5900:5900"
    networks:
      - grid
    restart: unless-stopped

  firefox:
    image: selenium/node-firefox:latest
    container_name: firefox
    shm_size: 2gb
    depends_on:
      - selenium-hub
    env_file:
      - ./.env
    ports:
      - "5901:5900"
    networks:
      - grid
    restart: unless-stopped

  edge:
    image: selenium/node-edge:latest
    container_name: edge
    shm_size: 2gb
    depends_on:
      - selenium-hub
    env_file:
      - ./.env
    ports:
      - "5902:5900"
    networks:
      - grid
    restart: unless-stopped

  chrome-video:
    image: selenium/video:latest
    container_name: chrome-video
    shm_size: 2gb
    depends_on:
      - chrome
    environment:
      - DISPLAY_CONTAINER_NAME=chrome
    env_file:
      - ./.env
    networks:
      - grid
    restart: unless-stopped

  firefox-video:
    image: selenium/video:latest
    container_name: firefox-video
    shm_size: 2gb
    depends_on:
      - firefox
    environment:
      - DISPLAY_CONTAINER_NAME=firefox
    env_file:
      - ./.env
    networks:
      - grid
    restart: unless-stopped

  edge-video:
    image: selenium/video:latest
    container_name: edge-video
    shm_size: 2gb
    depends_on:
      - edge
    environment:
      - DISPLAY_CONTAINER_NAME=edge
    env_file:
      - ./.env
    networks:
      - grid
    restart: unless-stopped

  selenium-hub:
    image: selenium/hub:latest
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    networks:
      - grid
    restart: unless-stopped

networks:
  grid:
    driver: bridge

Relevant log output

Waiting for /tmp/uploadpipe to be created

Operating System

ubuntu

Docker Selenium version (image tag)

ffmpeg-6.1.1-20240621

Selenium Grid chart version (chart version)

No response

[github-actions[bot]]

@nandorpal, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

With removing secrets info if any, I want to see all ENV var that you provided in .env that mounted to the container

[nandorpal]

@VietND96 thanks for the quick reply! Here's my env file with secrets removed:

SE_EVENT_BUS_HOST=selenium-hub
SE_EVENT_BUS_PUBLISH_PORT=4442
SE_EVENT_BUS_SUBSCRIBE_PORT=4443
SE_VNC_NO_PASSWORD=1
SE_ENABLE_BROWSER_LEFTOVERS_CLEANUP=true

# HUB

# VIDEO
SE_NODE_GRID_URL=http://selenium-hub:4444
SE_VIDEO_FILE_NAME=auto
SE_VIDEO_UPLOAD_ENABLED=true
SE_VIDEO_INTERNAL_UPLOAD=true
SE_UPLOAD_DESTINATION_PREFIX=s3://selenium-test-videos
RCLONE_CONFIG_S3_TYPE=s3
RCLONE_CONFIG_S3_PROVIDER=AWS
RCLONE_CONFIG_S3_ENV_AUTH=true
RCLONE_CONFIG_S3_REGION=eu-central-1
RCLONE_CONFIG_S3_LOCATION_CONSTRAINT=eu-central-1
RCLONE_CONFIG_S3_ACL=private
RCLONE_CONFIG_S3_ACCESS_KEY_ID=
RCLONE_CONFIG_S3_SECRET_ACCESS_KEY=
RCLONE_CONFIG_S3_NO_CHECK_BUCKET=true

[nandorpal]

@VietND96 any update on this one? Thanks in advance

[nandorpal]

I updated the docker images to the newer ffmpeg-7.0.1-20240812 version and it seems to be running as expected

[VietND96]

Thank you for your feedback! It was probably an issue with the env var missing the default value set. Few implementations come in between fixing it. Soon, I will add a test to the CI workflow to guard against regression issue and also using for reference.

[nandorpal]

@VietND96 unfortunately the issue still persists, after a couple of uploads I got the Waiting for /tmp/uploadpipe to be created again. I'll try to figure out what's the root cause of the issue. Thanks for your help.

[nandorpal]

I think I found the issue. There's an issue with the uploadpipe in the upload.sh file, it seems like the named pipe file might not always be detected as expected. I created some tests and so far the results are promising

[VietND96]

Thank you for your contribution. The fix will be included in next deployment!