Struggling to Get Selenium-Wire Working in Docker with Remote Webdriver

[BenA-SA]

Hi all,

Apologies for the formatting in advance, I've

I'm trying to get selenium-wire working in docker with a remote webdriver, but I'm struggling to make it work. I've included my local.yml file at the bottom of the thread.

The function I'm using to try and start the webdriver is:

def start_browser():
    chrome_options = webdriverwire.ChromeOptions()
    chrome_options.add_argument("no-sandbox")
    chrome_options.add_argument("disable-dev-shm-usage")
    chrome_options.add_argument("start-maximized")
    chrome_options.add_argument("disable-infobars")
    chrome_options.add_argument("--disable-blink-features=AutomationControlled")
    prefs = {
        "download.default_directory": "/home/seluser/Downloads/",
        "download.prompt_for_download": False,
        "download.directory_upgrade": True,
    }
    seleniumwire_options = {
        "auto_config": False,
        "port": 8000,
        "addr": "127.0.0.1",
    }
    chrome_options.add_argument("--proxy-server=django:8000")
    chrome_options.add_argument("--ignore-certificate-errors")
    chrome_options.add_experimental_option("prefs", prefs)
    return webdriverwire.Remote(
        command_executor="http://selenium:4444/wd/hub",
        desired_capabilities=chrome_options.to_capabilities(),
        seleniumwire_options=seleniumwire_options,
    )

This function works, however when I try and use the driver to get a website using https, I get a ERR_TUNNEL_CONNECTION_FAILED error message, and if I try and get a website using http, it just shows me the homepage of my django project, regardless of what URL I try and get (e.g. http://www.bbc.co.uk).

I have tried to follow all the recommended steps to the letter and I can't see what I'm missing - can anyone help please?

Thanks for any help anyone can offer! Docker local.yml file below.

services:
  django: &django
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile
    image: project_local_django
    container_name: project_local_django
    depends_on:
      - postgres
      - redis
    volumes:
      - .:/app:z
      - sel-downloads:/app/sel_downloads
    env_file:
      - ./.envs/.local/.django
      - ./.envs/.local/.postgres
    ports:
      - "8000:8000"
    command: /start

  postgres:
    build:
      context: .
      dockerfile: ./compose/production/postgres/Dockerfile
    image: project_production_postgres
    container_name: project_local_postgres
    volumes:
      - project_local_postgres_data:/var/lib/postgresql/data
      - project_local_postgres_data_backups:/backups
    env_file:
      - ./.envs/.local/.postgres

  docs:
    image: project_local_docs
    container_name: project_local_docs
    build:
      context: .
      dockerfile: ./compose/local/docs/Dockerfile
    env_file:
      - ./.envs/.local/.django
    volumes:
      - ./docs:/docs:z
      - ./config:/app/config:z
      - ./project:/app/project:z
    ports:
      - "9000:9000"
    command: /start-docs

  redis:
    image: redis:6
    container_name: project_local_redis

  celeryworker:
    <<: *django
    image: project_local_celeryworker
    container_name: project_local_celeryworker
    depends_on:
      - redis
      - postgres
    ports: []
    command: /start-celeryworker

  celerybeat:
    <<: *django
    image: project_local_celerybeat
    container_name:project_local_celerybeat
    depends_on:
      - redis
      - postgres
    ports: []
    command: /start-celerybeat

  flower:
    <<: *django
    image: project_local_flower
    container_name: project_local_flower
    ports:
      - "5555:5555"
    command: /start-flower

  selenium:
    image: selenium/standalone-chrome-debug:3.141.59-20210105
    volumes:
      - .:/app
      - sel-downloads:/home/seluser/Downloads
    ports:
      - "5910:5900"

[rgehy84]

Hi @BenAkko!

I know it's been about two months since you've posted this. I was working through this yesterday and I found some solutions that might be able to help you out.

While I didn't use the docker file, I think this may still be able to help.

I want to confirm that you are running selenium wire from you local computer and the remote driver is connecting to a selenium grid on a docker container?

I'll respond to your post with that assumption and if I'm wrong, we can go from there.

Here is how I configured mine: seleniumwire_options = { "auto_config": False, "port": 8000, "addr": "127.0.0.1", } chrome_options.add_argument("--proxy-server=host.docker.internal:8000") chrome_options.add_argument("--ignore-certificate-errors")

So the key is in the proxy-server, using the host.docker.internal to get from inside the container to outside of the container, where your selenium-wire is running. This is what it means in the documentation of accessing the "host" where selenium-wire is running.

So what I used to make sure my selenium-wire port was open (8000 in your case) is after the tests started running, (on mac) I ran this command to make sure the port (8000) was open so docker could respond to it sudo lsof -PiTCP -sTCP:LISTEN

So on the local computer 0.0.0.0:8000 is where selenium-wire is running as a proxy, and in the chrome_options, it's setting the proxy server as what we set, BUT because it's running inside of Docker, we have to set it as host.docker.internal:8000.

One last thing, I'm not sure if this is necessary but I ended up adding it yesterday. So I added it when running the container from the command line (not via a yml or docker-compose file):

--add-host host.docker.internal:host-gateway

Also, after setting all that up, I logged into the docker container via docker exec -it <container_name> and then ping then telnet/curl/ping the host machine to make sure you have connectivity between the two.

If you can add that to your yml file, I think you should see some progress.

One thing I would recommend is maybe getting it to work with the regular docker run and then setting up the yml/docker compose file.

Please let me know how it goes and if I can be of any more assistance! Good luck!