seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.46k stars 910 forks source link

Driver Exits with STATUS (selenium-grid-hub (terminated by SIGTERM)) #2726

Closed kreethandsouza closed 2 months ago

kreethandsouza commented 2 months ago

I am spawning a seleniumbase broswer through docker in this way.

chrome_driver = Driver(log_cdp=True, headless=False, no_sandbox=True, agent=user_agent, proxy=proxy, protocol=protocol, servername=server_name, port=port)

where the browser is spawned in the selenium Grid. The broswer gets spawned for few seconds and suddenly it terminates with

06:44:43.243 INFO [LocalDistributor.newSession] - Session request received by the Distributor: 2024-04-29T06:44:43.258918762Z [Capabilities {browserName: chrome, cloud:options: {browserName: chrome, goog:chromeOptions: {args: [], extensions: []}, pageLoadStrategy: normal}, goog:chromeOptions: {args: [--window-size=1280,840, --proxy-server=205.164.39.6..., --ignore-certificate-errors, --no-sandbox, --disable-gpu, --disable-dev-shm-usage, --user-agent=Mozilla/5.0 (L..., --disable-browser-side-navi..., --disable-save-password-bubble, --disable-single-click-auto..., --allow-file-access-from-files, --disable-prompt-on-repost, --dns-prefetch-disable, --disable-translate, --disable-3d-apis, --disable-backgrounding-occ..., --disable-client-side-phish..., --disable-oopr-debug-crash-..., --disable-top-sites, --ash-no-nudges, --no-crash-upload, --deny-permission-prompts, --disable-features=Optimiza..., --test-type, --log-level=3, --no-first-run, --allow-insecure-localhost, --allow-running-insecure-co..., --disable-infobars, --disable-notifications, --disable-autofill-keyboard..., --homepage=about:blank, --dom-automation, --disable-hang-monitor], excludeSwitches: [enable-automation, enable-logging, enable-blink-features], extensions: [UEsDBBQAAAAAAJU1nVh6sEvBNgI...], prefs: {content_settings.exceptions.automatic_downloads.*.setting: 1, credentials_enable_service: false, default_content_setting_values.notifications: 0, default_content_settings.popups: 0, download.default_directory: /seleniumbase/downloaded_files, download.directory_upgrade: true, download.prompt_for_download: false, local_discovery.notifications_enabled: false, managed_default_content_settings.popups: 0, omnibox-local-history-zero-suggest-beyond-ntp: 0, omnibox-max-zero-suggest-matches: 0, omnibox-on-focus-suggestions-contextual-web: 0, omnibox-on-focus-suggestions-srp: 0, omnibox-trending-zero-prefix-suggestions-on-ntp: 0, omnibox-use-existing-autocomplete-client: 0, omnibox-zero-suggest-in-memory-caching: 0, omnibox-zero-sug gest-prefetching: 0, omnibox-zero-suggest-prefetching-on-srp: 0, omnibox-zero-suggest-prefetching-on-web: 0, profile.default_content_setting_values.automatic_downloads: 1, profile.default_content_setting_values.notifications: 2, profile.default_content_settings.popups: 0, profile.managed_default_content_settings.popups: 0, profile.password_manager_enabled: false, safebrowsing.disable_download_protection: true, safebrowsing.enabled: false, webrtc.ip_handling_policy: disable_non_proxied_udp, webrtc.multiple_routes_enabled: false, webrtc.nonproxied_udp_enabled: false}}, goog:loggingPrefs: {browser: ALL, performance: ALL}, pageLoadStrategy: normal}] 2024-04-29T06:45:30.521278889Z 06:45:30.520 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: a45a0dd51eeb9948c29d43a858bcdf21 2024-04-29T06:45:30.521364130Z Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 114.0.5735.133, chrome: {chromedriverVersion: 114.0.5735.90 (386bc09e8f4f..., userDataDir: /tmp/.com.google.Chrome.h8km4j}, cloud:options: {browserName: chrome, goog:chromeOptions: {args: [], extensions: []}, pageLoadStrategy: normal}, goog:chromeOptions: {debuggerAddress: localhost:37383}, goog:loggingPrefs: {browser: ALL, performance: ALL}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://10.32.192.28:4444/sess..., se:cdpVersion: 114.0.5735.133, se:vnc: ws://10.32.192.28:4444/sess..., se:vncEnabled: true, se:vncLocalAddress: ws://10.32.192.28:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true} Trapped SIGTERM/SIGINT/x so shutting down supervisord... 2024-04-29 06:47:33,647 WARN received SIGTERM indicating exit request 2024-04-29 06:47:33,666 INFO waiting for selenium-grid-hub to die 2024-04-29 06:47:34,679 INFO stopped: selenium-grid-hub (terminated by SIGTERM) Shutdown complete

when checked in the selenium Grid fron-end it says '503' request Error.

the worked in local but throws this issue while it was deployed in kubernetes.

mdmintz commented 2 months ago

This Docker/Grid issue is not a SeleniumBase issue: (Trapped SIGTERM/SIGINT/x so shutting down supervisord):

It would also be impossible to debug from my end without knowing the exact details of your Dockerfile and your other configuration options. Maybe try without all those extra options such as proxy so that you can figure out which one is causing the bad configuration. And you'll also need to use SB() instead of Driver() because you want to run in a headless display system without using headless mode. The Driver() can't do that because it doesn't start Xvfb. SB() does.

kreethandsouza commented 2 months ago

I made changes as you said for Driver. removed the configurations but it didnt work.

services: service-selenium-hub: image: selenium/hub:4.10.0 container_name: service-selenium-hub ports:

chrome: image: selenium/node-chrome:4.10.0 depends_on:

when i spawn the browser from another container say

FROM python:3.9-alpine3.15

WORKDIR /seleniumbase

COPY / /seleniumbase

COPY requirements.txt /requirements.txt RUN apk update && apk add --no-cache gcc python3-dev musl-dev linux-headers chromium chromium-chromedriver && pip install -r requirements.txt && apk del gcc musl-dev linux-headers && rm -rf /var/cache/apk/ /root/.cache/

This would be my code to spawn it from another container

`from seleniumbase import Driver

agent = "Mozilla/5.0 (Linux; Android 13; CPH2487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Mobile Safari/537.36" proxy = 'user:password@ip:port"' driver = Driver(uc=True, log_cdp=True, headless=False, no_sandbox=True, agent=agent, proxy=proxy, protocol='http', servername="service-selenium-hub", port="4444")

url = "https://www.google.com/" driver.get(url)`

this would be my selenium grid setup. the browser spawns and exists giving this error code from the grid. "There has been an error while loading running and queued Sessions from the Grid. Response not successful: Received status code 503"

then the deployed container in kubernetes get restarted.

could you help me with what the solution would be for this? or could you tell how can i use SB() driver to do it? @mdmintz

mdmintz commented 2 months ago

As mentioned in https://github.com/seleniumbase/SeleniumBase/issues/2675#issuecomment-2047434434: UC Mode is not compatible with the Grid.

Also, running an automated web browser in a Docker container makes it detectable. Possibly someone will one day figure out how to create a Dockerfile that allows stealth browsers, but that day hasn't happened yet.

kreethandsouza commented 2 months ago

Yeah i get that it UC Mode is not compatible with the Grid. but a normal spawning of selenium browser of seleniumbase would work right? but even that isn't happening.

driver = Driver(protocol='http', servername="service-selenium-hub", port="4444")

just spawning the browser with this also exits.

Again amazing work on seleniumbase, really amazing integration with selenium.

whats the best way to spawn the browser in docker? using SB()? if yes could you give some example? thank you.

@mdmintz

mdmintz commented 2 months ago

@kreethandsouza service-selenium-hub is not a server name. You need either a full IP Address or a full web address.

For Docker, there's a ReadMe here, and see the Dockerfile that goes with it.