Closed IlyaUk closed 4 days ago
@IlyaUk, thank you for creating this issue. We will troubleshoot it as soon as we can.
Triage this issue by using labels.
If information is missing, add a helpful comment and then I-issue-template
label.
If the issue is a question, add the I-question
label.
If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted
label.
If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C),
add the applicable G-*
label, and it will provide the correct link and auto-close the
issue.
After troubleshooting the issue, please add the R-awaiting answer
label.
Thank you!
This is the recommended method for downloads: https://www.selenium.dev/documentation/grid/configuration/cli_options/#enabling-managed-downloads-by-the-node.
However, this is not a tool to download massive files. It is a use case in automation, but it does not cover all situations.
You might try to give more memory to the Hub and adjust it to your use case. Memory will be released when the GC passes, but it is also possible that the size of the file is big enough to exhaust the memory available.
Hi @diemol, Thanks for the advice with --enable-managed-downloads true option. This option is enabled in my Selenium configuration. The above described problem can not be resolved by memory increase. I gave more than 4Gb for Hub and still no effect. Also I don't try to download large files. Each of my 10 tests that use CDP downloads a file with size around 10-50 Kb. And in general the problem with CDP is not with files download only. An attempt to use CDP for request interceptions also shows problems with memory consumption on Linux machine. But I have only one such case so the negative trend with memory is not so noticeable.
We need a way to reproduce the issue.
Hi, @IlyaUk. Please follow the issue template, we need more information to reproduce the issue.
Either a complete code snippet and URL/HTML (if more than one file is needed, provide a GitHub repo and instructions to run the code), the specific versions used, or a more detailed description to help us understand the issue.
Note: If you cannot share your code and URL/HTML, any complete code snippet and URL/HTML that reproduces the issue is good enough.
Reply to this issue when all information is provided, thank you.
We need a way to reproduce the issue.
Hi @diemol, I sent an invitation to my project (JavaProject) with small test that reproduces the problem. Docker compose file was previously attached to the issue.
Hello, Same problem here (same version of Selenium, same stacktrace in Hub logs). Tests run during ~20 min on a grid of 8 Firefox nodes, errors start to occur near the end of job.
For me, disabling the VNC client help :
SE_START_VNC: "false"
The DriverServiceSessionFactory
is not closing the HttpClient
when the session fails to start, will fix this the next days.
https://github.com/SeleniumHQ/selenium/commit/97d56d04e1b4ab4f8e527f8849b777c1e91d13f7 is the promised commit @IlyaUk could you update to the next nightly and check?
Hi @joerg1985, @diemol The fix https://github.com/SeleniumHQ/selenium/commit/97d56d04e1b4ab4f8e527f8849b777c1e91d13f7 didn't help. Selenium Hub revision: 7c8727b Docker-compose file:
version: "3"
services:
tmp-hub:
image: selenium/hub:nightly
container_name: tmp-hub
ports:
- "4444:4444"
- "4443:4443"
- "4442:4442"
healthcheck:
test: "/opt/bin/check-grid.sh --host 0.0.0.0 --port 4444"
interval: "15s"
timeout: "30s"
retries: 5
restart: always
chrome:
image: selenium/node-chrome:nightly
shm_size: 2gb
depends_on:
- tmp-hub
environment:
- SE_EVENT_BUS_HOST=tmp-hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
- SE_VNC_NO_PASSWORD=1
- SE_SCREEN_WIDTH=1600
- SE_SCREEN_HEIGHT=900
- SE_NODE_MAX_SESSIONS=1
- SE_NODE_MAX_INSTANCES=1
- SE_NODE_GRID_URL=http://localhost:4444
- SE_OPTS=--enable-managed-downloads true
restart: always
deploy:
mode: replicated
replicas: 16
Selenium hub logs output:
13:12:05.172 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, se:downloadsEnabled: false, unhandledPromptBehavior: accept}]
13:12:44.808 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: c6bb0fbdfc014f5f695c0ffdb8532013
Caps: Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 127.0.6533.72, chrome: {chromedriverVersion: 127.0.6533.72 (9755e24ca85a..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:41275}, networkConnectionEnabled: false, pageLoadStrategy: eager, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://172.30.4.67:4444/sessi..., se:cdpVersion: 127.0.6533.72, se:downloadsEnabled: false, se:vnc: ws://172.30.4.67:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://172.18.0.6:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: accept, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
13:12:45.467 INFO [GridModel.release] - Releasing slot for session id 50a6c73f8fe0bf337f8334af9fe6bfed
13:12:45.467 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 50a6c73f8fe0bf337f8334af9fe6bfed
13:12:55.143 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, unhandledPromptBehavior: accept}]
13:13:52.086 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 641ec35ff3f68944efa27fef028c7c63
Caps: Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 127.0.6533.72, chrome: {chromedriverVersion: 127.0.6533.72 (9755e24ca85a..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:39097}, networkConnectionEnabled: false, pageLoadStrategy: eager, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://172.30.4.67:4444/sessi..., se:cdpVersion: 127.0.6533.72, se:vnc: ws://172.30.4.67:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://172.18.0.8:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: accept, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
13:15:19.227 INFO [GridModel.release] - Releasing slot for session id 9fe2f02f0f83e27320023f8037a61163
13:15:19.227 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 9fe2f02f0f83e27320023f8037a61163
13:15:20.133 INFO [GridModel.release] - Releasing slot for session id bde56e1d476bd0088fa9fe1aa26d374e
13:15:20.133 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: bde56e1d476bd0088fa9fe1aa26d374e
13:15:23.337 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, unhandledPromptBehavior: accept}]
13:15:24.336 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 030bb33f2b4520f77b9a09c085b8e46f
Caps: Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 127.0.6533.72, chrome: {chromedriverVersion: 127.0.6533.72 (9755e24ca85a..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:37113}, networkConnectionEnabled: false, pageLoadStrategy: eager, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://172.30.4.67:4444/sessi..., se:cdpVersion: 127.0.6533.72, se:vnc: ws://172.30.4.67:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://172.18.0.16:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: accept, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
13:15:32.698 INFO [GridModel.release] - Releasing slot for session id 030bb33f2b4520f77b9a09c085b8e46f
13:15:32.698 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 030bb33f2b4520f77b9a09c085b8e46f
13:15:32.759 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, unhandledPromptBehavior: accept}]
13:15:34.300 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: cfc8d3ee00b9149909c19243817d69d5
Caps: Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 127.0.6533.72, chrome: {chromedriverVersion: 127.0.6533.72 (9755e24ca85a..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:43473}, networkConnectionEnabled: false, pageLoadStrategy: eager, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://172.30.4.67:4444/sessi..., se:cdpVersion: 127.0.6533.72, se:vnc: ws://172.30.4.67:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://172.18.0.13:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: accept, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
13:15:59.677 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: cfc8d3ee00b9149909c19243817d69d5
13:15:59.677 INFO [GridModel.release] - Releasing slot for session id cfc8d3ee00b9149909c19243817d69d5
13:15:59.922 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, unhandledPromptBehavior: accept}]
13:16:23.700 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, se:downloadsEnabled: false, unhandledPromptBehavior: accept}]
13:16:57.771 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: a4277b3e4668ec24a5f32e77d1b073b2
Caps: Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 127.0.6533.72, chrome: {chromedriverVersion: 127.0.6533.72 (9755e24ca85a..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:36559}, networkConnectionEnabled: false, pageLoadStrategy: eager, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://172.30.4.67:4444/sessi..., se:cdpVersion: 127.0.6533.72, se:vnc: ws://172.30.4.67:4444/sessi..., se:vncEnabled: true, se:vncLocalAddress: ws://172.18.0.10:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: accept, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
13:18:15.592 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, se:downloadsEnabled: false, unhandledPromptBehavior: accept}]
/opt/bin/start-selenium-grid-hub.sh: line 124: 10 Killed java ${JAVA_OPTS:-$SE_JAVA_OPTS} -jar /opt/selenium/selenium-server.jar ${EXTRA_LIBS} hub --session-request-timeout ${SE_SESSION_REQUEST_TIMEOUT} --session-retry-interval ${SE_SESSION_RETRY_INTERVAL} --healthcheck-interval ${SE_HEALTHCHECK_INTERVAL} --relax-checks ${SE_RELAX_CHECKS} --bind-host ${SE_BIND_HOST} --config /opt/selenium/config.toml ${HOST_CONFIG} ${PORT_CONFIG} ${SUB_PATH_CONFIG} ${SE_OPTS}
2024-08-06 13:31:38,980 WARN exited: selenium-grid-hub (exit status 137; not expected)`
@IlyaUk @EvgenyMarchuk i have done some fixed related releasing the websocket in https://github.com/SeleniumHQ/selenium/commit/ed3edee0ac4f09a5555aec7c7ab20609d8b394f2, https://github.com/SeleniumHQ/selenium/commit/5bac4795ff6acc9f1ca1a6436aea0970ff98fb07 and https://github.com/SeleniumHQ/selenium/commit/7612405e34d282992b26a22d7bee921753020026.
I hope the leak is gone now, could you check this?
Hi @joerg1985. Unfortunately the above fixes didn't help. Relevant logs output is below:
19:21:11.496 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-search-engine-cho..., --unsafely-disable-devtools..., --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {autofill.credit_card_enabled: false, autofill.profile_enabled: false, credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, se:downloadsEnabled: false, unhandledPromptBehavior: accept}]
19:22:43.911 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=<-loopb..., --disable-dev-shm-usage, --disable-search-engine-cho..., --unsafely-disable-devtools..., --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {autofill.credit_card_enabled: false, autofill.profile_enabled: false, credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, se:downloadsEnabled: false, unhandledPromptBehavior: accept}]
19:23:18.028 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "04d7720543da77be538cb871ceb71dd9","eventTime": 1724786471496188585,"eventName": "Session request received by the Distributor","attributes": {"logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=\u003c-loopb..., --disable-dev-shm-usage, --disable-search-engine-cho..., --unsafely-disable-devtools..., --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {autofill.credit_card_enabled: false, autofill.profile_enabled: false, credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, se:downloadsEnabled: false, unhandledPromptBehavior: accept}]"}}
19:23:18.028 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: session not created
from chrome not reachable
Host info: host: '8c61b151e7d6', ip: '172.19.0.16'
Build info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'
Driver info: driver.version: unknown
Build info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'
Driver info: driver.version: unknown
Build info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'
Driver info: driver.version: unknown
at org.openqa.selenium.grid.node.remote.RemoteNode.newSession(RemoteNode.java:157)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.startSession(LocalDistributor.java:654)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.newSession(LocalDistributor.java:573)
at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.handleNewSessionRequest(LocalDistributor.java:836)
at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.lambda$run$1(LocalDistributor.java:793)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
19:23:18.028 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to create session: Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: session not created
from chrome not reachable
Host info: host: '8c61b151e7d6', ip: '172.19.0.16'
Build info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'
Driver info: driver.version: unknown
Build info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'
Driver info: driver.version: unknown
Build info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'
Driver info: driver.version: unknown
19:23:18.028 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "04d7720543da77be538cb871ceb71dd9","eventTime": 1724786597988818214,"eventName": "exception","attributes": {"exception.message": "Unable to create session: Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: session not created\nfrom chrome not reachable \nHost info: host: '8c61b151e7d6', ip: '172.19.0.16'\nBuild info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'\nDriver info: driver.version: unknown\nBuild info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'\nDriver info: driver.version: unknown\nBuild info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: session not created\nfrom chrome not reachable \nHost info: host: '8c61b151e7d6', ip: '172.19.0.16'\nBuild info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'\nDriver info: driver.version: unknown\nBuild info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'\nDriver info: driver.version: unknown\nBuild info: version: '4.24.0-SNAPSHOT', revision: '7c8727b'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1062-aws', java.version: '17.0.12'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.node.remote.RemoteNode.newSession(RemoteNode.java:157)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor.startSession(LocalDistributor.java:654)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor.newSession(LocalDistributor.java:573)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.handleNewSessionRequest(LocalDistributor.java:836)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.lambda$run$1(LocalDistributor.java:793)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:840)\n","exception.type": "org.openqa.selenium.SessionNotCreatedException","logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--headless=new, --proxy-bypass-list=\u003c-loopb..., --disable-dev-shm-usage, --disable-search-engine-cho..., --unsafely-disable-devtools..., --disable-background-networ..., --enable-features=NetworkSe..., --disable-background-timer-..., --disable-backgrounding-occ..., --disable-breakpad, --disable-client-side-phish..., --disable-component-extensi..., --disable-default-apps, --disable-features=TranslateUI, --disable-hang-monitor, --disable-ipc-flooding-prot..., --disable-popup-blocking, --disable-prompt-on-repost, --disable-renderer-backgrou..., --disable-sync, --force-color-profile=srgb, --metrics-recording-only, --no-first-run, --password-store=basic, --use-mock-keychain, --hide-scrollbars, --mute-audio, --window-size=1600,900], excludeSwitches: [enable-automation, load-extension], extensions: [], prefs: {autofill.credit_card_enabled: false, autofill.profile_enabled: false, credentials_enable_service: false, plugins.always_open_pdf_externally: true, profile.default_content_setting_values.automatic_downloads: 1, profile.password_manager_enabled: false, profile.password_manager_leak_detection: false, safebrowsing.enabled: true}}, pageLoadStrategy: eager, se:downloadsEnabled: false, unhandledPromptBehavior: accept}]"}}
19:23:18.244 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "04d7720543da77be538cb871ceb71dd9","eventTime": 1724786598226963867,"eventName": "HTTP request execution complete","attributes": {"http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.local.LocalNewSessionQueue","http.host": "localhost:4444","http.method": "POST","http.request_content_length": "2185","http.scheme": "HTTP","http.status_code": 500,"http.target": "\u002fsession","http.user_agent": "selenium\u002f4.23.1 (java unix)"}}
/opt/bin/start-selenium-grid-hub.sh: line 124: 10 Killed java ${JAVA_OPTS:-$SE_JAVA_OPTS} -jar /opt/selenium/selenium-server.jar ${EXTRA_LIBS} hub --session-request-timeout ${SE_SESSION_REQUEST_TIMEOUT} --session-retry-interval ${SE_SESSION_RETRY_INTERVAL} --healthcheck-interval ${SE_HEALTHCHECK_INTERVAL} --relax-checks ${SE_RELAX_CHECKS} --bind-host ${SE_BIND_HOST} --config /opt/selenium/config.toml ${HOST_CONFIG} ${PORT_CONFIG} ${SUB_PATH_CONFIG} ${SE_OPTS}
2024-08-27 19:33:50,677 WARN exited: selenium-grid-hub (exit status 137; not expected)
But those fixes have not been released, are only available in the Nightly version.
But those fixes have not been released, are only available in the Nightly version.
Sure, I tried to reproduce the issue using the nightly builds. Here is my docker compose file:
version: "3"
services:
hub:
image: selenium/hub:nightly
container_name: hub
ports:
- "4444:4444"
- "4443:4443"
- "4442:4442"
healthcheck:
test: "/opt/bin/check-grid.sh --host 0.0.0.0 --port 4444"
interval: "15s"
timeout: "30s"
retries: 5
restart: always
chrome:
image: selenium/node-chrome:nightly
shm_size: 2gb
depends_on:
- hub
environment:
- SE_EVENT_BUS_HOST=hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
- SE_VNC_NO_PASSWORD=1
- SE_SCREEN_WIDTH=1600
- SE_SCREEN_HEIGHT=900
- SE_NODE_MAX_SESSIONS=1
- SE_NODE_MAX_INSTANCES=1
- SE_NODE_GRID_URL=http://localhost:4444
- SE_OPTS=--enable-managed-downloads true
restart: always
deploy:
mode: replicated
replicas: 16
@IlyaUk from todays log the revision is 7c8727b
, this revision is the same you tested 3 weeks ago.
You might have used a cached version of the selenium/hub:nightly
image?
@IlyaUk from todays log the revision is
7c8727b
, this revision is the same you tested 3 weeks ago. You might have used a cached version of theselenium/hub:nightly
image?
May I ask you please to provide the expected revision number so I can double check the issue thorougly?
The current nightly is https://github.com/SeleniumHQ/selenium/commit/58ca1331ab8671090f2cdb05c17a9af54841523e
Or you could try 4.24.0, this release is brand new and contains the fixes.
Thanks a lot for the help @joerg1985. The original issue has been fixed and not reproducible anymore.
@IlyaUk thanks for reporting and testing the fix.
What happened?
An attempt to run tests that use CDP for files download on remote machine via Selenium Hub (hub and nodes are on the same machine) shows a problem with memory leak on Hub side. As a result test run (around 200 tests and about 10 tests that use CDP) fails with the following exception: Response code 500. Message: session not created: DevToolsActivePort file doesn't exist After that Docker stops the Hub container with OutOfMemory exception. If we dive deep into Selenium server thread dump then we see the following: About 16132 threads open with stacktrace: java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@17.0.11/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17.0.11/Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.11/Unknown Source)
The only one workaround to avoid the issue - set Selenium flag --enable-cdp false for Chrome nodes and disable all tests that use CDP features.
Command used to start Selenium Grid with Docker (or Kubernetes)
Relevant log output
Operating System
Ubuntu
Docker Selenium version (image tag)
4.22.0-20240621
Selenium Grid chart version (chart version)
-