ATH does not startup with BROWSER=chrome-container (or firefox-container) anymore

[uhafner]

In https://github.com/jenkinsci/acceptance-test-harness/releases/tag/5527.vf5d62a_388c39 the containers have been adapted to match the Selenium 4 upgrade (https://github.com/jenkinsci/acceptance-test-harness/pull/1077).

This update does not work for my plugins, it breaks all of my tests:

• https://github.com/jenkinsci/warnings-ng-plugin/pull/1470: https://github.com/jenkinsci/warnings-ng-plugin/actions/runs/4577019077/jobs/8081916367?pr=1470
• https://github.com/jenkinsci/git-forensics-plugin/pull/577: https://github.com/jenkinsci/git-forensics-plugin/pull/577/checks?check_run_id=12404986224

[jtnord]

In order to fix this I need much more information that is available from GitHub.

the container logs should say why the browser could not start - but these are not available (at least as far as I can tell).

And it does not completely break as https://github.com/jenkinsci/warnings-ng-plugin/actions/runs/4967128160/jobs/8888944269?pr=1470 shows that things did work

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.476 s - in io.jenkins.plugins.analysis.warnings.SnippetGeneratorUiTest [INFO] [INFO] Results: [INFO] [WARNING] Flakes: [WARNING] io.jenkins.plugins.analysis.warnings.SnippetGeneratorUiTest.defaultConfigurationTest [ERROR] Run 1: SnippetGeneratorUiTest.defaultConfigurationTest » Provision Unable to provision, see the following errors:

yes it is flaky - but to understand why without any more information is required.

Additionally - I am not sure that the Diagnostic rule works correctly here - and could hide issues where a test fails by trying to capture diagnostics when it is not possible.

[uhafner]

I'm not sure if I can follow. You already started work on an additional PR https://github.com/jenkinsci/warnings-ng-plugin/pull/1507 where you could see those problems on your machine. Or does that PR work now in your setup and just fails on GitHub? I tried to align my PR https://github.com/jenkinsci/warnings-ng-plugin/pull/1470 with the changes of your PR but also without success. What information do you need from me? Aren't the failing PRs in the warnings and git-forensics plugins not a good start to investigate what is broken in the Selenium 4 container tests?

[uhafner]

And it does not completely break as https://github.com/jenkinsci/warnings-ng-plugin/actions/runs/4967128160/jobs/8888944269?pr=1470 shows that things did work

This is also a good point, from the log it seems to use firefox even though BROWSER=chrome-container. I have no idea why some tests work here and some not.

[jtnord]

where you could see those problems on your machine. Or does that PR work now in your setup and just fails on GitHut

I could reproduce container failing to start consistently and I got to the point where the container would start correctly. container-xxx requires bridge networking which is not available on docker-for-windows, so I could go no further on windows. I did spin up a Linux VM and ran CreateItemTest test with both the firefox-container and chrome container.

[jtnord]

[INFO: Performing cleanup tasks in order: [Close WebDriver after test, Kill and remove selenium container]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 222.257 s - in io.jenkins.plugins.analysis.warnings.SmokeTests
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:57 min
[INFO] Finished at: 2023-05-19T15:51:20+01:00
[INFO] ------------------------------------------------------------------------

jtnord@debian:~/warnings-ng-plugin/ui-tests$ git log
commit ce5fca8e6e53cfe0048981b58cac7d38dff1a6c5 (HEAD, origin/dependabot/maven/org.jenkins-ci-acceptance-test-harness-5509.v07855a_003fc1)
Author: Ulli Hafner <ullrich.hafner@gmail.com>

There are however copious amounts of warnings produced as popper2 adjunct does not exist.

master00000|2023-05-19 14:51:04.787+0000 [id=60]        WARNING o.k.s.f.adjunct.AdjunctsInPage#findNeeded: No such adjunct found: io.jenkins.plugins.popper2
master00000|org.kohsuke.stapler.framework.adjunct.NoSuchAdjunctException: Neither io.jenkins.plugins.popper2.css, .js, .html, nor .jelly were found
master00000|    at org.kohsuke.stapler.framework.adjunct.Adjunct.<init>(Adjunct.java:125)
master00000|    at org.kohsuke.stapler.framework.adjunct.AdjunctManager.get(AdjunctManager.java:148)
master00000|    at org.kohsuke.stapler.framework.adjunct.AdjunctsInPage.findNeeded(AdjunctsInPage.java:161)
master00000|    at org.kohsuke.stapler.framework.adjunct.AdjunctsInPage.generate(AdjunctsInPage.java:113)
master00000|    at org.kohsuke.stapler.jelly.AdjunctTag.doTag(AdjunctTag.java:82)
master00000|    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
master00000|    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
master00000|    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
master00000|    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
master00000|    at org.kohsuke.stapler.framework.adjunct.Adjunct.write(Adjunct.java:199)
master00000|    at org.kohsuke.stapler.framework.adjunct.AdjunctsInPage.generate(AdjunctsInPage.java:116)
master00000|    at org.kohsuke.stapler.jelly.AdjunctTag.doTag(AdjunctTag.java:82)
master00000|    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)

[jtnord]

io.jenkins.plugins.analysis.warnings.IssuesColumnUiTest also works correctly

master00000|2023-05-19 14:56:14.074+0000 [id=50]        INFO    hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
May 19, 2023 3:56:18 PM org.jenkinsci.test.acceptance.guice.Cleaner performCleanUp
INFO: Performing cleanup tasks in order: [Close WebDriver after test, Kill and remove selenium container]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.803 s - in io.jenkins.plugins.analysis.warnings.IssuesColumnUiTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:30 min
[INFO] Finished at: 2023-05-19T15:56:19+01:00
[INFO] ------------------------------------------------------------------------

I can not reproduce this locally inside a Linux VM, and am only lead to the conclusion that something is wrong in the github actions environment which I have no way to debug.

[jtnord]

I have updated to 5588.vd13b_52985008 and run the tests to double check the container and it works flawlessly - but it is useing firefox for whatever reason...? (and version 4.6.0 but still it works)...

[main] INFO org.littleshoot.proxy.impl.DefaultHttpProxyServer - Starting proxy at address: localhost/127.0.0.1:0
[main] INFO org.littleshoot.proxy.impl.DefaultHttpProxyServer - Proxy listening with TCP transport
[main] INFO org.littleshoot.proxy.impl.DefaultHttpProxyServer - Proxy started at address: /127.0.0.1:36683
May 19, 2023 4:01:11 PM org.jenkinsci.test.acceptance.FallbackConfig createContainerWebDriver
INFO: Starting selenium container 'selenium/standalone-firefox:4.6.0'. Logs in /tmp/ath-docker-browser10983914033274902478log

[jtnord]

works with chrome also.

May 19, 2023 4:13:57 PM org.jenkinsci.test.acceptance.FallbackConfig createContainerWebDriver
INFO: Starting selenium container 'selenium/standalone-chrome:4.6.0'. Logs in /tmp/ath-docker-browser11091666104457607879log
...
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 88.199 s - in io.jenkins.plugins.analysis.warnings.IssuesColumnUiTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

diff --git a/ui-tests/pom.xml b/ui-tests/pom.xml
index cd2c5a919..6fe83f104 100644
--- a/ui-tests/pom.xml
+++ b/ui-tests/pom.xml
@@ -48,7 +48,7 @@
     <dependency>
       <groupId>org.jenkins-ci</groupId>
       <artifactId>acceptance-test-harness</artifactId>
-      <version>5581.vfd8e43f46a_03</version>
+      <version>5588.vd13b_52985008</version>
     </dependency>
     <dependency>
       <groupId>com.fasterxml.jackson.core</groupId>
@@ -223,7 +223,7 @@
           <environmentVariables>
             <PLUGINS_DIR>../plugin/target/test-classes/test-dependencies</PLUGINS_DIR>
             <JENKINS_VERSION>${jenkins.version}</JENKINS_VERSION>
-            <BROWSER>firefox-container</BROWSER>
+            <BROWSER>chrome-container</BROWSER>
           </environmentVariables>
           <rerunFailingTestsCount>1</rerunFailingTestsCount>
           <includes>

[jtnord]

the github runners have 7GB of ram.

the warnings-ng project does not appear to set any maximum memory for the maven maven and appears to use pooled Jenkins' for tests in UI tests. Surefire itself is constrained to 1GB.

These Jenkins's for test at first glance do not appear to have any memory limits set either.
as far as I am aware the JVM for the remote selenium in the firefox ./ chrome container also will run unconstrained unless you pass SE_JAVA_OPTS. firefox/chrome will in the contain try and have 2GB for shared memory (unless otherwise tuned).

With all this unconstrained memory usage I would not be surprised if something is getting OOMKiller by the OS and would explain the randomness failures and failed to connect.

[uhafner]

There are however copious amounts of warnings produced as popper2 adjunct does not exist.

Ah, thanks, this is a bug indeed. Opened https://github.com/jenkinsci/bootstrap5-api-plugin/pull/213 as a fix.

[uhafner]

With all this unconstrained memory usage I would not be surprised if something is getting OOMKiller by the OS and would explain the randomness failures and failed to connect.

The interesting thing is, that the old container worked flawlessly. Now everything is flaky: in https://github.com/jenkinsci/warnings-ng-plugin/pull/1470 I have a green build for one commit (https://github.com/jenkinsci/warnings-ng-plugin/pull/1470/commits/83a315ca0b1cd18b44366b886ed09674353ef8ea) and then again some failures. Are there some additional logs available that may explain the problem? The error message actually does not help very much.

And https://github.com/jenkinsci/git-forensics-plugin/pull/602 is green as well. Really strange...

[jtnord]

Are there some additional logs available that may explain the problem

there are docker logs for the container - https://github.com/jenkinsci/acceptance-test-harness/blob/c141c9a74536b66600d0951eb3445bb49b927d15/src/main/java/org/jenkinsci/test/acceptance/FallbackConfig.java#L228C46-L246

some things to possibly try 1) disable rerunning tests 2) disable the Support bundle recorder (is support-core is installed CAPTURE_SUPPORT_BUNDLE=false 3) try running without the polled controller (using LocalController)

I'm wondering if the issue at heart is that the tests are flaky and something is trying to capture diagnostics at a point in time that is too late (causing the original error to get masked).

Can you reproduce this locally at all?

[jtnord]

The interesting thing is, that the old container worked flawlessly

the new container starts an extra service IIRC - it could just be that the memory was on the limit and it has not just tipped over. but I did manage to run the select warnings-ng tests in a Linux VM with not so much memory :-/

OT: the container-firefox is pretty much how I run locally on windows (except I am manually launching the selenium container because docker host networking is baad)

[uhafner]

Are there some additional logs available that may explain the problem

there are docker logs for the container - https://github.com/jenkinsci/acceptance-test-harness/blob/c141c9a74536b66600d0951eb3445bb49b927d15/src/main/java/org/jenkinsci/test/acceptance/FallbackConfig.java#L228C46-L246

Yes, I have seen these. They do not contain anything useful.

but I did manage to run the select warnings-ng tests in a Linux VM with not so much memory :-/

In my linux based virtual machine all tests fail with the errors from above. I'll see if I can find something useful...

[jtnord]

Yes, I have seen these. They do not contain anything useful.

can you share them anyway?

@jeromepochat saw some similar failures in a CI environment (not using the chrome-container / firefox-container) and that was caused by a failure in the VNC server in the selenium container because there was a conflict of the VNC/X display ports.

[uhafner]

Here are the logs when I run one of my tests in a Linux based virtual machine. I can reproduce these failures as soon as I switch to the new ATH. When switching back to the old ATH everything works like a charm.

I wonder if it is worth to switch locally to the old 3.1 docker images to see if the problem is in the image change or in the source code changes of the ATH?

2023-06-06 11:04:11,303 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2023-06-06 11:04:11,324 INFO RPC interface 'supervisor' initialized
2023-06-06 11:04:11,325 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-06-06 11:04:11,330 INFO supervisord started with pid 8
2023-06-06 11:04:12,345 INFO spawned: 'xvfb' with pid 10
2023-06-06 11:04:12,351 INFO spawned: 'vnc' with pid 11
2023-06-06 11:04:12,355 INFO spawned: 'novnc' with pid 13
2023-06-06 11:04:12,412 INFO spawned: 'selenium-standalone' with pid 15
Appending Selenium options: --port 64597
2023-06-06 11:04:12,435 INFO success: xvfb entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2023-06-06 11:04:12,435 INFO success: vnc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2023-06-06 11:04:12,435 INFO success: novnc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2023-06-06 11:04:12,435 INFO success: selenium-standalone entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Setting up SE_NODE_GRID_URL...
Selenium Grid Standalone configuration: 
[network]
relax-checks = true

[node]
session-timeout = "300"
override-max-sessions = false
detect-drivers = false
drain-after-session-count = 0
max-sessions = 1

[[node.driver-configuration]]
display-name = "chrome"
stereotype = '{"browserName": "chrome", "browserVersion": "113.0", "platformName": "Linux"}'
max-sessions = 1

Starting Selenium Grid Standalone...
Tracing is disabled
11:04:17.073 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
11:04:17.089 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing

The test failure is here:

[INFO] Running io.jenkins.plugins.analysis.warnings.SmokeTests
Jun 06, 2023 1:04:24 PM org.jenkinsci.test.acceptance.FallbackConfig createController
WARNING: No pooled jenkins controller listening on socket /home/devbox/jenkins.sock
Jun 06, 2023 1:04:24 PM org.jenkinsci.test.acceptance.controller.LocalController postConstruct
INFO: Running with given plugins: [config-file-provider.hpi, credentials.hpi, apache-httpcomponents-client-4-api.hpi, workflow-basic-steps.hpi, job-dsl.hpi, index, javadoc.hpi, branch-api.hpi, pull-request-monitoring.hpi, bouncycastle-api.hpi, git.hpi, forensics-api.hpi, git-forensics.hpi, durable-task.hpi, run-condition.hpi, jakarta-mail-api.hpi, script-security.hpi, gradle.hpi, bootstrap5-api.hpi, echarts-api.hpi, configuration-as-code.hpi, timestamper.hpi, checks-api.hpi, workflow-support.hpi, variant.hpi, data-tables-api.hpi, ionicons-api.hpi, plugin-util-api.hpi, workflow-step-api.hpi, analysis-model-api.hpi, workflow-cps.hpi, jackson2-api.hpi, commons-lang3-api.hpi, select2-api.hpi, prism-api.hpi, mina-sshd-api-core.hpi, warnings-ng.hpi, antisamy-markup-formatter.hpi, snakeyaml-api.hpi, instance-identity.hpi, scm-api.hpi, dashboard-view.hpi, pipeline-stage-tags-metadata.hpi, workflow-durable-task-step.hpi, trilead-api.hpi, caffeine-api.hpi, workflow-api.hpi, flexible-publish.hpi, muuri-api.hpi, jquery3-api.hpi, jdk-tool.hpi, pipeline-model-api.hpi, jakarta-activation-api.hpi, font-awesome-api.hpi, ssh-credentials.hpi, ant.hpi, ansicolor.hpi, form-element-path.hpi, jaxb.hpi, jsch.hpi, structs.hpi, workflow-scm-step.hpi, pipeline-model-definition.hpi, workflow-job.hpi, display-url-api.hpi, javax-activation-api.hpi, pipeline-groovy-lib.hpi, workflow-multibranch.hpi, maven-plugin.hpi, mailer.hpi, mina-sshd-api-common.hpi, pipeline-input-step.hpi, git-client.hpi, commons-text-api.hpi, token-macro.hpi, junit.hpi, credentials-binding.hpi, ssh-slaves.hpi, pipeline-model-extensions.hpi, plain-credentials.hpi, matrix-project.hpi, cloudbees-folder.hpi, pipeline-stage-step.hpi]
Jun 06, 2023 1:04:24 PM org.jenkinsci.test.acceptance.FallbackConfig createContainerWebDriver
INFO: Starting selenium container 'selenium/standalone-chrome:4.9.1'. Logs in /tmp/ath-docker-browser784725216297115973log
Jun 06, 2023 1:04:29 PM org.jenkinsci.test.acceptance.guice.Cleaner performCleanUp
INFO: Performing cleanup tasks in order: []
Jun 06, 2023 1:04:29 PM org.jenkinsci.test.acceptance.FallbackConfig createController
WARNING: No pooled jenkins controller listening on socket /home/devbox/jenkins.sock
Jun 06, 2023 1:04:29 PM org.jenkinsci.test.acceptance.controller.LocalController postConstruct
INFO: Running with given plugins: [config-file-provider.hpi, credentials.hpi, apache-httpcomponents-client-4-api.hpi, workflow-basic-steps.hpi, job-dsl.hpi, index, javadoc.hpi, branch-api.hpi, pull-request-monitoring.hpi, bouncycastle-api.hpi, git.hpi, forensics-api.hpi, git-forensics.hpi, durable-task.hpi, run-condition.hpi, jakarta-mail-api.hpi, script-security.hpi, gradle.hpi, bootstrap5-api.hpi, echarts-api.hpi, configuration-as-code.hpi, timestamper.hpi, checks-api.hpi, workflow-support.hpi, variant.hpi, data-tables-api.hpi, ionicons-api.hpi, plugin-util-api.hpi, workflow-step-api.hpi, analysis-model-api.hpi, workflow-cps.hpi, jackson2-api.hpi, commons-lang3-api.hpi, select2-api.hpi, prism-api.hpi, mina-sshd-api-core.hpi, warnings-ng.hpi, antisamy-markup-formatter.hpi, snakeyaml-api.hpi, instance-identity.hpi, scm-api.hpi, dashboard-view.hpi, pipeline-stage-tags-metadata.hpi, workflow-durable-task-step.hpi, trilead-api.hpi, caffeine-api.hpi, workflow-api.hpi, flexible-publish.hpi, muuri-api.hpi, jquery3-api.hpi, jdk-tool.hpi, pipeline-model-api.hpi, jakarta-activation-api.hpi, font-awesome-api.hpi, ssh-credentials.hpi, ant.hpi, ansicolor.hpi, form-element-path.hpi, jaxb.hpi, jsch.hpi, structs.hpi, workflow-scm-step.hpi, pipeline-model-definition.hpi, workflow-job.hpi, display-url-api.hpi, javax-activation-api.hpi, pipeline-groovy-lib.hpi, workflow-multibranch.hpi, maven-plugin.hpi, mailer.hpi, mina-sshd-api-common.hpi, pipeline-input-step.hpi, git-client.hpi, commons-text-api.hpi, token-macro.hpi, junit.hpi, credentials-binding.hpi, ssh-slaves.hpi, pipeline-model-extensions.hpi, plain-credentials.hpi, matrix-project.hpi, cloudbees-folder.hpi, pipeline-stage-step.hpi]
Jun 06, 2023 1:04:29 PM org.jenkinsci.test.acceptance.FallbackConfig createContainerWebDriver
INFO: Starting selenium container 'selenium/standalone-chrome:4.9.1'. Logs in /tmp/ath-docker-browser7049901900603807611log
Jun 06, 2023 1:04:34 PM org.jenkinsci.test.acceptance.guice.Cleaner performCleanUp
INFO: Performing cleanup tasks in order: []
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 10.155 s <<< FAILURE! - in io.jenkins.plugins.analysis.warnings.SmokeTests
[ERROR] io.jenkins.plugins.analysis.warnings.SmokeTests.shouldRecordIssuesInPipelineAndExpandTokens  Time elapsed: 5.074 s  <<< ERROR!
com.google.inject.ProvisionException: 
Unable to provision, see the following errors:

1) [Guice/ErrorInCustomProvider]: SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
  at FallbackConfig.createWebDriver(FallbackConfig.java:335)
      \_ installed by: World -> Modules$OverrideModule -> Modules$OverrideModule -> FallbackConfig
  at DiagnosticRule.driver(DiagnosticRule.java:18)
      \_ for field driver
  while locating DiagnosticRule

Learn more:
  https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER

1 error

======================
Full classname legend:
======================
DiagnosticRule:             "org.jenkinsci.test.acceptance.junit.DiagnosticRule"
FallbackConfig:             "org.jenkinsci.test.acceptance.FallbackConfig"
Modules$OverrideModule:     "com.google.inject.util.Modules$OverrideModule"
RemoteWebDriver:            "org.openqa.selenium.remote.RemoteWebDriver"
SessionNotCreatedException: "org.openqa.selenium.SessionNotCreatedException"
World:                      "org.jenkinsci.test.acceptance.guice.World"
========================
End of classname legend:
========================

    at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1151)
    at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1186)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.addRule(JenkinsAcceptanceTestRule.java:141)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.collectGlobalRules(JenkinsAcceptanceTestRule.java:117)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.decorateWithRules(JenkinsAcceptanceTestRule.java:86)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.evaluate(JenkinsAcceptanceTestRule.java:55)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
    at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
    at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
    at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:55)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:195)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:135)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
Caused by: org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:561)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:229)
    at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:157)
    at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:139)
    at org.jenkinsci.test.acceptance.FallbackConfig.createContainerWebDriver(FallbackConfig.java:263)
    at org.jenkinsci.test.acceptance.FallbackConfig.createWebDriver(FallbackConfig.java:130)
    at org.jenkinsci.test.acceptance.FallbackConfig.createWebDriver(FallbackConfig.java:335)
    at org.jenkinsci.test.acceptance.FallbackConfig$$FastClassByGuice$$678e1.GUICE$TRAMPOLINE(<generated>)
    at org.jenkinsci.test.acceptance.FallbackConfig$$FastClassByGuice$$678e1.apply(<generated>)
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:260)
    at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:171)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at org.jenkinsci.test.acceptance.guice.TestLifecycle.lambda$scope$0(TestLifecycle.java:59)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
    at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:50)
    at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:146)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148)
    ... 37 more
Caused by: java.io.UncheckedIOException: java.net.ConnectException: Connection refused: /127.0.0.1:56707
    at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:73)
    at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:49)
    at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:99)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:112)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:94)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:68)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:165)
    at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:543)
    ... 58 more
Caused by: java.net.ConnectException: Connection refused: /127.0.0.1:56707
    at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179)
    at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
    at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:56707
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.Net.pollConnect(Native Method)
    at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)

[ERROR] io.jenkins.plugins.analysis.warnings.SmokeTests.shouldShowBuildSummaryAndLinkToDetails  Time elapsed: 5.052 s  <<< ERROR!
com.google.inject.ProvisionException: 
Unable to provision, see the following errors:

1) [Guice/ErrorInCustomProvider]: SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
  at FallbackConfig.createWebDriver(FallbackConfig.java:335)
      \_ installed by: World -> Modules$OverrideModule -> Modules$OverrideModule -> FallbackConfig
  at DiagnosticRule.driver(DiagnosticRule.java:18)
      \_ for field driver
  while locating DiagnosticRule

Learn more:
  https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER

1 error

======================
Full classname legend:
======================
DiagnosticRule:             "org.jenkinsci.test.acceptance.junit.DiagnosticRule"
FallbackConfig:             "org.jenkinsci.test.acceptance.FallbackConfig"
Modules$OverrideModule:     "com.google.inject.util.Modules$OverrideModule"
RemoteWebDriver:            "org.openqa.selenium.remote.RemoteWebDriver"
SessionNotCreatedException: "org.openqa.selenium.SessionNotCreatedException"
World:                      "org.jenkinsci.test.acceptance.guice.World"
========================
End of classname legend:
========================

    at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1151)
    at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1186)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.addRule(JenkinsAcceptanceTestRule.java:141)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.collectGlobalRules(JenkinsAcceptanceTestRule.java:117)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.decorateWithRules(JenkinsAcceptanceTestRule.java:86)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.evaluate(JenkinsAcceptanceTestRule.java:55)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
    at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
    at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
    at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:55)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:195)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:135)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
Caused by: org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:561)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:229)
    at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:157)
    at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:139)
    at org.jenkinsci.test.acceptance.FallbackConfig.createContainerWebDriver(FallbackConfig.java:263)
    at org.jenkinsci.test.acceptance.FallbackConfig.createWebDriver(FallbackConfig.java:130)
    at org.jenkinsci.test.acceptance.FallbackConfig.createWebDriver(FallbackConfig.java:335)
    at org.jenkinsci.test.acceptance.FallbackConfig$$FastClassByGuice$$678e1.GUICE$TRAMPOLINE(<generated>)
    at org.jenkinsci.test.acceptance.FallbackConfig$$FastClassByGuice$$678e1.apply(<generated>)
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:260)
    at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:171)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
    at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at org.jenkinsci.test.acceptance.guice.TestLifecycle.lambda$scope$0(TestLifecycle.java:59)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
    at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:50)
    at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:146)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148)
    ... 37 more
Caused by: java.io.UncheckedIOException: java.net.ConnectException: Connection refused: /127.0.0.1:57958
    at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:73)
    at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:49)
    at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:99)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:112)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:94)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:68)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:165)
    at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:543)
    ... 58 more
Caused by: java.net.ConnectException: Connection refused: /127.0.0.1:57958
    at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179)
    at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
    at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:57958
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.Net.pollConnect(Native Method)
    at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] io.jenkins.plugins.analysis.warnings.SmokeTests.shouldRecordIssuesInPipelineAndExpandTokens
[ERROR]   Run 1: SmokeTests.shouldRecordIssuesInPipelineAndExpandTokens » Provision Unable to provision, see the following errors:

1) [Guice/ErrorInCustomProvider]: SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
  at FallbackConfig.createWebDriver(FallbackConfig.java:335)
      \_ installed by: World -> Modules$OverrideModule -> Modules$OverrideModule -> FallbackConfig
  at DiagnosticRule.driver(DiagnosticRule.java:18)
      \_ for field driver
  while locating DiagnosticRule

Learn more:
  https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER

1 error

======================
Full classname legend:
======================
DiagnosticRule:             "org.jenkinsci.test.acceptance.junit.DiagnosticRule"
FallbackConfig:             "org.jenkinsci.test.acceptance.FallbackConfig"
Modules$OverrideModule:     "com.google.inject.util.Modules$OverrideModule"
RemoteWebDriver:            "org.openqa.selenium.remote.RemoteWebDriver"
SessionNotCreatedException: "org.openqa.selenium.SessionNotCreatedException"
World:                      "org.jenkinsci.test.acceptance.guice.World"
========================
End of classname legend:
========================

[ERROR]   Run 2: SmokeTests.shouldRecordIssuesInPipelineAndExpandTokens » Provision Unable to provision, see the following errors:

1) [Guice/ErrorInCustomProvider]: SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
  at FallbackConfig.createWebDriver(FallbackConfig.java:335)
      \_ installed by: World -> Modules$OverrideModule -> Modules$OverrideModule -> FallbackConfig
  at DiagnosticRule.driver(DiagnosticRule.java:18)
      \_ for field driver
  while locating DiagnosticRule

Learn more:
  https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER

1 error

======================
Full classname legend:
======================
DiagnosticRule:             "org.jenkinsci.test.acceptance.junit.DiagnosticRule"
FallbackConfig:             "org.jenkinsci.test.acceptance.FallbackConfig"
Modules$OverrideModule:     "com.google.inject.util.Modules$OverrideModule"
RemoteWebDriver:            "org.openqa.selenium.remote.RemoteWebDriver"
SessionNotCreatedException: "org.openqa.selenium.SessionNotCreatedException"
World:                      "org.jenkinsci.test.acceptance.guice.World"
========================
End of classname legend:
========================

[INFO] 
[ERROR] io.jenkins.plugins.analysis.warnings.SmokeTests.shouldShowBuildSummaryAndLinkToDetails
[ERROR]   Run 1: SmokeTests.shouldShowBuildSummaryAndLinkToDetails » Provision Unable to provision, see the following errors:

1) [Guice/ErrorInCustomProvider]: SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
  at FallbackConfig.createWebDriver(FallbackConfig.java:335)
      \_ installed by: World -> Modules$OverrideModule -> Modules$OverrideModule -> FallbackConfig
  at DiagnosticRule.driver(DiagnosticRule.java:18)
      \_ for field driver
  while locating DiagnosticRule

Learn more:
  https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER

1 error

======================
Full classname legend:
======================
DiagnosticRule:             "org.jenkinsci.test.acceptance.junit.DiagnosticRule"
FallbackConfig:             "org.jenkinsci.test.acceptance.FallbackConfig"
Modules$OverrideModule:     "com.google.inject.util.Modules$OverrideModule"
RemoteWebDriver:            "org.openqa.selenium.remote.RemoteWebDriver"
SessionNotCreatedException: "org.openqa.selenium.SessionNotCreatedException"
World:                      "org.jenkinsci.test.acceptance.guice.World"
========================
End of classname legend:
========================

[ERROR]   Run 2: SmokeTests.shouldShowBuildSummaryAndLinkToDetails » Provision Unable to provision, see the following errors:

1) [Guice/ErrorInCustomProvider]: SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. 
Host info: host: 'devbox', ip: '127.0.1.1'
Build info: version: '4.9.1', revision: 'eb2032df7f'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-47-generic', java.version: '17.0.4'
Driver info: RemoteWebDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}]}]
Capabilities {browserName: chrome, goog:chromeOptions: {args: [--remote-allow-origins=*], extensions: []}}
  at FallbackConfig.createWebDriver(FallbackConfig.java:335)
      \_ installed by: World -> Modules$OverrideModule -> Modules$OverrideModule -> FallbackConfig
  at DiagnosticRule.driver(DiagnosticRule.java:18)
      \_ for field driver
  while locating DiagnosticRule

Learn more:
  https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER

1 error

======================
Full classname legend:
======================
DiagnosticRule:             "org.jenkinsci.test.acceptance.junit.DiagnosticRule"
FallbackConfig:             "org.jenkinsci.test.acceptance.FallbackConfig"
Modules$OverrideModule:     "com.google.inject.util.Modules$OverrideModule"
RemoteWebDriver:            "org.openqa.selenium.remote.RemoteWebDriver"
SessionNotCreatedException: "org.openqa.selenium.SessionNotCreatedException"
World:                      "org.jenkinsci.test.acceptance.guice.World"
========================
End of classname legend:
========================

[INFO] 
[INFO] 
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:33 min
[INFO] Finished at: 2023-06-06T13:04:35+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M7:test (default-test) on project warnings-ng-ui-tests: 
[ERROR] 
[ERROR] Please refer to /home/devbox/Documents/warnings-ng-plugin-devenv/warnings-ng-plugin/ui-tests/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
devbox@devbox:~/Documents/warnings-ng-plugin-devenv/warnings-ng-plugin/ui-tests$ 

[jtnord]

I wonder if it is worth to switch locally to the old 3.1 docker images to see if the problem is in the image change or in the source code changes of the ATH?

can you run the tests differently? ie using BROWSER=remote-webdriver-chrome and firing up the container locally docker run --rm --shm-size=256m -d -p 127.0.0.1:4444:4444 -p 127.0.0.1:5900:5900 -e no_proxy=localhost -e SCREEN_WIDTH=1680 -e SCREEN_HEIGHT=1090 selenium/standalone-firefox:4.9.1 and REMOTE_WEBDRIVER_URL=http://127.0.0.1:4444/wd/hub to be passed to the ATH?

the other thing that jumpes out is the devbox Ip 127.0.1.1 which whilst it is still a loopback - if something is trying to use localhost or 127.0.0.1 then that could get funky.

[jtnord]

do you have a funky firewall or anything like that, other than that a broken docker version or some other Linux kernel bug?

The following is not setting anything special - relying on the project setup for everything

jtnord@debian:~/warnings-ng-plugin$ git fetch --all && git status
Fetching origin
On branch dependabot/maven/org.jenkins-ci-acceptance-test-harness-5509.v07855a_003fc1
Your branch is up to date with 'origin/dependabot/maven/org.jenkins-ci-acceptance-test-harness-5509.v07855a_003fc1'.

nothing to commit, working tree clean
jtnord@debian:~/warnings-ng-plugin$ git revparse HEAD
jtnord@debian:~/warnings-ng-plugin$ git rev-parse HEAD
df3b113f0ba12d7b8d2c32419eafa59297c92c2c

(the tip of https://github.com/jenkinsci/warnings-ng-plugin/pull/1470)

jtnord@debian:~/warnings-ng-plugin$ mvn -ntp clean install -DskipTests
... lots of stuff
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Warnings Next Generation Plugin 10.3.0-SNAPSHOT .... SUCCESS [01:56 min]
[INFO] UI Tests of Warnings Plugin UNVERSIONED ............ SUCCESS [ 29.224 s]
[INFO] Aggregator for Warnings Plugin UNVERSIONED ......... SUCCESS [  0.025 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

followed by

jtnord@debian:~/warnings-ng-plugin$ mvn test -Dtest=SmokeTests
....
INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.jenkins.plugins.analysis.warnings.SmokeTests
Jun 13, 2023 10:14:53 PM org.jenkinsci.test.acceptance.FallbackConfig createController
WARNING: No pooled jenkins controller listening on socket /home/jtnord/jenkins.sock
Jun 13, 2023 10:14:53 PM org.jenkinsci.test.acceptance.controller.LocalController postConstruct
.....
master00000|    at java.base/java.lang.Thread.run(Thread.java:833)
Jun 13, 2023 10:18:24 PM org.jenkinsci.test.acceptance.guice.Cleaner performCleanUp
INFO: Performing cleanup tasks in order: [Close WebDriver after test, Kill and remove selenium container]
Jun 13, 2023 10:18:24 PM org.jenkinsci.test.acceptance.update_center.MockUpdateCenter close
INFO: stopping MockUpdateCenter on http://0.0.0.0:43247/update-center.json
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 211.737 s - in io.jenkins.plugins.analysis.warnings.SmokeTests
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] ---------------< io.jenkins.plugins:warnings-ng-parent >----------------
[INFO] Building Aggregator for Warnings Plugin UNVERSIONED                [3/3]
[INFO]   from pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Warnings Next Generation Plugin 10.3.0-SNAPSHOT .... SUCCESS [01:27 min]
[INFO] UI Tests of Warnings Plugin UNVERSIONED ............ SUCCESS [03:38 min]
[INFO] Aggregator for Warnings Plugin UNVERSIONED ......... SUCCESS [  0.128 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  05:07 min
[INFO] Finished at: 2023-06-13T22:18:25+01:00
[INFO] ------------------------------------------------------------------------

jtnord@debian:~/warnings-ng-plugin$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

jtnord@debian:~/warnings-ng-plugin$ mvn --version
Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8)
Maven home: /home/jtnord/.sdkman/candidates/maven/current
Java version: 17.0.7, vendor: Eclipse Adoptium, runtime: /home/jtnord/.sdkman/candidates/java/17.0.7-tem
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux", version: "5.10.0-21-amd64", arch: "amd64", family: "unix"

jtnord@debian:~/warnings-ng-plugin$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 29
 Server Version: 20.10.5+dfsg1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1.4.13~ds1-1~deb11u3
 runc version: 1.0.0~rc93+ds1-5+deb11u2
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.0-21-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.838GiB
 Name: debian
 ID: 7MZR:6XKK:XMTK:FZU7:VMAI:K72Z:NO5H:2JAA:GXSU:C6YS:FZAB:JZAE
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Support for cgroup v2 is experimental

[jtnord]

@uhafner can you run a netstat -an and find out what address and port selenium is binding to for you?
You may need to enable INTERACTIVE mode ) so that the containers is not immediatly killed and removed. May be that it is using IPv6 and we are trying to use ipv4.

I observed for the first time today an almost similar stack trace - where I start selenium myself and had set the remote webdriver url to be http://0.0.0.0:4444/ which normally works - but this one time for whatever reason it did not.

[jtnord]

no idea if this is related or not. when using a REMOTE_WEBDRIVER_URL and starting standalone-firefox if I have a failing test and use INTERACTIVE=true after terminating maven and closing the firefox window when I run the next test it either takes a long time to start firefox or it times out with the same / similar error.

stack trace whilst it is "hanging" of the selenium server shows the following thread which appears to be suspect: (selenium 4.6.0)

"pool-3-thread-7" #170 prio=5 os_prio=0 cpu=1.55ms elapsed=163.65s tid=0x00007f12d800d000 nid=0x95d waiting on condition  [0x00007f13a0a64000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
        - parking to wait for  <0x000000071647baa8> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:1079)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:1369)
        at java.util.concurrent.CountDownLatch.await(java.base@11.0.16/CountDownLatch.java:278)
        at org.openqa.selenium.grid.sessionqueue.local.LocalNewSessionQueue.addToQueue(LocalNewSessionQueue.java:199)
        at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:69)
        at org.openqa.selenium.grid.sessionqueue.NewSessionQueue$$Lambda$830/0x0000000840329840.execute(Unknown Source)
        at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
        at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
        at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:123)
        at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
        at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
        at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
        at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
        at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
        at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders$$Lambda$816/0x0000000840315c40.execute(Unknown Source)
        at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
        at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
        at org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
        at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
        at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
        at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
        at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
        at org.openqa.selenium.remote.AddWebDriverSpecHeaders$$Lambda$794/0x0000000840269840.execute(Unknown Source)
        at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
        at org.openqa.selenium.remote.ErrorFilter$$Lambda$793/0x000000084026a440.execute(Unknown Source)
        at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
        at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
        at org.openqa.selenium.remote.ErrorFilter$$Lambda$793/0x000000084026a440.execute(Unknown Source)
        at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
        at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
        at org.openqa.selenium.netty.server.SeleniumHandler$$Lambda$792/0x000000084026a040.run(Unknown Source)

full stack trace :

``` "Reference Handler" #2 daemon prio=10 os_prio=0 cpu=7.50ms elapsed=2774.22s tid=0x00007f13e8249800 nid=0x5c waiting on condition [0x00007f13c04cd000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.16/Native Method) at java.lang.ref.Reference.processPendingReferences(java.base@11.0.16/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.16/Reference.java:213) "Finalizer" #3 daemon prio=8 os_prio=0 cpu=6.68ms elapsed=2774.22s tid=0x00007f13e824b800 nid=0x5d in Object.wait() [0x00007f13c03cc000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.16/Native Method) - waiting on at java.lang.ref.ReferenceQueue.remove(java.base@11.0.16/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x000000070f202d58> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.16/ReferenceQueue.java:176) at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.16/Finalizer.java:170) "Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.59ms elapsed=2774.21s tid=0x00007f13e8251000 nid=0x5e waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Service Thread" #5 daemon prio=9 os_prio=0 cpu=0.09ms elapsed=2774.21s tid=0x00007f13e8253000 nid=0x5f runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=18390.21ms elapsed=2774.21s tid=0x00007f13e8255000 nid=0x60 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task "C1 CompilerThread0" #14 daemon prio=9 os_prio=0 cpu=1550.13ms elapsed=2774.21s tid=0x00007f13e825f000 nid=0x61 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task "Sweeper thread" #18 daemon prio=9 os_prio=0 cpu=11.73ms elapsed=2774.21s tid=0x00007f13e8261000 nid=0x62 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Common-Cleaner" #19 daemon prio=8 os_prio=0 cpu=4.84ms elapsed=2774.19s tid=0x00007f13e8294000 nid=0x65 in Object.wait() [0x00007f13a3588000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.16/Native Method) - waiting on at java.lang.ref.ReferenceQueue.remove(java.base@11.0.16/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x000000070f21eb38> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run(java.base@11.0.16/CleanerImpl.java:148) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) at jdk.internal.misc.InnocuousThread.run(java.base@11.0.16/InnocuousThread.java:161) "Local New Session Queue" #27 daemon prio=5 os_prio=0 cpu=32.50ms elapsed=2773.38s tid=0x00007f13e86c4000 nid=0x88 waiting on condition [0x00007f13a2674000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f200cd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Purge Dead Nodes" #28 daemon prio=5 os_prio=0 cpu=20.42ms elapsed=2773.36s tid=0x00007f13e86d5800 nid=0x89 waiting on condition [0x00007f13a2373000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f202898> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Node Health Check" #29 daemon prio=5 os_prio=0 cpu=7.90ms elapsed=2773.36s tid=0x00007f13e86d8800 nid=0x8a waiting on condition [0x00007f13a2272000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f203810> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - New Session Queue" #30 daemon prio=5 os_prio=0 cpu=18655.90ms elapsed=2773.36s tid=0x00007f13e86da000 nid=0x8b waiting on condition [0x00007f13a2171000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f202ad0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "HandleSession - Clean up http clients cache" #31 daemon prio=5 os_prio=0 cpu=4.83ms elapsed=2773.35s tid=0x00007f13e86df800 nid=0x8c waiting on condition [0x00007f13a2070000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f202f08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Node - Session Cleanup http://172.17.0.2:4444" #32 daemon prio=5 os_prio=0 cpu=26.73ms elapsed=2772.95s tid=0x00007f13e8939800 nid=0x8d waiting on condition [0x00007f13a3285000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f202d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "TempFile Cleanup Node http://172.17.0.2:4444" #33 daemon prio=5 os_prio=0 cpu=6.08ms elapsed=2772.95s tid=0x00007f13e893f000 nid=0x8e waiting on condition [0x00007f13a1b6f000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f203140> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "HeartBeat Node http://172.17.0.2:4444" #34 daemon prio=5 os_prio=0 cpu=137.03ms elapsed=2772.95s tid=0x00007f13e8941000 nid=0x8f waiting on condition [0x00007f13a1a6e000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f203b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.16/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "nioEventLoopGroup-2-1" #36 prio=10 os_prio=0 cpu=18.72ms elapsed=2772.79s tid=0x00007f13e8a63000 nid=0x92 runnable [0x00007f13a156d000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f2034e8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f203490> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "DestroyJavaVM" #37 prio=5 os_prio=0 cpu=1136.88ms elapsed=2772.79s tid=0x00007f13e8016000 nid=0x33 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "nioEventLoopGroup-3-1" #38 prio=10 os_prio=0 cpu=942.03ms elapsed=2745.96s tid=0x00007f132405e800 nid=0x97 runnable [0x00007f13a378b000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f549a38> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f549a50> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "pool-1-thread-1" #40 prio=5 os_prio=0 cpu=1035.82ms elapsed=2745.89s tid=0x00007f1384154800 nid=0x9b waiting on condition [0x00007f13a2c7f000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f802bb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #41 daemon prio=5 os_prio=0 cpu=223.32ms elapsed=2745.82s tid=0x00007f1334015000 nid=0x9f waiting on condition [0x00007f13a0b66000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "pool-2-thread-1" #56 prio=5 os_prio=0 cpu=7.42ms elapsed=2712.65s tid=0x00007f13840b4000 nid=0x1b4 waiting on condition [0x00007f13a025f000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f801188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "nioEventLoopGroup-3-2" #59 prio=10 os_prio=0 cpu=501.65ms elapsed=2365.21s tid=0x00007f132404a000 nid=0x1d9 runnable [0x00007f13a0c67000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f5449b8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f5449d0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #61 daemon prio=5 os_prio=0 cpu=14.65ms elapsed=2365.17s tid=0x00007f1334018800 nid=0x1db waiting on condition [0x00007f12fb8fb000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "nioEventLoopGroup-3-3" #75 prio=10 os_prio=0 cpu=499.85ms elapsed=1618.29s tid=0x00007f132404b000 nid=0x2f2 runnable [0x00007f13a3386000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f540148> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f540160> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #77 daemon prio=5 os_prio=0 cpu=10.80ms elapsed=1618.26s tid=0x00007f1334019800 nid=0x2f4 waiting on condition [0x00007f13a3083000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "nioEventLoopGroup-3-4" #95 prio=10 os_prio=0 cpu=444.65ms elapsed=1511.64s tid=0x00007f132404c000 nid=0x403 runnable [0x00007f12fb6f9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f53b0c8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f53b0e0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #96 daemon prio=5 os_prio=0 cpu=9.58ms elapsed=1511.61s tid=0x00007f133401a800 nid=0x404 waiting on condition [0x00007f13a2f82000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "nioEventLoopGroup-3-5" #105 prio=10 os_prio=0 cpu=3.01ms elapsed=1399.95s tid=0x00007f132405f800 nid=0x51f runnable [0x00007f13a0661000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f536048> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f536060> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #108 daemon prio=5 os_prio=0 cpu=8.53ms elapsed=1152.88s tid=0x00007f133401b000 nid=0x522 waiting on condition [0x00007f13a0964000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "nioEventLoopGroup-3-6" #118 prio=10 os_prio=0 cpu=3.00ms elapsed=1029.89s tid=0x00007f1324060800 nid=0x626 runnable [0x00007f12fb5f8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f530fc8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f530fe0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #122 daemon prio=5 os_prio=0 cpu=12.84ms elapsed=822.87s tid=0x00007f1334020000 nid=0x637 waiting on condition [0x00007f13a0762000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "nioEventLoopGroup-3-7" #133 prio=10 os_prio=0 cpu=470.16ms elapsed=577.38s tid=0x00007f1324061800 nid=0x744 runnable [0x00007f13a0360000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f52bf48> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f52bf60> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #137 daemon prio=5 os_prio=0 cpu=7.74ms elapsed=492.62s tid=0x00007f1334021000 nid=0x752 waiting on condition [0x00007f13a2e81000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "ForkJoinPool.commonPool-worker-21" #144 daemon prio=5 os_prio=0 cpu=30.62ms elapsed=492.61s tid=0x00007f12e000c000 nid=0x75c waiting on condition [0x00007f12f9ce3000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f2008e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.16/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.16/ForkJoinWorkerThread.java:183) "ForkJoinPool.commonPool-worker-25" #146 daemon prio=5 os_prio=0 cpu=21.28ms elapsed=492.61s tid=0x00007f12e000d000 nid=0x75e waiting on condition [0x00007f12f8ede000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f2008e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.16/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.16/ForkJoinWorkerThread.java:183) "ForkJoinPool.commonPool-worker-11" #149 daemon prio=5 os_prio=0 cpu=32.54ms elapsed=491.11s tid=0x00007f1328072800 nid=0x828 waiting on condition [0x00007f12fbcfd000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f2008e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.16/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.16/ForkJoinWorkerThread.java:183) "ForkJoinPool.commonPool-worker-15" #152 daemon prio=5 os_prio=0 cpu=29.11ms elapsed=491.02s tid=0x00007f12b0003800 nid=0x82b waiting on condition [0x00007f12f8ada000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f2008e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.16/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.16/ForkJoinWorkerThread.java:183) "ForkJoinPool.commonPool-worker-5" #154 daemon prio=5 os_prio=0 cpu=19.72ms elapsed=441.42s tid=0x00007f1328075800 nid=0x853 waiting on condition [0x00007f12f9de4000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f2008e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.parkUntil(java.base@11.0.16/LockSupport.java:275) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.16/ForkJoinPool.java:1619) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.16/ForkJoinWorkerThread.java:183) "nioEventLoopGroup-3-8" #155 prio=10 os_prio=0 cpu=311.56ms elapsed=309.49s tid=0x00007f1324062800 nid=0x858 runnable [0x00007f12f89d9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f526ec8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f526ee0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Local Distributor - Session Creation" #157 daemon prio=5 os_prio=0 cpu=9.57ms elapsed=309.45s tid=0x00007f1334022800 nid=0x85a waiting on condition [0x00007f12f8ddd000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f21bc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Exec Default Executor" #158 daemon prio=5 os_prio=0 cpu=1.09ms elapsed=309.45s tid=0x00007f130c005800 nid=0x85c in Object.wait() [0x00007f12f88d8000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.16/Native Method) - waiting on at java.lang.Object.wait(java.base@11.0.16/Object.java:328) at java.lang.ProcessImpl.waitFor(java.base@11.0.16/ProcessImpl.java:495) - waiting to re-lock in wait() <0x00000007162011b0> (a java.lang.ProcessImpl) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364) at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Driver Service Executor" #159 daemon prio=5 os_prio=0 cpu=1.57ms elapsed=309.45s tid=0x00007f130c006800 nid=0x85d waiting on condition [0x00007f13a2d80000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x00000007162021f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Driver Service Executor" #160 daemon prio=5 os_prio=0 cpu=22.48ms elapsed=309.45s tid=0x00007f130c007800 nid=0x85e waiting on condition [0x00007f13a3184000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x00000007162021f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.16/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.16/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.16/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "process reaper" #161 daemon prio=10 os_prio=0 cpu=0.11ms elapsed=309.45s tid=0x00007f129c002000 nid=0x860 runnable [0x00007f13ec02d000] java.lang.Thread.State: RUNNABLE at java.lang.ProcessHandleImpl.waitForProcessExit0(java.base@11.0.16/Native Method) at java.lang.ProcessHandleImpl$1.run(java.base@11.0.16/ProcessHandleImpl.java:138) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Exec Stream Pumper" #162 daemon prio=5 os_prio=0 cpu=1.69ms elapsed=309.45s tid=0x00007f129c003000 nid=0x861 runnable [0x00007f12faceb000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(java.base@11.0.16/Native Method) at java.io.FileInputStream.read(java.base@11.0.16/FileInputStream.java:279) at java.io.BufferedInputStream.fill(java.base@11.0.16/BufferedInputStream.java:252) at java.io.BufferedInputStream.read1(java.base@11.0.16/BufferedInputStream.java:292) at java.io.BufferedInputStream.read(java.base@11.0.16/BufferedInputStream.java:351) - locked <0x0000000716201bd0> (a java.lang.ProcessImpl$ProcessPipeInputStream) at java.io.FilterInputStream.read(java.base@11.0.16/FilterInputStream.java:107) at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "Exec Stream Pumper" #163 daemon prio=5 os_prio=0 cpu=0.85ms elapsed=309.45s tid=0x00007f129c004800 nid=0x862 runnable [0x00007f12fa6e7000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(java.base@11.0.16/Native Method) at java.io.FileInputStream.read(java.base@11.0.16/FileInputStream.java:279) at java.io.BufferedInputStream.fill(java.base@11.0.16/BufferedInputStream.java:252) at java.io.BufferedInputStream.read1(java.base@11.0.16/BufferedInputStream.java:292) at java.io.BufferedInputStream.read(java.base@11.0.16/BufferedInputStream.java:351) - locked <0x0000000716202c18> (a java.lang.ProcessImpl$ProcessPipeInputStream) at java.io.FilterInputStream.read(java.base@11.0.16/FilterInputStream.java:107) at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "HttpClient-8-SelectorManager" #164 daemon prio=5 os_prio=0 cpu=162.91ms elapsed=309.45s tid=0x00007f130c00c000 nid=0x865 runnable [0x00007f12fbdfe000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x00000007162005c8> (a sun.nio.ch.Util$2) - locked <0x0000000716200570> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:136) at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(java.net.http@11.0.16/HttpClientImpl.java:867) "ForkJoinPool.commonPool-worker-23" #167 daemon prio=5 os_prio=0 cpu=14.32ms elapsed=307.62s tid=0x00007f1294001000 nid=0x93f waiting on condition [0x00007f12f8cdc000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000070f2008e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.16/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.16/ForkJoinWorkerThread.java:183) "nioEventLoopGroup-3-9" #169 prio=10 os_prio=0 cpu=1.99ms elapsed=163.67s tid=0x00007f1324063800 nid=0x95c runnable [0x00007f12fb7fa000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@11.0.16/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.16/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.16/SelectorImpl.java:124) - locked <0x000000070f520e28> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x000000070f520e40> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@11.0.16/SelectorImpl.java:141) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "pool-3-thread-7" #170 prio=5 os_prio=0 cpu=1.55ms elapsed=163.65s tid=0x00007f12d800d000 nid=0x95d waiting on condition [0x00007f13a0a64000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method) - parking to wait for <0x000000071647baa8> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:1079) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.16/AbstractQueuedSynchronizer.java:1369) at java.util.concurrent.CountDownLatch.await(java.base@11.0.16/CountDownLatch.java:278) at org.openqa.selenium.grid.sessionqueue.local.LocalNewSessionQueue.addToQueue(LocalNewSessionQueue.java:199) at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:69) at org.openqa.selenium.grid.sessionqueue.NewSessionQueue$$Lambda$830/0x0000000840329840.execute(Unknown Source) at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:123) at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86) at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64) at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.grid.router.Router.execute(Router.java:91) at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34) at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders$$Lambda$816/0x0000000840315c40.execute(Unknown Source) at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64) at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35) at org.openqa.selenium.remote.AddWebDriverSpecHeaders$$Lambda$794/0x0000000840269840.execute(Unknown Source) at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44) at org.openqa.selenium.remote.ErrorFilter$$Lambda$793/0x000000084026a440.execute(Unknown Source) at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64) at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44) at org.openqa.selenium.remote.ErrorFilter$$Lambda$793/0x000000084026a440.execute(Unknown Source) at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64) at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44) at org.openqa.selenium.netty.server.SeleniumHandler$$Lambda$792/0x000000084026a040.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.16/Executors.java:515) at java.util.concurrent.FutureTask.run(java.base@11.0.16/FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.16/Thread.java:829) "VM Thread" os_prio=0 cpu=848.94ms elapsed=2774.23s tid=0x00007f13e8246800 nid=0x5b runnable "GC Thread#0" os_prio=0 cpu=455.88ms elapsed=2774.28s tid=0x00007f13e802e000 nid=0x35 runnable "GC Thread#1" os_prio=0 cpu=441.61ms elapsed=2773.63s tid=0x00007f13b0099000 nid=0x80 runnable "GC Thread#2" os_prio=0 cpu=461.52ms elapsed=2773.63s tid=0x00007f13b009a800 nid=0x81 runnable "GC Thread#3" os_prio=0 cpu=459.42ms elapsed=2773.63s tid=0x00007f13b009c000 nid=0x82 runnable "GC Thread#4" os_prio=0 cpu=466.64ms elapsed=2773.63s tid=0x00007f13b009d800 nid=0x83 runnable "GC Thread#5" os_prio=0 cpu=433.97ms elapsed=2773.63s tid=0x00007f13b009f000 nid=0x84 runnable "GC Thread#6" os_prio=0 cpu=434.19ms elapsed=2711.09s tid=0x00007f13b0097800 nid=0x1b9 runnable "GC Thread#7" os_prio=0 cpu=450.03ms elapsed=2711.09s tid=0x00007f13b0132000 nid=0x1ba runnable "GC Thread#8" os_prio=0 cpu=418.73ms elapsed=2711.09s tid=0x00007f13b0133000 nid=0x1bb runnable "GC Thread#9" os_prio=0 cpu=442.97ms elapsed=2711.09s tid=0x00007f13b0134800 nid=0x1bc runnable "GC Thread#10" os_prio=0 cpu=417.81ms elapsed=2711.09s tid=0x00007f13b0136000 nid=0x1bd runnable "GC Thread#11" os_prio=0 cpu=422.60ms elapsed=2711.09s tid=0x00007f13b0137800 nid=0x1be runnable "GC Thread#12" os_prio=0 cpu=338.28ms elapsed=2333.65s tid=0x00007f13b0090800 nid=0x2de runnable "G1 Main Marker" os_prio=0 cpu=54.68ms elapsed=2774.28s tid=0x00007f13e808c800 nid=0x36 runnable "G1 Conc#0" os_prio=0 cpu=1123.19ms elapsed=2774.28s tid=0x00007f13e808e000 nid=0x37 runnable "G1 Conc#1" os_prio=0 cpu=1118.10ms elapsed=2745.93s tid=0x00007f13bc001000 nid=0x98 runnable "G1 Conc#2" os_prio=0 cpu=1111.79ms elapsed=2745.93s tid=0x00007f13bc002000 nid=0x99 runnable "G1 Refine#0" os_prio=0 cpu=1.13ms elapsed=2774.28s tid=0x00007f13e81a1000 nid=0x38 runnable "G1 Young RemSet Sampling" os_prio=0 cpu=788.48ms elapsed=2774.28s tid=0x00007f13e81a3000 nid=0x39 runnable "VM Periodic Task Thread" os_prio=0 cpu=2451.07ms elapsed=2774.20s tid=0x00007f13e8292000 nid=0x64 waiting on condition JNI global refs: 30, weak refs: 0 Heap garbage-first heap total 256000K, used 111946K [0x0000000708400000, 0x0000000800000000) region size 1024K, 40 young (40960K), 1 survivors (1024K) Metaspace used 30744K, capacity 32395K, committed 32640K, reserved 1077248K class space used 3669K, capacity 4382K, committed 4480K, reserved 1048576K ```

[uhafner]

Seems to be fixed now, see https://github.com/jenkinsci/warnings-ng-plugin/pull/1643.

[jtnord]

https://github.com/SeleniumHQ/selenium/pull/12848 ?