search

SeleniumHQ / docker-selenium

Provides a simple way to run Selenium Grid with Chrome, Firefox, and Edge using Docker, making it easier to perform browser automation
http://www.selenium.dev/docker-selenium/
Other
7.86k stars 2.51k forks source link

[🐛 Bug]: On kubernetes, the distributor doesn't answer after a while. #1768

Closed qalinn closed 1 year ago

qalinn commented 1 year ago

What happened?

I have deployed Selenium Grid 4 using the helm repo and I noticed after a while the Router doesn't respond, and on logs I can see a lot of warnings "Unable to execute request for an existing session: java.net.ConnectException: Connection refused " . The first warning is different:

10:40:54.110 WARN [DefaultChannelPipeline.onUnhandledInboundException] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.lang.NullPointerException at org.openqa.selenium.netty.server.RequestConverter.channelRead0(RequestConverter.java:131) at org.openqa.selenium.netty.server.RequestConverter.channelRead0(RequestConverter.java:55) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)

After that no tests can be executed Please see the whole log. It is happened from time to time. I am running 6 tests in parallel. Sometimes works. The pod is available on Kubernetes but the service doesn't respond. I know this because keda is trying to communicate and is receiving: Connection refused.

Command used to start Selenium Grid with Docker

helm install selenium-grid docker-selenium/selenium-grid --set isolateComponents=true

Relevant log output

2023-01-11 09:18:54,680 INFO Included extra file "/etc/supervisor/conf.d/selenium-grid-router.conf" during parsing
2023-01-11 09:18:54,684 INFO RPC interface 'supervisor' initialized
2023-01-11 09:18:54,684 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-11 09:18:54,685 INFO supervisord started with pid 8
2023-01-11 09:18:55,687 INFO spawned: 'selenium-grid-router' with pid 10
Starting Selenium Grid Router...
Tracing is disabled
2023-01-11 09:18:55,693 INFO success: selenium-grid-router entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
09:18:56.287 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
09:18:56.295 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
09:18:57.494 INFO [RouterServer.createHandlers] - Requiring authentication to connect
09:18:57.695 INFO [RouterServer.execute] - Started Selenium Router 4.7.2 (revision 4d4020c3b7): http://192.168.145.96:4444
10:40:54.110 WARN [DefaultChannelPipeline.onUnhandledInboundException] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.NullPointerException
    at org.openqa.selenium.netty.server.RequestConverter.channelRead0(RequestConverter.java:131)
    at org.openqa.selenium.netty.server.RequestConverter.channelRead0(RequestConverter.java:55)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
    at org.openqa.selenium.netty.server.WebSocketUpgradeHandler.channelRead(WebSocketUpgradeHandler.java:100)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
    at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
    at io.netty.handler.codec.http.HttpServerKeepAliveHandler.channelRead(HttpServerKeepAliveHandler.java:64)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
12:02:04.535 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: java.net.ConnectException: Connection refused
12:02:04.540 WARN [SeleniumSpanExporter$1.lambda$export$1] - java.io.UncheckedIOException: java.net.ConnectException: Connection refused
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:296)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:92)
    at org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)
    at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.ConnectException: Connection refused
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:561)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 27 more
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
    at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent.handle(PlainHttpConnection.java:128)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)

12:02:04.543 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "a09f2d1ade108ae01d4659526f1be589","eventTime": 1673438524534138760,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: java.net.ConnectException: Connection refused","exception.stacktrace": "java.io.UncheckedIOException: java.net.ConnectException: Connection refused\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:296)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:92)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.ConnectException: Connection refused\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:561)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 27 more\nCaused by: java.net.ConnectException: Connection refused\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)\n\tat java.net.http\u002fjdk.internal.net.http.PlainHttpConnection$ConnectEvent.handle(PlainHttpConnection.java:128)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)\n\tat java.base\u002fjava.util.ArrayList.forEach(ArrayList.java:1541)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)\n","exception.type": "java.io.UncheckedIOException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "selenium.mydomain.com:4444","http.method": "POST","http.request_content_length": "272","http.scheme": "HTTP","http.target": "\u002fsession\u002f93702e22d626ad3f86b46738a647f9e3\u002furl","http.user_agent": "selenium\u002f4.7.2 (java unix)","session.id": "93702e22d626ad3f86b46738a647f9e3"}}

12:02:04.584 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: java.net.ConnectException: Connection refused
12:02:04.585 WARN [SeleniumSpanExporter$1.lambda$export$1] - java.io.UncheckedIOException: java.net.ConnectException: Connection refused
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:296)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:92)
    at org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)
    at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.ConnectException: Connection refused
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:561)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 27 more
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
    at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent.handle(PlainHttpConnection.java:128)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)

12:02:04.586 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "ba880acd923f720002bb7a15053a2996","eventTime": 1673438524584280444,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: java.net.ConnectException: Connection refused","exception.stacktrace": "java.io.UncheckedIOException: java.net.ConnectException: Connection refused\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:296)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:92)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.ConnectException: Connection refused\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:561)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 27 more\nCaused by: java.net.ConnectException: Connection refused\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)\n\tat java.net.http\u002fjdk.internal.net.http.PlainHttpConnection$ConnectEvent.handle(PlainHttpConnection.java:128)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)\n\tat java.base\u002fjava.util.ArrayList.forEach(ArrayList.java:1541)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)\n","exception.type": "java.io.UncheckedIOException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "selenium.mydomain.com:4444","http.method": "GET","http.scheme": "HTTP","http.target": "\u002fsession\u002f93702e22d626ad3f86b46738a647f9e3\u002fscreenshot","http.user_agent": "selenium\u002f4.7.2 (java unix)","session.id": "93702e22d626ad3f86b46738a647f9e3"}}

12:02:04.618 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: java.net.ConnectException: Connection refused
12:02:04.619 WARN [SeleniumSpanExporter$1.lambda$export$1] - java.io.UncheckedIOException: java.net.ConnectException: Connection refused
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:296)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:92)
    at org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)
    at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.ConnectException: Connection refused
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:561)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 27 more
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
    at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent.handle(PlainHttpConnection.java:128)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)

12:02:04.620 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "2949d1eefd7fdbeaaabcd3c210c57333","eventTime": 1673438524617593414,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: java.net.ConnectException: Connection refused","exception.stacktrace": "java.io.UncheckedIOException: java.net.ConnectException: Connection refused\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:296)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:92)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.ConnectException: Connection refused\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:561)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 27 more\nCaused by: java.net.ConnectException: Connection refused\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)\n\tat java.net.http\u002fjdk.internal.net.http.PlainHttpConnection$ConnectEvent.handle(PlainHttpConnection.java:128)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)\n\tat java.base\u002fjava.util.ArrayList.forEach(ArrayList.java:1541)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)\n","exception.type": "java.io.UncheckedIOException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "selenium.mydomain.com:4444","http.method": "DELETE","http.scheme": "HTTP","http.target": "\u002fsession\u002f93702e22d626ad3f86b46738a647f9e3","http.user_agent": "selenium\u002f4.7.2 (java unix)","session.id": "93702e22d626ad3f86b46738a647f9e3"}}

Operating System

linux

Docker Selenium version (tag)

4.7.2-20221219

diemol commented 1 year ago

Connection refused means that Grid components cannot talk to each other. When it happens, are you double checking the different components can connect between themselves?

qalinn commented 1 year ago

@diemol Could you, please, tell me how should I try?

diemol commented 1 year ago

You can go inside the pods and make a curl request to the other ones. Also, the /status endpoint should be giving you information about all components. Also, please check your networking rules in the Kubernetes cluster.

qalinn commented 1 year ago

@diemol I was trying from distributor to router: curl http://192.168.229.67:4444/readyz and returned: "Router is true". Could you tell me how can I call the endpoit /status, please? I have tryied curl http://192.168.229.67:4444/status and nothing is retourned.

That ip(192.168.229.67) is the router pod ip.

diemol commented 1 year ago

If /readyz works, /status should work. What do you see in the logs of the Router?

qalinn commented 1 year ago

@diemol I can see the error what I have pasted on above. This time it happened after 3 days

diemol commented 1 year ago

I meant when you check the /status endpoint.

qalinn commented 1 year ago

@diemol I have checked the /status from router to all pods and it seems to be ok:

 curl http://192.168.129.243:5559/status
{
  "value": {
    "ready": true,
    "message": "New Session Queue is ready."
  }
}seluser@selenium-router-696f97cf48-tnsjs:/$ curl http://192.168.179.204:5553/status
{
  "value": {
    "ready": true,
    "message": "Distributor is ready"
  }
}seluser@selenium-router-696f97cf48-tnsjs:/$ curl http://192.168.204.139:5557/status
{
  "value": {
    "ready": true,
    "message": "Event bus running"
  }
}seluser@selenium-router-696f97cf48-tnsjs:/$ curl http://192.168.183.111:5556/status
{
  "value": {
    "ready": true,
    "message": "Session map is ready."
  }
}seluser@selenium-router-696f97cf48-tnsjs:/$ curl http://192.168.129.243:5559/status
{
  "value": {
    "ready": true,
    "message": "New Session Queue is ready."
  }
}

But still have the issue. I have also restarted the router pod and the issue is still there: 

14:21:31.457 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
14:21:31.457 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more

14:21:31.457 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "1dadb9796bb21ed1b0683662d7d59b68","eventTime": 1673878891456825132,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.http.HttpTimeoutException: request timed out\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 34 more\n","exception.type": "org.openqa.selenium.TimeoutException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue","http.host": "mySeleniumgrid4.com:4444","http.method": "POST","http.request_content_length": "1269","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "selenium\u002f4.7.2 (java unix)"}}

14:21:31.464 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more
14:21:31.465 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
14:21:31.465 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more

14:21:31.465 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "1e05fe54950f6271b622d9c8715bc912","eventTime": 1673878891463496066,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.http.HttpTimeoutException: request timed out\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 34 more\n","exception.type": "org.openqa.selenium.TimeoutException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue","http.host": "mySeleniumgrid4.com:4444","http.method": "POST","http.request_content_length": "1269","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "selenium\u002f4.7.2 (java unix)"}}

14:21:44.474 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more
14:21:44.474 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
14:21:44.474 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more

14:21:44.475 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "c1c437daa85c998b4ee6223141fed639","eventTime": 1673878904473485262,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.http.HttpTimeoutException: request timed out\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 34 more\n","exception.type": "org.openqa.selenium.TimeoutException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue","http.host": "mySeleniumgrid4.com:4444","http.method": "POST","http.request_content_length": "1269","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "selenium\u002f4.7.2 (java unix)"}}

14:21:48.458 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more
14:21:48.458 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
14:21:48.458 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more

14:21:48.459 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "ff416a1393dd17da042e9d4d6194d178","eventTime": 1673878908457678802,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.http.HttpTimeoutException: request timed out\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 34 more\n","exception.type": "org.openqa.selenium.TimeoutException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue","http.host": "mySeleniumgrid4.com:4444","http.method": "POST","http.request_content_length": "1269","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "selenium\u002f4.7.2 (java unix)"}}

14:22:03.082 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more
14:22:03.082 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
14:22:03.082 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more

14:22:03.083 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "cb65c1371316bb693cd5fbc990acab87","eventTime": 1673878923081557661,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.http.HttpTimeoutException: request timed out\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 34 more\n","exception.type": "org.openqa.selenium.TimeoutException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue","http.host": "mySeleniumgrid4.com:4444","http.method": "POST","http.request_content_length": "1269","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "selenium\u002f4.7.2 (java unix)"}}

14:22:44.460 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more
14:22:44.461 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
14:22:44.461 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more

14:22:44.461 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "d637857401862b68dacc299d9c227083","eventTime": 1673878964459657971,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.http.HttpTimeoutException: request timed out\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 34 more\n","exception.type": "org.openqa.selenium.TimeoutException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue","http.host": "mySeleniumgrid4.com:4444","http.method": "POST","http.request_content_length": "1269","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "selenium\u002f4.7.2 (java unix)"}}

14:22:48.452 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more
14:22:48.452 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
14:22:48.452 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out
Build info: version: '4.7.2', revision: '4d4020c3b7'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)
    at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
    at org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
    at org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)
    at org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)
    at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.router.Router.execute(Router.java:91)
    at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)
    at org.openqa.selenium.remote.http.Route.execute(Route.java:68)
    at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
    at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)
    at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.http.HttpTimeoutException: request timed out
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)
    ... 34 more

14:22:48.453 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "74cd87fdb33c7a05b41c7d00efe7d4f7","eventTime": 1673878968451063980,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.TimeoutException: java.net.http.HttpTimeoutException: request timed out\nBuild info: version: '4.7.2', revision: '4d4020c3b7'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.219-126.411.amzn2.x86_64', java.version: '11.0.17'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:294)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.security.AddSecretFilter.lambda$apply$0(AddSecretFilter.java:40)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:68)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:122)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:55)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.http.HttpTimeoutException: request timed out\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)\n\tat java.net.http\u002fjdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)\n\tat org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:292)\n\t... 34 more\n","exception.type": "org.openqa.selenium.TimeoutException","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue","http.host": "mySeleniumgrid4.com:4444","http.method": "POST","http.request_content_length": "1269","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "selenium\u002f4.7.2 (java unix)"}}
qalinn commented 1 year ago

@diemol Do you have any ide what something else should I check?

diemol commented 1 year ago

org.openqa.selenium.grid.sessionqueue.remote.RemoteNewSessionQueue.addToQueue(RemoteNewSessionQueue.java:99) means that a session request timed out while waiting in the queue to be served. You should have seen a message about that in the logs. I think you are mixing two things, at one point you got the connection refused because something in the cluster had an erroneous setup, now you are getting timeouts because the session request is waiting for too long before it gets processed.

Please check the values for the Queue https://www.selenium.dev/documentation/grid/configuration/cli_options/#sessionqueue. You also need to have similar timeouts on the client side. If you need help understanding those configuration settings, please join us in one of the chats shown at https://www.selenium.dev/support/

qalinn commented 1 year ago

@diemol Hello, after a careful investigation, I noticed the distributor is somehow unresponsive. I have checked to see if I can get an answer from router to distributor and it seems to be fine this: seluser@selenium-router-7f8bf897d7-nnf8b:/$ curl 192.168.140.27:5553/status { "value": { "ready": true, "message": "Distributor is ready" } }

On the other hand, if I am checking chrome node logs I can see the following:

`kubectl logs -f -n selenium selenium-chrome-node-6b9f56ddc5-pwmxz 2023-01-30 12:38:08,754 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing 2023-01-30 12:38:08,766 INFO RPC interface 'supervisor' initialized 2023-01-30 12:38:08,766 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2023-01-30 12:38:08,767 INFO supervisord started with pid 9 2023-01-30 12:38:09,769 INFO spawned: 'xvfb' with pid 11 2023-01-30 12:38:09,773 INFO spawned: 'vnc' with pid 12 2023-01-30 12:38:09,775 INFO spawned: 'novnc' with pid 13 2023-01-30 12:38:09,781 INFO spawned: 'selenium-node' with pid 16 2023-01-30 12:38:09,811 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) Generating Selenium Config Configuring server... Setting up SE_NODE_HOST... Setting up SE_NODE_PORT... Setting up SE_NODE_GRID_URL... Tracing is disabled Selenium Grid Node configuration: [events] publish = "tcp://selenium-event-bus:4442" subscribe = "tcp://selenium-event-bus:4443"

[node] session-timeout = "300" override-max-sessions = false detect-drivers = false drain-after-session-count = 0 max-sessions = 1

[[node.driver-configuration]] display-name = "chrome" stereotype = '{"browserName": "chrome", "browserVersion": "109.0", "platformName": "Linux"}' max-sessions = 1

Starting Selenium Grid Node... 2023-01-30 12:38:10,854 INFO success: xvfb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-30 12:38:10,854 INFO success: vnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-30 12:38:10,854 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 12:38:11.586 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding 12:38:11.595 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing 12:38:12.075 INFO [UnboundZmqEventBus.] - Connecting to tcp://selenium-event-bus:4442 and tcp://selenium-event-bus:4443 12:38:12.283 INFO [UnboundZmqEventBus.] - Sockets created 12:38:13.287 INFO [UnboundZmqEventBus.] - Event bus ready 12:38:13.577 INFO [NodeServer.createHandlers] - Reporting self as: http://192.168.134.73:5555 12:38:13.623 INFO [NodeOptions.getSessionFactories] - Detected 1 available processors 12:38:13.759 INFO [NodeOptions.report] - Adding chrome for {"browserVersion": "109.0","se:noVncPort": 7900,"browserName": "chrome","platformName": "LINUX","se:vncEnabled": true} 1 times 12:38:13.787 INFO [Node.] - Binding additional locator mechanisms: relative 12:38:14.277 INFO [NodeServer$1.start] - Starting registration process for Node http://192.168.134.73:5555 12:38:14.281 INFO [NodeServer.execute] - Started Selenium node 4.8.0 (revision 267030adea): http://192.168.134.73:5555 12:38:14.299 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:38:24.303 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:38:34.306 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:38:44.309 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:38:54.311 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:39:04.313 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:39:14.317 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:39:24.324 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 12:39:34.327 INFO [NodeServer$1.lambda$start$1] - Sending registration event...`

And in the router, I cannot see the chrome browser under overview page.

I checked the sessionqueue configuration and is similar to what is on the documentation. I tried version 4.8.0 with keda plugin.

Thank you!

diemol commented 1 year ago

That is because the Distributor cannot reach the Node, maybe the network rules are not updated and therefore there is no route between the Distributor and the launched Node?

Also check the resources for the Distributor, if it has a single CPU and a single GB of RAM, it could suffer while creating sessions.

qalinn commented 1 year ago

@diemol regarding the network rules, I cannot understand what should be updated, because is working for a few days and after that is happening what I have described above. If I am restarting the distributor (killing and starting again) all the sessions which are in the queue are registered and work perfectly (the same if I am creating a new session, will work after that).

The distributor has the default values how is on GitHub, on helm charts. Shall I try also something else?

Thank you for your fast reply!

diemol commented 1 year ago

If it works sometimes and then it does not, most likely it has to be with your cluster. It could be that you are going over a quota and a new Kubernetes Node is being started, maybe that is why it takes longer? That is something you need to check in your environment.

The distributor has the default values how is on GitHub, on helm charts.

Those are default values, you only know the loads your Grid manages. Are you recording metrics and checking when the Distributor becomes unresponsive? What is the CPU load and RAM usage when that happens?

qalinn commented 1 year ago

I don't have more than 20 chrome nodes at the same time. Now I have set 3 CPU and 8Gi. I will let you know next few days if is still happening. I will check the CPU and RAM usage when will happen.

qalinn commented 1 year ago

@diemol

I have again this issue and for sure is not from my k8s network, I am the only one who is administrating this k8s cluster. I have checked how much memory and CPU is consuming:

NAME                                      CPU(cores)   MEMORY(bytes)   
selenium-chrome-node-6b9f56ddc5-2ddgl     2m           130Mi           
selenium-chrome-node-6b9f56ddc5-2w44s     2m           130Mi           
selenium-chrome-node-6b9f56ddc5-5g8nz     3m           131Mi           
selenium-chrome-node-6b9f56ddc5-9phqp     2m           135Mi           
selenium-chrome-node-6b9f56ddc5-b4wrw     3m           132Mi           
selenium-chrome-node-6b9f56ddc5-bz6kz     2m           129Mi           
selenium-chrome-node-6b9f56ddc5-c7srr     5m           131Mi           
selenium-chrome-node-6b9f56ddc5-g9jcc     2m           137Mi           
selenium-chrome-node-6b9f56ddc5-h5fnp     3m           131Mi           
selenium-chrome-node-6b9f56ddc5-hltgp     2m           129Mi           
selenium-chrome-node-6b9f56ddc5-j2d4l     3m           130Mi           
selenium-chrome-node-6b9f56ddc5-m2bvs     3m           131Mi           
selenium-chrome-node-6b9f56ddc5-mv2dh     2m           138Mi           
selenium-chrome-node-6b9f56ddc5-nbpg8     2m           134Mi           
selenium-chrome-node-6b9f56ddc5-plz92     2m           130Mi           
selenium-chrome-node-6b9f56ddc5-ptsls     2m           130Mi           
selenium-chrome-node-6b9f56ddc5-q8xcs     2m           130Mi           
selenium-chrome-node-6b9f56ddc5-rb25n     3m           135Mi           
selenium-chrome-node-6b9f56ddc5-s8t7m     2m           131Mi           
selenium-chrome-node-6b9f56ddc5-sdffk     3m           136Mi           
selenium-chrome-node-6b9f56ddc5-vf8gg     2m           142Mi           
selenium-chrome-node-6b9f56ddc5-vsbbq     2m           130Mi           
selenium-chrome-node-6b9f56ddc5-x8h4j     3m           131Mi           
selenium-chrome-node-6b9f56ddc5-xbkwv     3m           131Mi           
**selenium-distributor-6f5cfd77b6-q6hcp     7m           991Mi           
selenium-event-bus-fff659546-t49wd        2m           228Mi           
selenium-router-7f8bf897d7-pzl4d          13m          1272Mi          
selenium-session-map-579974fffb-9pncj     2m           308Mi           
selenium-session-queue-6d9f769955-2hw5x   14m          618Mi**

As I mentioned above I have assigned 3CPU and 8Gi only for the distributor. Now is happening again after 6 days.

I can see on the router a lot of nodes (these chrome nodes were created because my tests asked for them) but cannot be used because the session cannot be created.

I have restarted the distributor and everything started to work correctly. What I noticed is that the memory is increasing during these days and after the distributor was restarted this is the memory consumption: selenium-distributor-6f5cfd77b6-65rfj 97m 354Mi

Could you tell me how can I investigate further?

qalinn commented 1 year ago

@diemol Is happening again. The same stuff, before restarting the distributor the chrome node cannot register and is sending registration evenst: `2023-02-21 09:05:39,338 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing 2023-02-21 09:05:39,353 INFO RPC interface 'supervisor' initialized 2023-02-21 09:05:39,353 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2023-02-21 09:05:39,354 INFO supervisord started with pid 8 2023-02-21 09:05:40,356 INFO spawned: 'xvfb' with pid 10 2023-02-21 09:05:40,358 INFO spawned: 'vnc' with pid 11 2023-02-21 09:05:40,360 INFO spawned: 'novnc' with pid 12 2023-02-21 09:05:40,363 INFO spawned: 'selenium-node' with pid 13 2023-02-21 09:05:40,380 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) Generating Selenium Config Configuring server... Setting up SE_NODE_HOST... Setting up SE_NODE_PORT... Setting up SE_NODE_GRID_URL... Tracing is disabled Selenium Grid Node configuration: [events] publish = "tcp://selenium-event-bus:4442" subscribe = "tcp://selenium-event-bus:4443"

[node] session-timeout = "300" override-max-sessions = false detect-drivers = false drain-after-session-count = 0 max-sessions = 1

[[node.driver-configuration]] display-name = "chrome" stereotype = '{"browserName": "chrome", "browserVersion": "109.0", "platformName": "Linux"}' max-sessions = 1

Starting Selenium Grid Node... 2023-02-21 09:05:41,392 INFO success: xvfb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-02-21 09:05:41,392 INFO success: vnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-02-21 09:05:41,392 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 09:05:42.245 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding 09:05:42.251 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing 09:05:42.644 INFO [UnboundZmqEventBus.] - Connecting to tcp://selenium-event-bus:4442 and tcp://selenium-event-bus:4443 09:05:42.781 INFO [UnboundZmqEventBus.] - Sockets created 09:05:43.784 INFO [UnboundZmqEventBus.] - Event bus ready 09:05:44.065 INFO [NodeServer.createHandlers] - Reporting self as: http://192.168.212.134:5555 09:05:44.159 INFO [NodeOptions.getSessionFactories] - Detected 1 available processors 09:05:44.322 INFO [NodeOptions.report] - Adding chrome for {"browserVersion": "109.0","se:noVncPort": 7900,"browserName": "chrome","platformName": "LINUX","se:vncEnabled": true} 1 times 09:05:44.365 INFO [Node.] - Binding additional locator mechanisms: relative 09:05:44.752 INFO [NodeServer$1.start] - Starting registration process for Node http://192.168.212.134:5555 09:05:44.754 INFO [NodeServer.execute] - Started Selenium node 4.8.0 (revision 267030adea): http://192.168.212.134:5555 09:05:44.768 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:05:54.773 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:06:04.775 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:06:14.777 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:06:24.782 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:06:34.784 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:06:44.788 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:06:54.791 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:07:04.794 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:07:14.799 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:07:24.801 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:07:34.804 INFO [NodeServer$1.lambda$start$1] - Sending registration event... 09:07:44.756 INFO [NodeServer$1.lambda$start$1] - Sending registration event... `

After the distributor was restated the node is registered fine: 09:09:44.875 INFO [NodeServer.lambda$createHandlers$2] - Node has been added

Is someone experiencing the same issue?

@diemol can I try something else? any suggestion?

diemol commented 1 year ago

As per the Slack conversation, please correlate the logs between Nodes when the Sending registration event... message is seen, and the Distributor with FINE log level.

With that you will be able to find out what the issue is, which is either the new Nodes are not being able to reach the Event Bus because the message/event is not show in the Distributor logs, or the event actually is received by the Distributor but while trying to ping the new Node a network error happens (a stack trace will show that).

diemol commented 1 year ago

Closing as no more feedback was provided.

github-actions[bot] commented 9 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.