Closed parhedberg closed 1 year ago
Still seeing this in 4.6.0 hub and nodes.
What the log shows is that the session request was sitting for a long time in the queue and it expired. Nothing possible related to certificates is present in the logs.
Maybe check how long sessions are running and if some of them are getting stuck using resources.
I am curius about that "http.status_code: 500" that I see in the logs.. Is that what happens when it is waiting to long?
There is no other sessions running when this is happening.. All that is visible in sessions page is a queue of 1 session. No active ones.
I also did setup Edge nodes, just to test but same problem there. And still only issues with the nodes running "edited images" as in added certificates.
What session timeouts should i test with?
The error comes from the queue. It responds with a 500 when the session was not created for any reason. Maybe too many sessions in parallel? What if you run them sequentially, then with two threads, and so on?
They are running seqentially from start. There are no more than 1 concurrent session, and actually no more than one node active at the time.
I have tried upping the possible concurrent browsers on the nodes, to see if the queued session tries to start a new one, but it does not.
I have also tried shortening the session timeout to 10s, but all that does is "failing faster" at starting a new session.
It is like the node does not get the request. As if communications between hub and pod stops.. In the nodes logs i dont see any requests either...
To test if it it the node that freezes i open a terminal on the node and it is responsive, so the node it-self has not died. And curling to it from the hub-node also gets a response that looks ok.
Would it be possible to use only the "homemade" nodes? It is probably hard to track the issue when they are mixed.
The tests I run when I try this is only requesting sessions with the two "homemade" nodes.. If i run other tests that requests sessions at the "vanilla" nodes, there is no issue. I could probably for testing pourposes kill all the other nodes and only have the two "homemade" nodes active.
Will see if I can do that easily..
Tried now with only 2 "homemade" nodes active on the hub. The tests alternate between these two, and works well at first. But after the test has made a logout (that includes a page.driver.quit
command) it is not able to recover and start another session on that node.
Now I also see this in the logs for the hub:
16:38:37.770 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "48284a7dbc098bcf5c76ce11878a5206","eventTime": 1667835517768345486,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: Unable to find session with ID: f4906669f1f9cb7c6f15d191d7816d9d\nBuild info: version: '4.6.0', revision: '79f1c02ae20'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-88-generic', java.version: '11.0.16'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: f4906669f1f9cb7c6f15d191d7816d9d\nBuild info: version: '4.6.0', revision: '79f1c02ae20'\nSystem info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-88-generic', java.version: '11.0.16'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:129)\n\tat org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:159)\n\tat io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)\n\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.sele nium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\n","exception.type": "org.openqa.selenium.NoSuchSessionException","http.client_ip": "10.6.3.5","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "selenium-hub.tnmt.nordicmedtest.se","http.method": "DELETE","http.scheme": "HTTP","http.target": "\u002fsession\u002ff4906669f1f9cb7c6f15d191d7816d9d\u002fcookie","http.user_agent": "selenium\u002f4.6.1 (ruby linux)","session.id": "f4906669f1f9cb7c6f15d191d7816d9d"}}
I am not sure if I saw that earlier, but i get one of these for each of the old 500 errors..
I think that I have identified the problem.
Before I did set the capabilites for the two "homemade" nodes to "personal1" and "personal2".
When i change them to this:
"networkname:applicationName" => "personal-bjorn",
"nodename:applicationName" => "personal-bjorn"
and
"networkname:applicationName" => "personal-karin",
"nodename:applicationName" => "personal-karin"
I have no more issues...
The error from above means that in your test you are trying to access a session that does not exist anymore, probably the one you just quit is having some sort of retry on the client side?
I am glad you are not having more issues, however, I did not get how it got solved, just by changing a custom capability value? Maybe you can please explain that a bit in case someone bumps into a similar issue and they can learn from your solution.
Yes.. Changing the custom capability when registering the node did the trick..
I went from:
- name: SE_NODE_STEREOTYPE
value: "{\"browserName\":\"chrome\",\"platformName\": \"Linux\", \"networkname:applicationName\":\"personal1\", \"nodename:applicationName\":\"personal1\" }"
to:
- name: SE_NODE_STEREOTYPE
value: "{\"browserName\":\"chrome\",\"platformName\": \"Linux\", \"networkname:applicationName\":\"personal-kerstin\", \"nodename:applicationName\":\"personal-kerstin\" }"
and now it works...
I could not really believe that this was the only thing, so I tried changing the edge browsers first, and changing them fixed it. So i went ahead and tried also the chrome browsers. I first thought that i had changed something else, or that the redeploy only made it work, but to test that i did a full re-deploy of the chrome nodes before i changed their names, and that did not solve it. So it is clearly the capability name change that did it.
OK, thanks for the explanation. I just checked the code and I cannot see why that was the issue. But happy you got it working. I will close this issue then.
What happened?
I have a setup with 5 nodes on a grid, all setup in kubernetes.
2 of the grids uses a special "home made" container that applies a certificate to login to our testenvironment (setup as the example here https://github.com/SeleniumHQ/docker-selenium#install-certificates-for-chromium-based-browsers) . This container works whitout problem locally. And so it does at first also in the grid environment on kubernetes. But after a while (not sure if it is timebased or after some sessions) the session request hangs, and does not get sent to the (available) node.
Maybe this is related to https://github.com/SeleniumHQ/docker-selenium/issues/1605 or https://github.com/SeleniumHQ/docker-selenium/issues/1701
The most strange thing is that the nodes that use "vanilla" images to build the pod (i.e. selenium/node-chrome from hub.docker.com and not the image that i am making with certificates that is based also based on selenium/node-chrome) does not have this problem. Som only the two nodes with added certificates have the problem.
Command used to start Selenium Grid with Docker
Relevant log output
Operating System
Windows 11 and Kubernetes
Docker Selenium version (tag)
4.5.3 both nodes and hub