Closed pandoras-toolbox closed 9 months ago
@pandoras-toolbox, thank you for creating this issue. We will troubleshoot it as soon as we can.
Triage this issue by using labels.
If information is missing, add a helpful comment and then I-issue-template
label.
If the issue is a question, add the I-question
label.
If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted
label.
If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C),
add the applicable G-*
label, and it will provide the correct link and auto-close the
issue.
After troubleshooting the issue, please add the R-awaiting answer
label.
Thank you!
Hi, @pandoras-toolbox. Please follow the issue template, we need more information to reproduce the issue.
Either a complete code snippet and URL/HTML (if more than one file is needed, provide a GitHub repo and instructions to run the code), the specific versions used, or a more detailed description to help us understand the issue.
Note: If you cannot share your code and URL/HTML, any complete code snippet and URL/HTML that reproduces the issue is good enough.
Reply to this issue when all information is provided, thank you.
Share the commands you are using to start the Grid and the test you are using to get that error.
I do not know how the Grid is started, it is done by people from operations, it is basically a YAML file which is redeployed whenever changes are merged to master branch, then a ArgoCD job starts automatically to apply the changes.
I wonder if that error is supposed to exist or not. Because if it is intended behavior, then I think you are aware of how to reproduce it. So I assume it is not a feature for you but a bug.
Sorry, but to create the code snippet in Java and all the other things in the whole ping pong until it is maybe solved by you, I am afraid that is more effort here to provide you all that stuff, because our test code was never made for that, it is no open source project but closed, I would have quite some effort into extracting something for you guys.
Then forget about it, i will run the gauntlets and provide that stupid HTTPS certificate, because in the end that is less effort as I see now than to give you all the extra information.
Well, we are here to help, we have tons of users who have been successfully upgraded to Grid 4. We have been helping them whenever questions come, but without details that help reproduce the error, it is hard to help.
I wonder if that error is supposed to exist or not. Because if it is intended behavior, then I think you are aware of how to reproduce it. So I assume it is not a feature for you but a bug.
Hard to say, it obviously depends on your environment.
I will leave this open in case you want to provide more information.
@pandoras-toolbox - Maybe you can help share these specific information (which is what @diemol is also asking)
This is just a guess. But from the stacktrace that you have shared, it looks to me that you need to have a keystore setup with the certificate that is being used to enable https at the Grid end.
FYI.,
If you run java -jar selenium-server-4.15.0.jar info security
you should see a whole lot of information around enabling https at your grid end. Maybe you can refer to that output to familiarise yourself with the details so that it becomes easy for you to ask your operations team on how the https is being setup at your grid end and also to get hold of the certificate.
I understand that you want to reproduce it. Perhaps I am the 1 percent. For me it would be solved if I can configure the Grid to ignore any SSL problems, if that is possible. I will now take a deeper look at the issue.
If I find something out I will let you know.
For me it would be solved if I can configure the Grid to ignore any SSL problems, if that is possible.
I think that the easiest way of getting past this problem is to grab the certificate that is being used to enable https at your grid end, and add it to your local Java keystore. That should fix the issue. The problem is coming up because your Grid is https enabled, its serving a self signed certificate that is NOT marked as a trusted certificate in your local Java Key store and so Java is refusing to allow you to connect to the https end point (Grid in this case)
Thanks @krmhadevan. But wouldn't it be easier not to enable HTTPS on the Grid? If I knew how.
Locally I can NOT reproduce the problem. I have set Selenium version to be 4.15.0 in the Maven POM file, on my client side.
This is how I installed and started the Grid locally on Fedora Linux:
mkdir selenium
cd selenium
# Download and extract Chromedriver in the right version for your computer
# (see: https://googlechromelabs.github.io/chrome-for-testing/):
chromedriver_version="119.0.6045.105"
wget https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/$chromedriver_version/linux64/chromedriver-linux64.zip --continue
unzip chromedriver-linux64.zip
rm chromedriver-linux64.zip
PATH=$(pwd)/chromedriver-linux64/chromedriver:$PATH
# Download and start Selenium Grid:
selenium_major_version="4.14.0"
selenium_server_version="4.14.1"
wget https://github.com/SeleniumHQ/selenium/releases/download/selenium-$selenium_major_version/selenium-server-$selenium_server_version.jar --continue
java -jar selenium-server-$selenium_server_version.jar hub --port 4444
# In a new terminal window in the same directory:
selenium_server_version="4.14.1"
java -jar selenium-server-$selenium_server_version.jar node --port 5555 --hub http://localhost:4444 --max-sessions 8
On Kubernetes we use 4.15.0 for the Grid, but the problem started with 4.14.0, so I think it does not matter the little version discrepancy between the local and remote Grid situation. But I will retest again, and if it is a different behavior locally with Grid 4.15.0 I will comment.
@pandoras-toolbox - Thanks for adding those details. The steps that you shared to start the hub and the node, will only run a grid working with http
. So this is fine.
You mentioned that you are using K8s for managing the grid. Would it be possible for you to please help share either the deployment yaml file or the helm chart that you are using to deploy the grid?
Also I remember you mentioning gauntlet in your earlier posts. Can you please share some additional details in terms of what role it plays?
There is something in your k8s cluster that is enabling https on the Grid.
@krmahadevan I need to first ask for permission to attach the YAML file here. With "gauntlet" I meant the English phrase "run the gauntlet" in a metaphoric way, when you Google it you will see what it means.
@pandoras-toolbox
With "gauntlet" I meant the English phrase "run the gauntlet" in a metaphoric way, when you Google it you will see what it means.
Lol! Silly me. I just tied it to that tool (Strange co-incidence aint it) ! Glad one of the confusions stands ruled out. We now have to just figure out what in your k8s cluster is enabling https :)
Thanks for clarifying the "gauntlet" part :)
I still do not know if I can share the Kubernetes YAML.
I had a wrong assumption, that we do not have HTTPS for the Selenium Grid. But we do have HTTPS. That has changed recently, before it was insecure HTTP. It is nevertheless related to which versions of Selenium I use in the Maven POM.
When I export the certificate in Chrome of the Selenium Grid and import it with the "keytool" command, then it works with Selenium 4.15.0 to run a test without the errors above.
Now I need to figure out the best way to avoid the error.
I did not understand how I can apply: "When you instantiate your RemoteWebDriver, do you setup some ssl context manually by passing in a JKS file ..."
Or maybe there is a setting or VM argument to ignore such errors, which would be the easiest.
Grid 3 did not support HTTPS out of the box, you had to follow several workarounds or put a layer on top that took care of that.
Grid 4 does support HTTPS and here are instructions that show how to enable it. The workaround of putting a layer on top to enable that is still valid (which is, I believe, what your ops team is doing).
Can I do that in Java also? Because then it would work in every situation where it is run.
I mean what is done with the command below to solve in Java Selenium code:
sudo keytool \
-import \
-file /path/to/minica.pem \
-alias minica \
-keystore $JAVA_HOME/jre/lib/security/cacerts \
-storepass changeit \
-cacerts
Any ideas? I now came up with this attempt, but it fails, see error below:
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
@Override
public void checkClientTrusted(java.security.cert.X509Certificate[] chain, String authType) {
}
@Override
public void checkServerTrusted(java.security.cert.X509Certificate[] chain, String authType) {
}
@Override
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return new java.security.cert.X509Certificate[]{};
}
}
};
SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
ClientConfig clientConfig = ClientConfig.defaultConfig()
.connectionTimeout(Duration.ofSeconds(10))
.readTimeout(Duration.ofSeconds(10))
.sslContext(sslContext);
RemoteWebDriver.builder()
.address("https://url-to-our-selenium-grid.xyz")
.oneOf(new ChromeOptions())
.setCapability("ext:options", Map.of("key", "value"))
.config(clientConfig)
.build();
org.openqa.selenium.remote.http.ConnectionFailedException: JdkWebSocket initial request execution error
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.6.4-200.fc39.x86_64', java.version: '17.0.6'
Driver info: driver.version: unknown
at org.openqa.selenium.remote.http.jdk.JdkHttpClient.openSocket(JdkHttpClient.java:256)
at org.openqa.selenium.devtools.Connection.<init>(Connection.java:84)
at org.openqa.selenium.devtools.SeleniumCdpConnection.<init>(SeleniumCdpConnection.java:36)
at org.openqa.selenium.devtools.SeleniumCdpConnection.lambda$create$3(SeleniumCdpConnection.java:103)
at java.base/java.util.Optional.map(Optional.java:260)
at org.openqa.selenium.devtools.SeleniumCdpConnection.create(SeleniumCdpConnection.java:103)
at org.openqa.selenium.devtools.SeleniumCdpConnection.create(SeleniumCdpConnection.java:49)
at org.openqa.selenium.devtools.DevToolsProvider.getImplementation(DevToolsProvider.java:50)
at org.openqa.selenium.devtools.DevToolsProvider.getImplementation(DevToolsProvider.java:29)
at org.openqa.selenium.remote.Augmenter.augment(Augmenter.java:191)
at org.openqa.selenium.remote.Augmenter.augment(Augmenter.java:162)
at org.openqa.selenium.remote.RemoteWebDriverBuilder.build(RemoteWebDriverBuilder.java:371)
at com.swisssign.at.swissid.e2e.RemoteWebDriverTest.remoteWebDriverBuilder(RemoteWebDriverTest.java:133)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: java.net.http.HttpConnectTimeoutException: HTTP connect timed out
at java.net.http/jdk.internal.net.http.MultiExchange.toTimeoutException(MultiExchange.java:580)
at java.net.http/jdk.internal.net.http.MultiExchange.getExceptionalCF(MultiExchange.java:527)
at java.net.http/jdk.internal.net.http.MultiExchange.lambda$responseAsyncImpl$7(MultiExchange.java:447)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at java.net.http/jdk.internal.net.http.Http1Exchange.lambda$cancelImpl$9(Http1Exchange.java:483)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.ConnectException: HTTP connect timed out
at java.net.http/jdk.internal.net.http.MultiExchange.toTimeoutException(MultiExchange.java:581)
... 10 more
That Java code would probably only work if it was executed inside the Grid. Running it on the client side has no effect on the server side at all.
If you want to locally setup a Grid with HTTPS, you can follow the link I shared before, but I believe you should share that link with your OPS team so they add the certificates to the Grid and it starts working properly.
@diemol - This is just a guess, but it looks like we currently don't have a mechanism to pass in the SSL Context which was passed via RemoteWebDriver.builder()
to DevToolsProvider.getImplementation
We seem to ignore the sslContext
set via the builder and we are working without any ssl context
Call goes from SeleniumCdpConnection.create() to CdpEndpointFinder.getHttpClient()
Please let me know if this analysis is valid.
@pandoras-toolbox - I think you would need to do something like the sample in this stackoverflow post which will setup a default ssl context for the entire jvm. Please see if this helps.
I am not sure. As always, we need something actionable to troubleshoot.
@krmahadevan Thank you, I will try that. But wouldn't it be good if the SSL Context set in the builder would not be ignored, if that is really the case?
@pandoras-toolbox - yes. But for us to be able to confirm that hunch, we would need to have a reproducible (which is what @diemol is also expecting).
I guess you mean me to provide something reproducible. I will try to do it on basis of your own examples, but fist I need to work on something else for a bit.
I also tried to explore enabling HTTPS in Selenium in the downstream repo https://github.com/SeleniumHQ/docker-selenium/pull/2080
java -jar -Djavax.net.ssl.trustStore=/path/to/truststore.jks \
-Djavax.net.ssl.trustStorePassword=your_passphase \
-Djdk.internal.httpclient.disableHostnameVerification=true \
selenium-server.jar \
--https-private-key /path/to/privateKey.pkcs8 \
--https-certificate /path/to/cert.pem
Internally components communicate with different IPs so without
disableHostnameVerification
some error related to not match host, DNS, or SAN will be seen, so keep it disabled
Java binding
mvn test -Djavax.net.ssl.trustStore=src/test/resources/configuration/selenium.jks \
-Djavax.net.ssl.trustStorePassword=changeit \
-Djdk.internal.httpclient.disableHostnameVerification=true \
-Dselenium.hub.url="https://ci.ndviet.org/selenium"
Python binding
export REQUESTS_CA_BUNDLE=/path/to/cert.pem
export GRID_URL=https://ci.ndviet.org/selenium
python test.py
You should be able to pass these things into the Http Client classes/configs as well.
I don't know much about certs, is there a way to test whether things are being applied properly?
I do not think this is a bug in the Grid, as this has been working fine for a while; we even have the instructions on how to roll this on your own at https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/grid/commands/security.txt.
When working with certs, you always need to make the cert store on the host trust the certificate.
I will close this, and I will be happy to reopen it when we have something that we can use to reproduce it.
If you need help to get this working, join our chat https://www.selenium.dev/support/#ChatRoom.
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
What happened?
Our remote Selenium Grid is currently on version 3.15.0.
In the Maven POM I cannot update the Selenium version to 3.14.0 or later, it has to stay at 3.13.0.
Otherwise the error below would occur, meaning, all Selenium tests fail because of that.
Our Selnium Grid is NOT configured with HTTPS and certificates. Do we really now have to spend time for that stuff in order to be able to increase the Selenium versions we use? I mean we have much more important things to work on, the Selenium Grid is behind a firewall and of no concern to security for us.
Ideally I would like to configure the Grid easily so that it does not throw this error, without having to run the gauntlets to make it support HTTPS. Is that possible, and how? And why does it fail now all of a sudden in such a scenario instead of just logging a warning? And why does the release notes not say that they will break existing test automations with that version in this scenario?
How can we reproduce the issue?
Relevant log output
Operating System
Fedora 39
Selenium version
3.14.0
What are the browser(s) and version(s) where you see this issue?
Chrome 119
What are the browser driver(s) and version(s) where you see this issue?
Chromedriver 119
Are you using Selenium Grid?
3.14.0