sbordet
commented
2 years ago I'm not sure I understand this, as all seems unrelated to Jetty? Why it is a Jetty bug if the HSM times out?
Closed kimmerin closed 2 months ago
sbordet
commented
2 years ago I'm not sure I understand this, as all seems unrelated to Jetty? Why it is a Jetty bug if the HSM times out?
kimmerin
commented
2 years ago It seems to me that Jetty calls keystore.getKey() once and keeps the returned key forever. If it's not used for some time, the underlying session to the HSM becomes invalid and using that key leads to an error. The KeyStore itself can still be used, i.e. getting the same key again by calling getKey() again returns a usable key.
Assuming that a key-instance is valid "forever" is a bug in my eyes. If it is not, I don't see, how to configure Jetty to use KeyStores of HSM-providers in a way that this problem can't occur. As said, as soon as this problem occurs, you need to restart Jetty, which is something you don't want in production-environments.
sbordet
commented
2 years ago Jetty does not call KeyStore.getKey().
Jetty just uses the KeyStore from the JDK implementation to set values to configure it, and then the JDK implementation does the rest, which may or may not call getKey() or cache it.
Assuming that a key-instance is valid "forever" is a bug in my eyes.
You should then file an issue to the JDK, not us.
I think it's wrong to assume that a class like KeyStore which was designed to support file-system resources to load cryptographic material will behave the same when used over the network with HSMs that time out.
I don't think we can solve this from Jetty.
We certainly don't want to trigger a "refresh" of the KeyStore usage and have all other use cases to pay this cost, just to support your use case.
I think you should implement your own workaround for your own use case.
kimmerin
commented
2 years ago I think it's wrong to assume that a class like KeyStore which was designed to support file-system resources to load cryptographic material will behave the same when used over the network with HSMs that time out.
I'm not assuming anything and using a network accessed HSM (every time I write this, my head explodes) isn't a choice of mine but an external requirement. As well, the way KeyStores were designed in the past and the caveats that produces now in these IT-cloudy times is not my fault ;-)
If Jetty is just passing the KeyStore to the JVM, you're right that the JVM's issue tracker is the right place. Thanks. But it might be considered to log this particular error with level error and not debug because it's something that leads to the inoperation of Jetty, so the reason for this should show up in a log with a level-setting that is common on production systems.
sbordet
commented
2 years ago But it might be considered to log this particular error with level error and not debug because it's something that leads to the inoperation of Jetty, so the reason for this should show up in a log with a level-setting that is common on production systems.
No, because your error is indistinguishable from a client error such as the client abruptly closing the connection. These client-side errors are so common that we cannot even log them at INFO level, never mind at ERROR level.
Your use case is very particular.
You can try to add a SslHandshakeListener, see for example here.
In that listener you can try to inspect the exception you get, and try to figure out if it's due to your HSM problems.
If it is, you can try to reload the SslContextFactory, see https://github.com/eclipse/jetty.project/blob/jetty-10.0.9/jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SslContextFactoryReloadTest.java, so you don't restart the whole server.
kimmerin
commented
2 years ago I'll check the test class but
No, because your error is indistinguishable from a client error such as the client abruptly closing the connection.
A client closing a connection is leading to a ProviderException?
sbordet
commented
2 years ago A client closing a connection is leading to a ProviderException?
A client can cause a large number of different exceptions, and we rarely differentiate -- it's a bad client and we don't bother.
For your unique case, see what I suggest above, or configure your HSM to not timeout, or poll periodically your HSM so it does not timeout.
Really, this is not a Jetty issue.
github-actions[bot]
commented
1 year ago This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.
github-actions[bot]
commented
2 months ago This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.
Jetty version(s) 10.0.9
Java version/vendor
(use: java -version)openjdk version "11.0.14" 2022-01-18 LTS OpenJDK Runtime Environment Zulu11.54+23-CA (build 11.0.14+9-LTS) OpenJDK 64-Bit Server VM Zulu11.54+23-CA (build 11.0.14+9-LTS, mixed mode)OS type/version Microsoft Windows [Version 10.0.19044.1706]
Description I've set up a TLS-connector using a SslContextFactory where I set an explicit KeyStore that "comes from" a PKCS#11-provider of an HSM:
The HSM in question is a "cloud-based HSM", i.e. you have network connections in order to access the HSM. This all works but if the connector stays idle for some time a new TLS-handshake will fail and the browser (here, Firefox) will report a "PR_END_OF_FILE_ERROR". The server log doesn't show anything unless you enable debug-level. Then you'll get an exception:
You have to restart Jetty in order to get this working again. I'm assuming that Jetty is retrieving the private key once and caches it. The long idle time leads to some timeout within the Provider or on the HSM side, so the session "behind" the key becomes invalid. Subsequent attempts to use the key then leads to the mentioned error.
To verify my theory I've created a test-class that - I think - confirms it:
In the version shown above, the whole thing breaks after 16 or 32 minutes:
When putting the key-retrieval within the for-loop, no error occurred and - because I haven't seen this effect in Jetty 9.4 - I assume that this was the way it was done in previous versions of Jetty (i.e. get the key every time it's needed).
"Activating" the commented out block would solve the problem as well but that doesn't help "here", because Jetty isn't able to do this call (
keystore.getProvider()exists but the password is unknown so you can't calllogin).How to reproduce?
Set up a Jetty instance with TLS connector using a network-based HSM (I had to deal with Utimaco but there might be others with similar effects). Go to your browser of choice and try to connect to the connector. Wait. Try again.