search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.35k stars 1.99k forks source link

[BUG] Tomcat10 in Windows as a Service Problems #28560

Closed ProphetManX closed 2 years ago

ProphetManX commented 2 years ago

Describe the bug I was following the guidance provided by: https://docs.microsoft.com/en-us/azure/developer/java/fundamentals/java-azure-keyvault-tomcat-integration I am unable to get Tomcat to start up and find the AzureKeyVault provider specified. I reached out to Manfred Riem and he helped verify that all my settings and configurations were matching the documentation, and was unable to identify any reasons why it wouldn't be working. He directed me to submit this bug ticket.

My setup is using a Windows VM hosted in Azure to host a Tomcat10 instance. I was able to configure the connection to support HTTP/80 traffic, but when attempting to use HTTPS/443 and keyvault as the SSL cert store, tomcat is unable to find the provider specified. I am configuring Tomcat with the Tomcat10w service tool, adding the security file and keyvault uri values to the Java Options section, and I have placed a copy of the custom created bootstrap.jar and jca.jar in the lib folder as well.

Exception or Stack Trace

27-Apr-2022 16:41:11.815 SEVERE [main] org.apache.tomcat.util.net.SSLUtilBase.getStore Failed to load keystore type [DKS] with path [] due to [no such provider: AzureKeyVault]
    java.security.NoSuchProviderException: no such provider: AzureKeyVault
        at java.base/sun.security.jca.GetInstance.getService(GetInstance.java:83)
        at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:206)
        at java.base/java.security.Security.getImpl(Security.java:703)
        at java.base/java.security.KeyStore.getInstance(KeyStore.java:926)
        at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:188)
        at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:207)
        at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:283)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:247)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:105)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1192)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1205)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:580)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1042)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:747)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:769)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
27-Apr-2022 16:41:11.815 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[org.apache.coyote.http11.Http11NioProtocol-443]]
    org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1055)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1042)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:747)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:769)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
    Caused by: java.lang.IllegalArgumentException: Failed to load keystore type [DKS] with path [] due to [no such provider: AzureKeyVault]
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:107)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1192)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1205)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:580)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)
        ... 13 more
    Caused by: java.io.IOException: Failed to load keystore type [DKS] with path [] due to [no such provider: AzureKeyVault]
        at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:229)
        at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:207)
        at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:283)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:247)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:105)
        ... 20 more

To Reproduce Create a Windows Server VM, install Java and Tomcat10, and then follow your documentation guidelines

Setup (please complete the following information):

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:

Additional context Add any other context about the problem here.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

ProphetManX commented 2 years ago

I apologize, but I don't think I mentioned this in my original post. The KeyVault I'm trying to hit is hosted within Azure Gov, not Commercial.

saragluna commented 2 years ago

Hi @ProphetManX, did you follow the instructions listed in this article? To be specific, adding the KeyVaultJcaProvider into your java.security file, like 

security.provider.1=com.azure.security.keyvault.jca.KeyVaultJcaProvider
security.provider.2=SUN
... more providers in the file

or using a separate file my.java.security and overriding it by specifying -Djava.security.properties==/somepath/my.java.security to override the original file (see this article for more details)?

ProphetManX commented 2 years ago

The instructions weren't super clear and consistent across both the JVM and Tomcat articles. I've changed my settings in the Tomcat service tool to the following: image You can see I have the double equals signs now, and I'm now getting the following error in Tomcat.

This time I included the startup variables as well in the stack trace.

02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/10.0.20
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Mar 31 2022 14:24:36 UTC
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.20.0
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Windows Server 2019
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            10.0
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             C:\Program Files\Java
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.12+7
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Eclipse Foundation
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         F:\Tomcat
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         F:\Tomcat
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=F:\Tomcat
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=F:\Tomcat
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=F:\Tomcat\temp
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=F:\Tomcat\conf\logging.properties
02-May-2022 09:52:17.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.properties==my.java.security 
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dazure.keyvault.uri=https://obo-certstore-dev-kv.vault.usgovcloudapi.net/
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024m
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx2048m
02-May-2022 09:52:17.467 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [F:\Tomcat\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files\Java\\bin;C:\Program Files\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\WindowsApps;.]
02-May-2022 09:52:18.024 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-80"]
02-May-2022 09:52:18.071 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-443"]
02-May-2022 09:52:18.102 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[org.apache.coyote.http11.Http11NioProtocol-443]]
    org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1055)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1042)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:747)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:769)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
    Caused by: java.lang.IllegalArgumentException: java.security.NoSuchAlgorithmException: TLS SSLContext not available
        at org.apache.tomcat.util.net.jsse.JSSEUtil.initialise(JSSEUtil.java:110)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.getImplementedProtocols(JSSEUtil.java:73)
        at org.apache.tomcat.util.net.SSLUtilBase.<init>(SSLUtilBase.java:92)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.<init>(JSSEUtil.java:61)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.<init>(JSSEUtil.java:56)
        at org.apache.tomcat.util.net.jsse.JSSEImplementation.getSSLUtil(JSSEImplementation.java:59)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:96)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1192)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1205)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:580)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)
        ... 13 more
    Caused by: java.security.NoSuchAlgorithmException: TLS SSLContext not available
        at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
        at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
        at org.apache.tomcat.util.net.jsse.JSSESSLContext.<init>(JSSESSLContext.java:45)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.initialise(JSSEUtil.java:105)
        ... 26 more
ProphetManX commented 2 years ago

Ok, I've been digging into this all morning and have some new details to share.

My colleague and I have gotten a normal java KeyStore setup locally to verify that we can get SSL working without AzureKeyVault. Then I was systematically "enabling" the changes from the instructions provided to see where/when things start breaking. I have setup the site to use HTTP, HTTPS, and HTTPS:8443 for testing different things. When I do not override the setting -Djava.security.properties==my.java.security, all sites are working correctly. As soon as I add the security override, I begin getting the following error:

02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/10.0.20
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Mar 31 2022 14:24:36 UTC
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.20.0
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Windows Server 2019
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            10.0
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             C:\Program Files\Java
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.12+7
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Eclipse Foundation
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         F:\Tomcat
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         F:\Tomcat
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.properties==my.java.security
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=F:\Tomcat
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=F:\Tomcat
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=F:\Tomcat\temp
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=F:\Tomcat\conf\logging.properties
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dazure.keyvault.uri=https://obo-certstore-dev-kv.vault.usgovcloudapi.net/
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024m
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx2048m
02-May-2022 11:42:07.336 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [F:\Tomcat\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files\Java\bin;C:\Program Files\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\WindowsApps;.]
02-May-2022 11:42:07.914 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-80"]
02-May-2022 11:42:07.961 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-8443"]
02-May-2022 11:42:07.992 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8443]]
    org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1055)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1042)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:747)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:769)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
    Caused by: java.lang.IllegalArgumentException: java.security.NoSuchAlgorithmException: TLS SSLContext not available
        at org.apache.tomcat.util.net.jsse.JSSEUtil.initialise(JSSEUtil.java:110)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.getImplementedProtocols(JSSEUtil.java:73)
        at org.apache.tomcat.util.net.SSLUtilBase.<init>(SSLUtilBase.java:92)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.<init>(JSSEUtil.java:61)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.<init>(JSSEUtil.java:56)
        at org.apache.tomcat.util.net.jsse.JSSEImplementation.getSSLUtil(JSSEImplementation.java:59)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:96)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1192)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1205)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:580)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)
        ... 13 more
    Caused by: java.security.NoSuchAlgorithmException: TLS SSLContext not available
        at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
        at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
        at org.apache.tomcat.util.net.jsse.JSSESSLContext.<init>(JSSESSLContext.java:45)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.initialise(JSSEUtil.java:105)
        ... 26 more

Again, this setup isn't even trying to use the Tomcat Connector that is pointing to AzureKeyVault yet. This leads me to believe I'm missing something within the security override changes I've made. As instructed I simply copied my current java.security file, renamed the copy to my.java.security and added the new entry for value 1, and incremented the other values originally there. Knowing these changes were pretty minor, I began thinking maybe the override wasn't "finding" the file I was specifying. I changed the override to -Djava.security.properties==C:\Program Files\Java\conf\security\my.java.security and then all the sites started working again. (again, no AzureKeyVault Connectors at this time)

Now that I know I have the security properties sorted out, it was time to re-enter the Azure Connector. Now when I start the service, I'm back to getting the stack trace from my initial post:

02-May-2022 12:04:06.582 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [truststorePass] to [tomcat]
02-May-2022 12:04:06.598 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [truststoreFile] to [conf/obolinkkeystore.jks]
02-May-2022 12:04:06.598 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [keystorePass] to [tomcat]
02-May-2022 12:04:06.598 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [keystoreFile] to [conf/obolinkkeystore.jks]
02-May-2022 12:04:06.598 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [keystoreType] to [PKCS12]
02-May-2022 12:04:06.598 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [sslProtocol] to [TLS]
02-May-2022 12:04:06.598 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [clientAuth] to [false]
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/10.0.20
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Mar 31 2022 14:24:36 UTC
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.20.0
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Windows Server 2019
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            10.0
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             C:\Program Files\Java
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.12+7
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Eclipse Foundation
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         F:\Tomcat
02-May-2022 12:04:06.613 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         F:\Tomcat
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=F:\Tomcat
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=F:\Tomcat
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=F:\Tomcat\temp
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=F:\Tomcat\conf\logging.properties
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.properties==C:\Program Files\Java\conf\security\my.java.security
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dazure.keyvault.uri=https://obo-certstore-dev-kv.vault.usgovcloudapi.net/
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024m
02-May-2022 12:04:06.629 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx2048m
02-May-2022 12:04:06.644 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [F:\Tomcat\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files\Java\bin;C:\Program Files\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\WindowsApps;.]
02-May-2022 12:04:07.222 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-80"]
02-May-2022 12:04:07.269 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-8443"]
02-May-2022 12:04:07.659 SEVERE [main] org.apache.tomcat.util.net.SSLUtilBase.getStore Failed to load keystore type [DKS] with path [] due to [no such provider: AzureKeyVault]
    java.security.NoSuchProviderException: no such provider: AzureKeyVault
        at java.base/sun.security.jca.GetInstance.getService(GetInstance.java:83)
        at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:206)
        at java.base/java.security.Security.getImpl(Security.java:703)
        at java.base/java.security.KeyStore.getInstance(KeyStore.java:926)
        at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:188)
        at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:207)
        at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:283)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:247)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:105)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1192)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1205)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:580)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1042)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:747)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:769)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
02-May-2022 12:04:07.659 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[org.apache.coyote.http11.Http11NioProtocol-8443]]
    org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1055)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1042)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:747)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:769)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
    Caused by: java.lang.IllegalArgumentException: Failed to load keystore type [DKS] with path [] due to [no such provider: AzureKeyVault]
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:107)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1192)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1205)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:580)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)
        ... 13 more
    Caused by: java.io.IOException: Failed to load keystore type [DKS] with path [] due to [no such provider: AzureKeyVault]
        at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:229)
        at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:207)
        at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:283)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:247)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:105)
        ... 20 more
02-May-2022 12:04:07.675 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-443"]
02-May-2022 12:04:08.034 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1769] milliseconds
02-May-2022 12:04:08.112 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
02-May-2022 12:04:08.112 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.0.20]
02-May-2022 12:04:08.128 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [F:\Tomcat\conf\Catalina\localhost\ot-authws.xml]

(added more details for helping triage things)

I had worked with Manfred Riem earlier, and we verified the bootstrap.jar and jca.jar files were in the correct location, and I the contents of the bootstrap.jar were matching the instructions provided. I tried changing the value from my.java.security from security.provider.1=AzureKeyVault to security.provider.1=com.azure.security.keyvault.jca.KeyVaultJcaProvider since the latter is what was to be used for Java8, but also matches the text used in the bootstrap.jar's java.security.provider file I created. This also did not work.

As it stands, I think the my.java.security override is ok since I now have a local keystore hosting the SSL and that is working, and HTTP is also working, but the 8443 Connector for AzureKeyVault is not working. If there is anything else I can try or provide additional details, please don't hesitate to reach out. And if you have a working example I can refer to and compare against, I would be happy to dig into that as well.

Cheers, Gordon

saragluna commented 2 years ago

Hi @ProphetManX, I am guessing the error Failed to load keystore type [DKS] with path [] due to [no such provider: AzureKeyVault] is because the jca doesn't on the classpath, because I've tried running the tomcat server in a Windows Datacenter 2019 according to following steps:

ProphetManX commented 2 years ago

Update!

So working with a colleague, we were able to tinker more and get this working via command line, and then even using the Tomcat Service launcher application.

The documentation provided isn't accurate, and should be updated, even for the command line setup, however I think it might be useful for others to provide the Tomcat Service launcher details as well for anyone else using Tomcat on Windows.

Updated Instructions

  1. you do NOT need to create that bootstrap.jar file as guided in the JVM article.
  2. you also do NOT need to put the azure-security-keyvault-jca-2.6.0.jar file into Tomcat's lib subfolder, as you WILL need to reference it via CLASSPATH
  3. when creating your own copy of the java.security file, you don't have to explicitly add the new "azure" provider at the top of the list. I added it as number 14 and it worked just fine.
  4. I am using Java v11 and the instructions provided in the JVM article specify to use AzureKeyVault as the provider name, however that did not work for us. We had to use the com.azure.security.keyvault.jca.KeyVaultJcaProvider value directed for Java v8 instead, and that ended up being the last peice to fall into place.
  5. you're setevn.bat file was great in troubleshooting and getting it working, however we did modify syntax and add a little bit to it. I know for our troubleshooting purposes it was exactly what we needed to see
    • Updated JAVA_OPTS to include the -Djava.keyvault.uri=https://customer-site.vault.usgovcloudapi.net/
    • Updated CLASSPATH so that the right side of the equals sign didn't have quotes wrapping the string. When I started catalina, it was dumping variables to the command line, and I could see the quotes were causing problems. 
      set "JAVA_OPTS=%JAVA_OPTS% -Djava.security.properties==C:\XXX\my.java.security -Dazure.keyvault.uri=https://customer-site.vault.usgovcloudapi.net/"
set CLASSPATH=C:\XXX\azure-security-keyvault-jca-2.6.0.jar

Using Tomcat Service Launcher

After we got it working in the command line, I was able to enter the proper settings in the Tomcat launcher window and get it working too. image

  1. You can see in the picture, I added the azure-security-keyvault-jca-2.6.0.jar (with full pathname) to the Java Classpath: field, adding it to the beginning of the line.
  2. Then I had to add the two options I needed into the Java Options field. These I added at the end.
    • -Djava.security.properties==C:\XXX\my.java.security
    • -Dazure.keyvault.uri=https://customer-site.vault.usgovcloudapi.net/

Overall setup for a Windows server using Tomcat is really quick and easy.

  1. Download the azure-security-keyvault-jca-2.6.0.jar and put it in a folder somewhere.
  2. Make a copy of the %JRE_HOME%\conf\security\java.security file and rename it to something like keyvault.java.security, then put it with the jca jar from Step 1
  3. Update the keyvault.java.security file, adding the line security.provider.14=com.azure.security.keyvault.jca.KeyVaultJcaProvider (you can modify the order number as you please)
  4. Run the Tomcat9w.exe launcher and update the following:
    • Java Classpath: prepend with the full path filename of the jca jar file you downloaded and terminate with a semicolon C:\XXX\azure-security-keyvault-jca-2.6.0.jar;
    • Java Options add -Djava.security.properties==C:\XXX\keyvault.java.security (pointing to the full path filename of your new security file) and then whatever additional keyvault specific options you need to access said keyvault. (ex, Managed Identity, only need to add -Dazure.keyvault.uri=https://KEY-VAULT-NAME.vault.usgovcloudapi.net/ <- this is an Azure Gov key vault url, will look different for Azure Commercial)
ProphetManX commented 2 years ago

Last Problem

Now that we have our Tomcat instance connecting and retrieving the SSL certificate from Azure KeyVault, the last problem we're running into is that our certificate doesn't include it's Cert Chain. When we load the site, we get a browser error image and when inspecting the certificate, the Certificate Path doesn't list either of the two Certificate Authorities (CA -> Intermediate CA -> My Cert). When I setup a dummy site in IIS and test My Cert, it does show all three certs in the path tab.

After speaking with Manfred, he found this Feature Request https://github.com/Azure/azure-sdk-for-java/issues/20591

I'm sure your sprints are full already, but I would like to voice my support for this feature to be added asap so that my customer can use Azure KeyVaults directly in their Tomcat hosted systems. Thanks for all the help on this, I really appreciate it.

Cheers!

saragluna commented 2 years ago

PR to fix the doc created https://github.com/MicrosoftDocs/azure-dev-docs-pr/pull/3024.

saragluna commented 2 years ago

Hi @ProphetManX, are we okay to close this issue now?

ProphetManX commented 2 years ago

Oh yes, I didn't realize you wanted me to close it. thanks again for all the support.