Support multiple certificates in truststore (or keystore?)

[sbernauer]

Since https://github.com/stackabletech/secret-operator/pull/350 secret-op can add multiple CAs to the created PEM and pkcs12 truststores. Some products (in some versions) have problems with this, as we noticed in HBase 2.4.12 (see error message below)

We need to check every product to see if it supports multiple certificates:

• HBase is fixed since 2.4.13, 2.5.0 and 3.0.0 issue commit
• Zookeeper is fixed since 3.9.0 code commit
• Hadoop: To be checked
• TODO: Check other products

### Acceptance - Lower CA lifetime and test multiple certs:
- [ ] https://github.com/stackabletech/issues/issues/522
- [ ] Airflow
- [ ] Druid
- [x] HBase (fixed in stackabletech/docker-images#540 by backporting the patch for [HBASE-27027](https://issues.apache.org/jira/browse/HBASE-27027) to HBase 2.4.12)
- [ ] Hadoop HDFS
- [ ] Hive
- [ ] Kafka
- [ ] NiFi
- [ ] Spark
- [ ] Superset
- [ ] Trino
- [ ] ZooKeeper

Failed construction RegionServer
java.io.IOException: Problem starting http server
    at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:1140)
    at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:151)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:2235)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:694)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:3203)
    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:63)
    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:152)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3221)
Caused by: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1288)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1270)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:372)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:243)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.server.Server.doStart(Server.java:401)
    at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
    at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:1109)
    ... 13 more

[siegfriedweber]

HBase operator tests

Test case kerberos_hbase-2.4.12_hdfs-3.3.6_zookeeper-latest-3.8.3_listener-class-cluster-internal_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit

hbase-operator commit 543f7395955763e4bc6c6c77be2f98bd3ab32611, 2024-02-01 docker-images commit c75007e0a63ea0bb5277b88e8cb6d4648028f8f2, 2024-02-05

test-files.tar.gz

[lfrancke]

We found out that CA rotation was not the culprit after all. It turns out that a recent PR to add Kerberos support caused the issue as it enabled TLS for the Web UIs at the same time. Unfortunately, the tests were only running against 2.4.17 and not 2.4.12 otherwise it would have failed already at that time.

This ticket can stay open but it is less severe now.