Support XSite Certificate Reloading

[ryanemerson]

ISPN-15916 added Keystore reloading capabilities to the Infinispan server, however updates to the cert secrets don't seem to have an impact on the server pods. This needs to be investigated and fixed either by changes to the server or operator code.

[ahus1]

I see that that feature was added only in 15.1.0.Dev01, and that might be more recent than the 15.0.x branch we're using in our testing in the Keycloak Benchmark repository.

[ryanemerson]

You're right @ahus1, ISPN-15916 was only added in 15.1.0.Dev01. However, more work is still required in order for us to support the automatic reloading of certificates in XSite deployments.

Infinispan 14/15.0.x behaviour

• Updates to Keystore's provided by spec.security.endpointEncryption.CertSecretName, result in a StatefulSet rolling update
• Updates to Truststores's provided by spec.security.endpointEncryption.ClientCertSecretName, result in a StatefulSet rolling update
• Updates to the XSite Keystores and Trustsores provided via spec.Service.Sites.Local.Encryption.* have no effect and both the Infinispan and GossipRouter pods must be restarted.

Infinispan 15.1.0.Dev01 behaviour

• Updates to Keystore's provided by spec.security.endpointEncryption.CertSecretName, result in a StatefulSet rolling update
• Updates to Truststores's provided by spec.security.endpointEncryption.ClientCertSecretName, result in a StatefulSet rolling update
• Updates to the XSite Keystores and Trustsores provided via spec.Service.Sites.Local.Encryption.TransportKeyStore and spec.Service.Sites.Local.Encryption.TransportTrustStore are automatically loaded at runtime on Secret update.
• Updates to the spec.Service.Sites.Local.Encryption.RouterKeyStore Secret are ignored until the GossipRouter pods are restarted.

Proposal

1. Prevent StatefulSet rolling update on Keystore/Truststore updates for Infinispan 15.1.0 onwards as these changes can be dynamically loaded without pod restarts #2125 .
2. Provide support for automatic cert reloading in the JGroups GossipRouter JGRP-2813