jeremyharisch
commented
6 months ago Detailed Investigation and Proposed Solutions
After thorough investigation, I've identified three potential solutions for achieving no-downtime certificate rotation, each with its own set of advantages and considerations. Let's delve into each solution:
1. Multiple Hosts:
- Description: This solution proposes the adoption of a multi-host configuration to manage certificate rotation seamlessly. Rather than relying on a single host for certificate management, we introduce redundancy and flexibility by configuring multiple hosts to distribute the load and facilitate uninterrupted certificate rotation.
- Implementation Steps:
- Initial Setup: During the initial setup phase, we configure our system with a primary host (e.g., listener.cp.kyma.cloud.sap) and ensure that it accepts the CA certificate as part of its configuration.
- Certificate Rotation: When it comes time to rotate the CA certificate, instead of updating the certificate on the existing host, we designate the next host in the rotation sequence to receive the new CA certificate. For instance, if we initially configured listener.cp.kyma.cloud.sap, the next host in line (e.g., one.listener.cp.kyma.cloud.sap) would receive the updated CA certificate during rotation.
- Runtime Watcher Deployment: An essential aspect of this solution is updating the runtime-watcher deployments to reflect the change in host addresses. The runtime-watcher components need to be aware of which host is currently serving certificates signed by the newly rotated CA certificate.
- Implementation Steps:
- Pros:
- Enhanced Certificate Management: By leveraging multiple hosts, we gain a comprehensive overview of active certificates, simplifying management and monitoring tasks.
- Improved Fault Tolerance: Redundancy introduced by multiple hosts enhances fault tolerance and resilience against host failures or maintenance events.
- Cons:
- Complexity: Implementing and managing multiple hosts introduces complexity into the system architecture and operational procedures.
- Resource Overhead: Maintaining redundant hosts may incur additional resource overhead in terms of infrastructure costs and management efforts.
- Configuration Synchronization: Ensuring consistent configuration across multiple hosts and runtime-watcher deployments requires careful coordination and synchronization mechanisms.
2. Intermediate CA-Certificate-Bundles:
- Description: This approach involves a two-phase migration of clients, allowing for a slow, gradual transition with minimal to zero downtime. In the table below, you can see the detailed steps of the procedure when the CA-Certificate gets rotated. 'rootA+rootB' signifies that the CA-certificates have been concatenated and set as the 'ca.crt' value in the certificate secret. When transitioning from 'rootA+rootB' to 'rootB', it entails truncating the CA-Certificate String and removing the first certificate from the concatenation.
- Pros:
- Allows for a gradual migration of clients.
- Provides flexibility with a long time window for parallel operation of old and new setups.
- Cons:
- Requires careful management and coordination during the migration process.
- Preferably introduce new component, like a side-car, to take care of managing such transition phases.
- A lot of string manipulation needs to be done Detailed description of how the procedure looks like
| Step | Step-Name | Gateway Server Cert | Gateway Accepts Clients (CACert on KCP) | Clients Accepts Server (CACert on SKR) | Client Cert | Note |
|---|---|---|---|---|---|---|
| 01 | Initial setup | rootA | rootA | rootA | rootA | "" |
| 02 | Generate rootB cert in KCP | rootA | rootA | rootA | rootA | "" |
| 03 | Reconfigure the Gateway in the KCP | rootA | rootA+rootB | rootA | rootA | All clients with the old Certificates signed by rootA still work |
| 04 | Migrate Clients to Certificates signed by rootB | rootA | rootA+rootB | rootA+rootB | rootB | "" |
| 05 | After alle Clients are migrated, switch Gate to accept only certs signed by rootB | rootB | rootB | rootB | rootB | "" |
3. Double-Gateway Setup:
- Description: In this solution, we introduce a second Gateway to our current setup. Both Gateways maintain identical configurations but reference distinct secrets. The Lifecycle-Manager is responsible for copying the CA certificate generated by Cert-Manager into a designated secret, referred to as 'Secret A' and utilized by Gateway A. Subsequently, when the certificate within the secret created by the Cert-Manager is rotated, the Lifecycle-Manager duplicates this updated certificate into 'Secret B', utilized by Gateway B. This process alternates between the two secrets with each rotation, ensuring seamless continuity. Consequently, our system accommodates both the old and new CA certificates, mitigating downtime effectively. I've validated this approach locally, confirming its functionality. Notably, this solution retains our existing leaf-certificate rotation logic while eliminating downtime, facilitating prompt adoption of rotated certificates.
- Pros:
- Simplifies implementation while ensuring no downtime.
- Allows for maintaining the existing code for certificate rotation.
- Cons:
- Introduces a new resource (second Gateway), adding complexity to the setup.
- Could maybe stop working in a future istio version, depends on how istio deals with have multiple Gateways specifying same hosts
Based on these considerations, I recommend either solution two or three. Solution three offers simplicity in implementation and ensures no downtime by leveraging two Gateways referencing different secrets. However, solution two provides a gradual migration process for clients, which might be advantageous depending on the specific requirements and constraints of the project.
Description:
As a followup to the this issue, we need to come up with a a no-downtime solution to have the leaf certificates rotated with the CA certificate rotation. The current implementation is here
ACs
Hint https://github.com/cert-manager/cert-manager/issues/2478