Credentials System

[DinaBelova]

Goals

In order for CAPI to work it needs credentials to talk to the infra providers, which have quite the high amount of permissions. We need to ensure that such credentials can only be seen by a handful of roles in a cluster, enterprises use zero trust models and therefore even internal platform engineers should not see any credentials unless they really have to. Therefore this Epic Goal is to ensure that credentials added by Platform Leads can only be seen by platform laeds, while still ensuring that for specific clusters a Platform Engineer could provide their own credentials.

Major deliverables

• Platform Leads installing 2A are required to add provider credentials to the 2A management cluster which cannot be seen or accessed in any way by a Platform Engineer creating a Cluster from a 2A Template
• Platform Engineer can add credentials to their own namespace and not using credentials provided by them
• Audit and Reporting
  • Audit reporting for CRUD Operations on credentials can be accessed

Who it benefits

• Customer Business: Security out of the box, ensuring that only roles have access to credentials that are actually needed, limiting possible security risks
• Platform: Possibility to implement trust less models for cluster creation, plus a central place to manage credentials

Acceptance criteria

• Credentials can be created and managed by Platform Leads
• Credentials can be assigned to namespaces the same way as we assign Templates to namespaces
• On creation of a ManagedCluster the credentials to be used can be selected
• There is a way to list all available credentials in a namespace
• As a Platform Engineer there is no way for me to see the credential content
• As a Platform Engineer I can create a credential that I can use for my own ManagedClusters
• As a Platform Engineer or Platform Lead I can update credentials that I have access to
• Credentials should have an expiration field and a status that alerts Platform Engineers when and if the credential is expired (assuming we can get this info through CAPI and not directly talking to the Infra APIs, if this capability does not exist we would check how we can contribute this to the CAPI providers)
• Creation of a ManagedCluster without the correct credentials is prevented and alerted before the ManagedCluster is created
• Audit and Reporting
  • Reporting on all Credential CRUD operations is easily accessible and traceable to a user.

Assumptions

• CAPI and the infra providers do not leak the credentials into the logs
• CAPI provides a solution that does not require us to inject credentials into the CAPI objects inside a namespace which would leak the credentials
• Audit and Alerting capability can be completely implemented in the Observability Initative
  • Alerts can be sent on Credential CRUD operations
  • Alerts can be sent warning of the imminent expiry of a credential
• Discussed in engineering call:
  • Having seperate credential CRDs per provider is ok
  • Using the existing CAPI ClusterIdentity object structure for the new CRDs is ok
  • Mutation of a CredentialCRDs to automatically convert sensitive data into a secret is fine (assumption is that someone that uses GitOps creates the secret from the very beginning)
  • If necessary the HMC controller can pass secrets into the child cluster in order for CCM/CSI to work
  • Specifically for Azure:
    • Inject CCM credentials through a secret and not via file injection (will help with rotation of secrets)

Out of scope

• No need for 2A to talk directly to the APIs of the Infra Providers and usage of Provider CLIs like clusterawsadm to create credentials is fine. We need to document though how to create the credentials

User stories

• As a platform lead I want to allow my Platform Engineers to create clusters with the infra provider credentials without them ever being able to see them.
• As a platform lead I want to define which namespaces can use which credentials
• As a platform engineer I want to list all credentials that I can use without actually seeing the credential content
• As a platform engineer I want to add credentials in my own namespaces myself which I then can use for my clusters

Tasks

• 321

• 322

• 324

• 325

• 326

• 327

• 328

• 392

• 445

[a13x5]

Step-by-step process

The following describes the process of using the credentials:

1. User creates common cluster identity objects with corresponding secrets
2. Then user must create the Credential object and reference the cluster identity from the step 1
3. Controller validates that the referenced cluster identity exists and sets status based on the check.
4. User then creates ManagedCluster where it's referencing the Credential object in the same namespace.
5. Controller validates that the credential has Ready status. If it's not, the .status.credentialState is set and provisioning stops (no HelmRelease objects created).
6. On correct credentials controller merging user provided values from .spec.config with values of the cluster identity referenced. These values must be uniform across all templates and well-known. If the user specified the same values they have lesser precedence and thus will be overwritten.
7. Upon Cluster is ready Controller applies necessary Secrets for CCM/CSI on the managed cluster using kube API calls. Data required is different for each provider and it could be required to query several provider-specific objects to collect it. CCM/CSI controllers on the managed cluster must be configured accordingly to accept these Secrets