ServiceAccount as CredentialsSource

[angelbarrera92]

What problem are you facing?

I want to enable users to provide their own ProviderConfig using their namespaced ServiceAccount. Currently, the only way of doing that is by providing a kubeconfig inside a secret. Then use that secret in the ProviderConfig. This works but would be amazing to have another simpler mechanism to provide inner kubernetes credentials per team.

How could Crossplane help solve your problem?

Adding a new CredentialsSource option: ServiceAccount to the ProviderConfig

[turkenh]

The proposed solution sounds reasonable to me; however, I am not sure how this could be used in a multitenant environment. I mean, if users could create a ProviderConfig, what would prevent them from to refer any ServiceAccount in any namespace? If this is achieved by an admin somehow, how would you ensure that you users always uses the provider config they are supposed to use?

It would be great if you could give some more details on the use case?

[angelbarrera92]

Hi @turkenh!

That's a fair point; not much different from using secrets - kubeconfig right?

Anyway, imagine a multi tenant k8s cluster where users have access only to their own namespace and they have a dedicated serviceAccount with a few privileges.

It would be amazing to have a Namespaced ProviderConfigs; AFAIK, this resource is cluster scoped, Am I right?

So, maybe, wrapping a ProviderConfig as an XRD + Composite could be a good idea to enable these users to request their own ProviderConfig.

Goal: Provide a self-onboarding into the crossplane kubernetes provider. There (in the composite) you can implement the logic to enforce the provided serviceaccount to come from the same namespace as the claim.

Something like:

• Kubernetes Provider + controllerConfig (setting a serviceAccount with a few permissions) + root providerConcifg.
• XRD ProviderConfigRequest + Composite. This Composite must:
  • Use the root providerConfig to create the providerConfig defined in the ProviderConfigRequest composite.
  • Hardcode the serviceAccount namespace in the providerConfig with the same value as the metadata.namespace given at the ProviderConfigRequest composite.

I've don't test this solution, but i would test it in the upcoming days (hopefully).

Am I the only one asking for namespaced providerConfigs?

[angelbarrera92]

I've create a Proof of concept here: https://github.com/angelbarrera92/kubernetes-provider-config-request

[turkenh]

Thanks @angelbarrera92 for the POC. Now, I can better see how this feature could be useful for users regarding self-servicing a provider Kubernetes ProviderConfig in a multi-tenant cluster.

What is still missing for me, though, is the next steps. So, a user created a new ProviderConfig using the ProviderConfigRequest you POCed, but what will force them to create Provider Kubernetes Objects using that specific ProviderConfig resource rather than using the default (or cluster-admin) one or one from some other namespace.

[angelbarrera92]

Thanks @angelbarrera92 for the POC. Now, I can better see how this feature could be useful for users regarding self-servicing a provider Kubernetes ProviderConfig in a multi-tenant cluster.

What is still missing for me, though, is the next steps. So, a user created a new ProviderConfig using the ProviderConfigRequest you POCed, but what will force them to create Provider Kubernetes Objects using that specific ProviderConfig resource rather than using the default (or cluster-admin) one or one from some other namespace.

Nice!!!

So this is something missing and I'm not sure what could be the best option. I mean, it depends on the platform implementation:

• Platforms already running OPA, will implement a validating and/or validating webhook in Rego.
• Kyverno can be used to do the same.
• inhouse developments (using python, bash, go, or whatever).

IMHO: On an event where a providerConfigRef is used (by creating an Object.kubernetes.crossplane.io for example), the validation engine (whatever implementation) should inspect the target namespace looking for providingConfigRequests, then reject requests that don't use an already existing providerConfigRequests.

This is probably not the best example (as Object is a cluster-scope resource, just imagine it is namespace-scoped).

Imagine I'm creating a Composite/XRD defining a "DB" with namespace-scoped claims. These claims must have the providerConfigRef so they can be validated.

In the end, it's all about how many permissions your users have. In a namespace-scoped environment, these must be designed very carefully. namespaces are far from being a perfect multi-tenant solution, but this is how Kubernetes was designed, unfortunately :(. I wish to have the OpenStack multi-tenancy model in Kubernetes. But that's not going to happen.

[turkenh]

I see, thanks! I am fine with the feature itself; adding ServiceAccount as another credential source sounds reasonable.

[angelbarrera92]

I see, thanks! I am fine with the feature itself; adding ServiceAccount as another credential source sounds reasonable.

I can work on it, but ill need to solve a few doubts.

the main one is...

• Where to implement it? i mean, as far as i discovered, this things must be implemented in the crossplane-runtime go package. But... serviceAccouts are so specific to Kubernetes.

WDYT?

[turkenh]

Yeah, good point. I believe this would also be helpful for provider-helm, and this is not the first time we need the same functionality for both. Ideally, it should live in crossplane-runtime with other auth methods; however, I would prefer to have it (migrating to runtime) as a separate work.

So, I am OK with adding it here for now (and duplicating it to provider-helm if needed).

[ravilr]

@turkenh @jbw976

Supporting ServiceAccountRef as a CredentialSource in providerConfigs is also useful in single-cluster multi-tenancy setups, where per tenant namespace is used as isolation mechanisms(along with tenants only allowed to consume XRC resources, not the cluster scoped provider MR resources directly) as described here: https://docs.crossplane.io/knowledge-base/guides/multi-tenant/#namespaces-as-an-isolation-mechanism

For example, When using provider-kubernetes in Compositions to provision Kubernetes resources as MR base resources, the most commonly used pattern is to patch the providerConfigRef from the XR claim namespace derived Providerconfig using a string transform %s-kpc, where the $claimNamespace-kpc kubernetes.crossplane.io/ProviderConfig resource is pre-provisioned by the crossplane/cluster operator during tenant onboarding.

With ServiceAccount as a CredentialSource, this allow the cluster operator to refer to a tenant-namespace RBAC scoped ServiceAccount in the per tenant ProviderConfig, without requiring to provision a kubeconfig secret. By using K8s TokenRequest API, the serviceAccount based CredentialSource fetcher would be able to fetch time-bound SA token and use that as a Bearer token for K8s API connections. IMO, the implementation would need to employ some kind of caching to avoid hitting TokenRequest API on every reconcile loop ?

the linked PR seems to be abandoned ? wanted to revive the discussion here on how can this feature make it into provider-kubernetes..