search

confidential-containers / trustee

Attestation and Secret Delivery Components
Apache License 2.0
56 stars 81 forks source link

Documenting KBS setup for different deployment scenarios #290

Open bpradipt opened 7 months ago

bpradipt commented 7 months ago

KBS supports different attestation model like passport and background check. Further, one can use the integrated KMS (filestore) to store the secrets, or add support for additional KMSes in the KBS. Add to that there is the aspect of providing reference measurement values.

KBS being a critical component of a CoCo setup it'll be helpful to provide guidance on KBS deployment and setup for admins. Further, with discussions about expanding the KBS usage beyond CoCo to standalone confidential VM and confidential clusters, having instructions for KBS deployment and setup becomes very important.

This issue is to discuss and conclude on a set of initial deployment scenarios and identify work needed to add the details in the documentation.

Putting a few initial scenarios.

  1. KBS deployed on the same Kubernetes cluster running CoCo workloads. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.
  2. KBS deployed external to the Kubernetes cluster running CoCo workloads. This could be another Kubernetes cluster or a standalone VM. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.
  3. Highly Available KBS deployment in a Kubernetes cluster. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.
  4. Highly Available KBS deployment in a standalone VM. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.

cc @fitzthum @Xynnn007 @mkulke @jensfr

mkulke commented 7 months ago

Possible unrelated, but afaik we haven't yet finalized a way to seed a kbs http cert to the TEE as part of a measured configuration. so which role does https play in the above scenario?

CDH is used to retrieve secrets from external KMS.

Doesn't CDH always need to retrieve secrets from a (resource) KBS, in both passport and bgcheck scenarios? Otherwise the KMS would need to be enlightened to understand AS/KBS tokens.

With integrated KMS (filestore) for storing secrets.

I wonder whether we can abstract from this. Afaik for Hashicorp Vault and Azure KV there are CSI drivers that would mount secrets transparently to a pod, so it would look like local fs.

Highly Available KBS deployment

I would just assume that any deployment that we prescribe should be HA in principle.

bpradipt commented 7 months ago

Possible unrelated, but afaik we haven't yet finalized a way to seed a kbs http cert to the TEE as part of a measured configuration. so which role does https play in the above scenario?

May not be totally unrelated.

With integrated KMS (filestore) for storing secrets.

I wonder whether we can abstract from this. Afaik for Hashicorp Vault and Azure KV there are CSI drivers that would mount secrets transparently to a pod, so it would look like local fs.

Did I understand correctly that you are suggesting mounting the HashiCorp Vault secrets inside the KBS pod ?

mkulke commented 7 months ago

Did I understand correctly that you are suggesting mounting the HashiCorp Vault secrets inside the KBS pod ?

yes

fitzthum commented 7 months ago

Doesn't CDH always need to retrieve secrets from a (resource) KBS, in both passport and bgcheck scenarios? Otherwise the KMS would need to be enlightened to understand AS/KBS tokens.

There are two ways to connect a KMS to a workload. First, you could implement a new Repository in the KBS (see trait definition). This would mean that when a guest requests a resource from the KBS the KBS would transparently fulfill that resource with a KMS.

Second, we support directly connecting to a KMS from inside the guest via the CDH (see here). Here the configuration of the KMS is scoped more to the workload. The secrets needed to connect to the KBS would be retrieved from the KBS and the KMS would not be enlightened for confidential computing.

fitzthum commented 7 months ago

Many permutations of the KBS/AS are possible. It's a bit of a challenge to outline them all, so I agree that we should pick a couple that we can document more clearly.

As of today maybe the simplest way to deploy is with docker compose. This sets up KBS, AS (with gRPC), RVPS, and coco-keyprovider as separate containers.

I'm trying to understand where we should draw the line for the scope of our support in this repo. Some of the permutations that @bpradipt mentions are defined by how the CDH is configured in the guest. Is that within the scope of this repo? Similarly does it matter if the KBS/AS runs in a VM or not?

One use case that we probably do need to think about more is how we support Kubernetes. We have an operator for the KBS, but I'm not sure exactly what deployment model(s) it supports. That might be something we should think about.

bpradipt commented 7 months ago

@fitzthum on the permutations using CDH configuration, I think you have a point. To start with, we should just stick to a CDH configuration using KBS and integrated KMS for secrets. IOW CDH using KBS resource provider. As for KBS/AS runs in a VM or not, I think it's better to qualify to distinguish it with KBS/AS running in a K8s cluster.

Further, as you mentioned about docker compose, I think we should call out that the deployment unit will be container image(s) which can be used for standalone deployment on a VM (using a compose file) or a K8s deployment using K8s yaml. Then we can add the KBS operator to the mix. Just to keep it simple to start with.

So the initial scenarios boils down to:

  1. KBS deployed on the same Kubernetes cluster running CoCo workloads.
  2. KBS deployed external to the Kubernetes cluster running CoCo workloads. This could be another Kubernetes cluster or a standalone VM.

High availability configurations for KBS whether deployed in a Kubernetes cluster or in a standalone VM can be taken up later.

Assumptions

  1. KBS uses background check mode
  2. KBS uses integrated KMS (filestore) for secrets
  3. KBS is deployed using container image (s).
  4. docker compose yaml will be provided for deployment in standalone VM
  5. K8s deployment yaml will be provided for deployment in a K8s cluster
  6. KBS uses HTTPS
  7. AA built using cc_kbc
  8. CDH uses KBS resource provider

Open Questions

  1. Are the assumptions correct?
  2. Should AS be built-in to KBS or be separate (using grpc) ?
fitzthum commented 7 months ago

Ok, I can update the docs to mention the two general use cases of deploying with docker compose and with Kubernetes (via the operator) using the release containers. I can also throw in the option of building things locally without a container because we already have a guide for that.

KBS deployed on the same Kubernetes cluster running CoCo workloads.

This might be a little fishy from a security standpoint but maybe in some cases it would make sense.

I think all those assumptions are generally correct.

For 4, we don't currently mandate where a user might run the docker cluster and we don't provide a VM image, but it is a very reasonable use case to deploy the cluster inside a VM.

For 6, we use HTTPS, but there are still some subtleties when it comes to how the guest verifies the connection (i.e. it doesn't, but it maybe doesn't matter). This might evolve some with the new configuration things we are working toward.

Should AS be built-in to KBS or be separate (using grpc) ?

In some use cases the AS and the KBS might not run in the same scope. Unless this is the case I would stick to the built-in AS, particularly because there are still some kinks to be worked out with the grpc interface between the two.

bpradipt commented 7 months ago

Ok, I can update the docs to mention the two general use cases of deploying with docker compose and with Kubernetes (via the operator) using the release containers. I can also throw in the option of building things locally without a container because we already have a guide for that.

For Kubernetes we can start with the K8s yaml - https://github.com/confidential-containers/kbs/tree/main/kbs/config/kubernetes and then add operator.

I will do some sample deployments and update as well.

I think all those assumptions are generally correct.

For 4, we don't currently mandate where a user might run the docker cluster and we don't provide a VM image, but it is a very reasonable use case to deploy the cluster inside a VM.

These will be initial set of tested and documented instructions. Downstream consumers might even create distro packages for installation inside a VM.

Should AS be built-in to KBS or be separate (using grpc) ?

In some use cases the AS and the KBS might not run in the same scope. Unless this is the case I would stick to the built-in AS, particularly because there are still some kinks to be worked out with the grpc interface between the two.

The less permutations, the better imho :-). So let's stick to builtin AS.

bpradipt commented 7 months ago

Initial deployment scenarios

  1. KBS deployed on the same Kubernetes cluster running CoCo workloads.
  2. KBS deployed external to the Kubernetes cluster running CoCo workloads. This could be another Kubernetes cluster or a standalone VM.

Assumptions

  1. KBS uses background check mode
  2. KBS uses integrated KMS (filestore) for secrets
  3. KBS is deployed using container image (s).
  4. docker compose yaml will be provided for deployment in standalone VM
  5. K8s deployment yaml will be provided for deployment in a K8s cluster
  6. KBS uses HTTPS
  7. KBS using builtin AS
  8. AA built using cc_kbc
  9. CDH uses KBS resource provider
bpradipt commented 7 months ago

@fitzthum another aspect we will need to decide is on the policy.

Should the initial scenarios use OPA policy? And if yes, any other policy required other than the default one?

surajssd commented 7 months ago

If I can boil down to the most basic forms, we need two modes of deployments:

PS: On public cloud with CVM support, I can think of a model where KBS is deployed using peer pods. Now how you get keys into that KBS is a different discussion.

fitzthum commented 7 months ago

Should the initial scenarios use OPA policy? And if yes, any other policy required other than the default one?

I assume this is referring to the attestation policies and resource policies used by KBS/AS (not the kata agent policy inside the guest). I'm not sure it's in scope for us to provide these policies. The resource policy in particular will depend on what resources a client wants to use. It would be tough for us to provide something for that. It's possible that we could provide some sane defaults for the attestation policy.

In any case we should make sure that we give the user an easy way to update the policies i.e. by shipping the client tool as part of the deployment.

bpradipt commented 7 months ago

I assume this is referring to the attestation policies and resource policies used by KBS/AS (not the kata agent policy inside the guest).

Yes

In any case we should make sure that we give the user an easy way to update the policies i.e. by shipping the client tool as part of the deployment.

+1

tylerfanelli commented 7 months ago
  1. KBS deployed on the same Kubernetes cluster running CoCo workloads. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.

Would this be considered an "ephemeral" KBS? In that, once all CoCo workloads in your cluster are attested, the KBS is no longer used?

  1. KBS deployed external to the Kubernetes cluster running CoCo workloads. This could be another Kubernetes cluster or a standalone VM. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.

Likewise, could we consider this as an ephemeral KBS?

  1. Highly Available KBS deployment in a Kubernetes cluster. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.

  2. Highly Available KBS deployment in a standalone VM. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.

Can you elaborate on what you mean by "Highly Available"?

bpradipt commented 7 months ago
  1. KBS deployed on the same Kubernetes cluster running CoCo workloads. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.

Would this be considered an "ephemeral" KBS? In that, once all CoCo workloads in your cluster are attested, the KBS is no longer used?

Yeah sort of. Note that for now, we decided to remove the CDH/KMS scenario to keep it simple.

  1. Highly Available KBS deployment in a standalone VM. a. With integrated KMS (filestore) for storing secrets. b. Without integrated KMS. CDH is used to retrieve secrets from external KMS.

Can you elaborate on what you mean by "Highly Available"?

Two KBS instances running in Active/Passive or Active/Active configuration

mkulke commented 7 months ago

Did I understand correctly that you are suggesting mounting the HashiCorp Vault secrets inside the KBS pod ?

yes

fwiw, I tried this w/ Azure Key Vault and it works. Obviously there's a fair bit of yaml involved, but this is mostly unavoidable for IAM reasons, would be similar for a native integration. Do we want to go down this path or should we implement native KMS plugins?

bpradipt commented 7 months ago

Did I understand correctly that you are suggesting mounting the HashiCorp Vault secrets inside the KBS pod ?

yes

fwiw, I tried this w/ Azure Key Vault and it works. Obviously there's a fair bit of yaml involved, but this is mostly unavoidable for IAM reasons, would be similar for a native integration. Do we want to go down this path or should we implement native KMS plugins?

Can you share the setup details? May be it's a good start before implementing native KMS plugins. Although I think native plugins will make it easier to run in standalone environment. Wdyt ?

mkulke commented 7 months ago

Did I understand correctly that you are suggesting mounting the HashiCorp Vault secrets inside the KBS pod ?

yes

fwiw, I tried this w/ Azure Key Vault and it works. Obviously there's a fair bit of yaml involved, but this is mostly unavoidable for IAM reasons, would be similar for a native integration. Do we want to go down this path or should we implement native KMS plugins?

Can you share the setup details? May be it's a good start before implementing native KMS plugins. Although I think native plugins will make it easier to run in standalone environment. Wdyt ?

I added instructions here, PTAL https://github.com/confidential-containers/confidentialcontainers.org/pull/32

fitzthum commented 6 months ago

Is there any outstanding work for this issue?

bpradipt commented 6 months ago

We have concluded on the scenarios and assumptions. There is a documentation on how to add external secrets to KBS/KMS. Also kbs-operator now supports both all-in-one deployments as well as microservices based.

However the documentation work w.r.to the scenarios are pending.

  1. KBS deployed on the same Kubernetes cluster running CoCo workloads.
  2. KBS deployed external to the Kubernetes cluster running CoCo workloads. This could be another Kubernetes cluster or a standalone VM.
mkulke commented 6 months ago

I think RVPS is also an open question for a real kbs deployment, because it uses a local embedded db as store, I opened an issue about this: https://github.com/confidential-containers/kbs/issues/338