search

confidential-containers / cloud-api-adaptor

Ability to create Kata pods using cloud provider APIs aka the peer-pods approach
Apache License 2.0
47 stars 77 forks source link

Security Questions & Discussion #1100

Open DeciderWill opened 1 year ago

DeciderWill commented 1 year ago

When running peer-pods and Kata remote hypervisor, it is most likely being deployed alongside a traditional Kata + Kubernetes setup. These tools often provide runtime security for a multi-tenant environment. In a conventional Kata deployment, if a bad actor can escape the container they are stuck/can run riot within the nested VM. In a peer-pod they are in the cloud VM. This raises additional security questions. For example: now that the bad actor has access to the cloud VM what can they do that they couldn't do in the nested VM.

One specific example is accessing cloud metadata APIs. Why is this risky? It returns service accounts and additional meta information that can be used to make requests on behalf of the node - like mount volumes or other actions. The user-data script is likely to also contain information to help the VM join the host peer-pods cluster? This node can also make network requests against other infrastructure within a project. Clearly, Workload Identify and least permission service accounts counter this but maybe for whatever reason that is not the case.

In a traditional K8s deployment with Kata, I can deploy Cilium on the host and use the network policies, host firewall and other features to prevent network access to certain ports and IPs f.e metadata IPs. When running peer-pods Cilium is not installed on the host node.

To start the discussion, other than metadata access what other attack vectors are now at risk or enabled?

bpradipt commented 1 year ago

@DeciderWill thanks for raising the issue. A related issue specific to metadata service is https://github.com/confidential-containers/cloud-api-adaptor/issues/1060

Marking few folks explicitly for their awareness @yoheiueda @stevenhorsman @huoqifeng @magowan @mkulke @surajssd @katexochen

mkulke commented 1 year ago

AFAIU, at the moment, the kata-agent jail for the container process is network namespaced so it's excluded from immediate access to a pod vm's metadata service.

In principle that is, because we just did implement a workaround to give a pod access to the metadata service anyway for remote attestation purpose.

I agree with you, though, that a peer pod should not have the means to access the pod vm's metadata service (it might require access to its k8s node's metadata service, however).

I think we want to remove the workaround and replace it with a fix in attestation-agent, so a pod doesn't need to call out to IMDS at runtime for the platform cert (VCEK).