search

confidential-containers / cloud-api-adaptor

Ability to create Kata pods using cloud provider APIs aka the peer-pods approach
Apache License 2.0
44 stars 71 forks source link

Get policies working on `main` based peer pods #1874

Open stevenhorsman opened 1 week ago

stevenhorsman commented 1 week ago

As a peer pods workload owner I want to have be able to block certain endpoints so that I can restrict the interactions that users can have with my workloads

Description In https://github.com/confidential-containers/cloud-api-adaptor/pull/1607 Pradipta added policy based on the CCv0 stream, however after some testing that @snir911 and I have done, we've found it doesn't work with the changed implementation on kata-containers main branch.

We should investigate this, try and get it working and ideally add the e2e test that were supposed to come following that PR.

Acceptance Criteria

Scenario: Block exec command Given a peer pods set-up including the "latest" kata-containers main codebase and a pod created, which includes the annotation io.katacontainers.config.agent.policy: which is set to the base64 versions of allow-all-except-exec-process.rego When we run kubectl exec <pod_name> -- <command> Then The request is blocked with an error that includes: failed to exec in container and Validation failed for data.policy.ExecProcess

Scenario: Fully permission policy Given a peer pods set-up including the "latest" kata-containers main codebase and a pod created, which includes the annotation io.katacontainers.config.agent.policy: which is set to the base64 versions of allow-all.rego When we run kubectl exec <pod_name> -- <command> Then The request is successful

Scenario: Check default policy is permissive Given a peer pods set-up including the "latest" kata-containers main codebase and a pod created, with no io.katacontainers.config.agent.policy: annotation When we run kubectl exec <pod_name> -- <command> Then The request is successful

Note: This is implicitly tested by several other tests that use kubectl exec

stevenhorsman commented 1 week ago

In our initial investigation it looks like there are potentially two problems:

  1. It looks like the policy stuff should be set in https://github.com/kata-containers/kata-containers/blob/b218c4bc1096f56b62830f892ff516cea44a60b2/src/runtime/pkg/oci/utils.go#L952 and read from the ociSpec annotations, but there doesn't seem to be anything in the logs to show that is going through and we end up with no policy in the AgentConfig: 
    AgentConfig:{KernelModules:[] ContainerPipeSize:0 DialTimeout:0 LongLiveConn:true Debug:false Trace:false EnableDebugConsole:false Policy:}

    . We probably need some more logging here to debug

  2. The SetPolicy command isn't invoked on the remote hypervisor: https://github.com/kata-containers/kata-containers/blob/cf372f41bf5a6e75b2f8aeb15038c2b68077e50c/src/runtime/virtcontainers/kata_agent.go#L767-L780 we will need a kata-containers change for this.
stevenhorsman commented 1 week ago

OK, I create a local branch which fixed part 2 and even though Policy wasn't logged in the AgentConfig that worked, so maybe it's missing for security reasons or something? I'll create the kata-containers PR and hopefully fix the problem soon.

stevenhorsman commented 1 week ago

https://github.com/kata-containers/kata-containers/pull/9881 created

stevenhorsman commented 1 week ago

9881 has been merged now, so I've created https://github.com/confidential-containers/cloud-api-adaptor/pull/1876 to add the e2e tests for this