Open wainersm opened 1 year ago
If you decode one of those base64 values in the org.opencontainers.image.enc.keys.provider.attestation-agent
annotation, you will see
{
"kid": "key_id1",
"wrapped_data": "spsZdrSmp42y0/nZ5NKILtb8ir1TArRV6kEBvxjD3ZY+ckN5I0j2nJI0fZ16ZgJtVo7BhpbR/vMYin1SQWdxbpvJndGF+pGLqg96+vs0RYXkaOxIsm4BdWPywLbPZZr65ZNxSoOKzK6DCTQkqN6mgBvfTg6cMZ7EmLBC2oS4nX3fqdTSQfnWHoh4A+ixfscGIZIKb6OzXg+V6gCaZWop2cVpWnn8sDYoTVbcE85QD3UVnYncVOCI+0g91P1trim9tQ==",
"iv": "U7Jmbot4v0FZ5m0TCwq1nA==",
"wrap_type": "aes_256_ctr"
}
Notice that kid
value is just key_id1
. This is an old image. After release 0.5.0 this key id is replaced by a more standardized resource id. The format of the resource id is checked somewhere before requesting a key from the KBS.
In the log you see that a request is made to the KBS, but this is the request for the launch bundle, rather than the key itself. The launch bundle just helps to setup the communication channel between the KBS and KBC. When using a new image, you should see another request for the key.
To generate a new image, you should check out the coco_keyprovider guide. Note that once you make this new image, you will need to provision the db of simple-kbs
with keys with a secret_id
that matches the new resource uri.
Hi @fitzthum, @wainersm. I'm facing a similar issue with v0.6.0 installed. I built a new image using the coco_keyprovider
by following this guide - https://www.redhat.com/en/blog/confidential-containers-amd-sev
Pod creation fails with: Failed to pull image "amulyam24/coco-custom-nginx:encrypted": rpc error: code = Internal desc = failed to handle layer: failed to get decrypt key missing private key needed for decryption
I can see that the policy is valid successfully in simple-kbs logs and the secret is also injected as per kata logs.
The error from logs
vmconsole="\x1b[0m\x1b[38;5;8m[\x1b[0m2023-08-18T10:54:39Z \x1b[0m\x1b[1m\x1b[31mERROR\x1b[0m attestation_agent::rpc::getresource::ttrpc\x1b[0m\x1b[38;5;8m]\x1b[0m Call AA-KBC to get resource failed: status: Unavailable, message: \"error trying to connect: tcp connect error: Connection refused (os error 111)\", details: [], metadata: MetadataMap { headers: {} }”
Any idea what's going wrong here or how to debug this issue further?
Hm. A bit hard to know without more info, but one thing to keep in mind is that there will actually be two connections to the KBS. First, the shim will get the launch bundle from the simple-kbs. This is probably what you are seeing as occurring successfully in the log of the KBS. Then the AA will try to fetch secrets from the KBS from inside the guest. This second connection seems to be failing. You should double check the KBS_URI parameter and make sure that it is reachable from inside the guest (don't set it to localhost, for instance).
You should double check the KBS_URI parameter and make sure that it is reachable from inside the guest
Thanks @fitzthum, this was the issue. I specified the KBS_URI as 0.0.0.0:{port} and it worked when I specified the right IP.
I got a single-node Kubernetes 1.24.0 cluster on an AMD SEV machine. Recently I got it installed CoCo 0.5.0 (previously using 0.3.0) but I am not being able to start a simple pod from an encrypted image (same image used to work with CoCo 0.3.0).
First question: should I rebuild the image for 0.5.0?
Assuming the old image should work with 0.5.0, here goes more information.
The output of
kubectl describe
indicates theimage_rs
didn't get the key:simple-kbs got the request, meaning the attestation-agent can talk with KBS, and validated the policy:
Should simple-kbs log a message to confirm the key set was released? Anyway...
The MySQL datase seems to have the correct data:
More details about the docker.io/wainersm/coco-custom-nginx:encrypted image: