search

confidential-containers / cloud-api-adaptor

Ability to create Kata pods using cloud provider APIs aka the peer-pods approach
Apache License 2.0
44 stars 71 forks source link

libvirt: e2e test for attestation for sample tee #1824

Closed huoqifeng closed 1 month ago

huoqifeng commented 2 months ago

Fixes: #1825

This PR is based on the azure approach: https://github.com/confidential-containers/cloud-api-adaptor/pull/1735

huoqifeng commented 2 months ago

Created a cluster and run e2e test like below:

huoqifeng commented 2 months ago

I'll use a new PR to enable this test case in libvirt_e2e GHA.

stevenhorsman commented 1 month ago

In the instructions can we use: make CLI_FEATURES=sample_only cli to reduce all the extra packages we need?

stevenhorsman commented 1 month ago

I tried to test this locally and hit an error:

=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
time="2024-05-07T14:45:09Z" level=error msg="Failed to access key.bin: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \\\"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\\\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
    assessment_runner.go:417: Command [wget -q -O- http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin] running in container busybox produced unexpected output on stdout: rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }

I'm not sure of the cause though at the moment.

stevenhorsman commented 1 month ago

Can we also add a negative test here, such that we confirm that set-ups without using the permissive policy get rejected as expected?

huoqifeng commented 1 month ago

In the instructions can we use: make CLI_FEATURES=sample_only cli to reduce all the extra packages we need?

Yes, agreed

huoqifeng commented 1 month ago

I tried to test this locally and hit an error:

=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
time="2024-05-07T14:45:09Z" level=error msg="Failed to access key.bin: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \\\"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\\\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
    assessment_runner.go:417: Command [wget -q -O- http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin] running in container busybox produced unexpected output on stdout: rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }

I'm not sure of the cause though at the moment.

I think it's because the parameter TEE_CUSTOMIZED_OPA is missed. Updated the steps above https://github.com/confidential-containers/cloud-api-adaptor/pull/1824#issuecomment-2084577039, also added negative test cases when TEE_CUSTOMIZED_OPA is missing.

huoqifeng commented 1 month ago

lint / govulncheck error here https://github.com/confidential-containers/cloud-api-adaptor/actions/runs/8996124825/job/24712137125?pr=1824 is tracked in https://github.com/confidential-containers/cloud-api-adaptor/pull/1836

stevenhorsman commented 1 month ago

It is probably user error, but the KeyReleaseFailure test is failing for me:

=== RUN   TestLibvirtKbsKeyRelease
time="2024-05-09T10:03:39Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/deny_all.rego"
time="2024-05-09T10:03:39Z" level=trace msg="./kbs-client --url http://192.168.122.76:30992 config --auth-private-key ../../kbs/config/kubernetes/base/kbs.key set-resource-policy --policy-file ../../kbs/sample_policies/deny_all.rego, output: Set resource policy success \n policy: CnBhY2thZ2UgcG9saWN5CgpkZWZhdWx0IGFsbG93ID0gZmFsc2UK\n"
=== PAUSE TestLibvirtKbsKeyRelease
=== CONT  TestLibvirtKbsKeyRelease
time="2024-05-09T10:03:39Z" level=info msg="Do test kbs key release failure case"
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget-failure are ready
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
time="2024-05-09T10:04:24Z" level=error msg="Failed to faliure case as: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \\\"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\\\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
    assessment_runner.go:417: Command [wget -q -O- http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin] running in container busybox produced unexpected output on stdout: rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
time="2024-05-09T10:04:24Z" level=info msg="Deleting pod busybox-wget-failure..."
time="2024-05-09T10:04:29Z" level=info msg="Pod busybox-wget-failure has been successfully deleted within 60s"
time="2024-05-09T10:04:29Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
time="2024-05-09T10:04:30Z" level=trace msg="./kbs-client --url http://192.168.122.76:30992 config --auth-private-key ../../kbs/config/kubernetes/base/kbs.key set-resource-policy --policy-file ../../kbs/sample_policies/allow_all.rego, output: Set resource policy success \n policy: CnBhY2thZ2UgcG9saWN5CgpkZWZhdWx0IGFsbG93ID0gdHJ1ZQoK\n"
time="2024-05-09T10:04:30Z" level=info msg="Do test kbs key release"
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
time="2024-05-09T10:05:15Z" level=error msg="Failed to access key.bin: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \\\"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\\\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
    assessment_runner.go:417: Command [wget -q -O- http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin] running in container busybox produced unexpected output on stdout: rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
time="2024-05-09T10:05:15Z" level=info msg="Deleting pod busybox-wget..."
time="2024-05-09T10:05:20Z" level=info msg="Pod busybox-wget has been successfully deleted within 60s"
--- FAIL: TestLibvirtKbsKeyRelease (100.40s)
    --- FAIL: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test (50.17s)
        --- FAIL: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed (5.12s)
    --- FAIL: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test (50.20s)
        --- FAIL: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful (5.16s)

It also doesn't report PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test like I would expect. Maybe as they are running in the same Test function?

huoqifeng commented 1 month ago

I run it again with after with new trustee repo cloned, looks good:

# cat /root/libvirt.properties
libvirt_uri="qemu+ssh://root@192.168.122.1/system?no_verify=1"
libvirt_ssh_key_file="id_rsa"
CLUSTER_NAME = "peer-pods"
KBS_IMAGE = "ghcr.io/confidential-containers/staged-images/kbs"
KBS_IMAGE_TAG = "dc01f454264fb4350e5f69eba05683a9a1882c41"

~/src/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e/trustee/kbs# cd ../
~/src/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e/trustee# git branch
* (HEAD detached at dc01f45)
  main

export TEST_PROVISION_FILE="/root/libvirt.properties"
export CLOUD_PROVIDER=libvirt
export DEPLOY_KBS=true
export TEST_INSTALL_CAA=yes
export TEST_TEARDOWN=no
make test-e2e
go test -v -tags=libvirt -timeout 60m -count=1 ./test/e2e
time="2024-05-09T11:22:16Z" level=info msg="Do setup"
time="2024-05-09T11:22:16Z" level=info msg="Deploying kbs"
time="2024-05-09T11:22:16Z" level=info msg="creating key.bin"
/root/src/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e
time="2024-05-09T11:22:16Z" level=info msg="Creating kbs install overlay"
time="2024-05-09T11:22:16Z" level=info msg="Customize the overlay yaml file"
time="2024-05-09T11:22:16Z" level=info msg="Updating kbs image with \"ghcr.io/confidential-containers/staged-images/kbs\""
time="2024-05-09T11:22:16Z" level=info msg="Updating kbs image tag with \"dc01f454264fb4350e5f69eba05683a9a1882c41\""
time="2024-05-09T11:22:16Z" level=info msg="Creating kbs install overlay"
time="2024-05-09T11:22:16Z" level=info msg="Install Kbs"
Wait for the kbs deployment be available
time="2024-05-09T11:22:21Z" level=info msg="KBS PARAMS: cc_kbc::http://192.168.122.123:31126"
time="2024-05-09T11:22:21Z" level=info msg="Install Cloud API Adaptor"
time="2024-05-09T11:22:21Z" level=info msg="Deploy the Cloud API Adaptor"
time="2024-05-09T11:22:21Z" level=info msg="Install the controller manager"
Wait for the cc-operator-controller-manager deployment be available
time="2024-05-09T11:22:39Z" level=info msg="Customize the overlay yaml file"
time="2024-05-09T11:22:41Z" level=info msg="Install the cloud-api-adaptor"
Wait for the pod cloud-api-adaptor-daemonset-jz9lv be ready
Wait for the kata-remote runtimeclass be created
time="2024-05-09T11:23:31Z" level=info msg="Installing peerpod-ctrl"
time="2024-05-09T11:23:32Z" level=info msg="Wait for the peerpod-ctrl deployment to be available"
time="2024-05-09T11:23:47Z" level=info msg="Creating namespace 'coco-pp-e2e-test-00b04760'..."
time="2024-05-09T11:23:47Z" level=info msg="Wait for namespace 'coco-pp-e2e-test-00b04760' be ready..."
time="2024-05-09T11:23:52Z" level=info msg="Wait for default serviceaccount in namespace 'coco-pp-e2e-test-00b04760'..."
time="2024-05-09T11:23:52Z" level=info msg="default serviceAccount exists, namespace 'coco-pp-e2e-test-00b04760' is ready for use"
=== RUN   TestLibvirtKbsKeyRelease
time="2024-05-09T11:23:52Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/deny_all.rego"
=== PAUSE TestLibvirtKbsKeyRelease
=== CONT  TestLibvirtKbsKeyRelease
time="2024-05-09T11:23:52Z" level=info msg="Do test kbs key release failure case"
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget-failure are ready
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
time="2024-05-09T11:24:38Z" level=info msg="Pass failure case as: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: request unautorized\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: request unautorized", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
time="2024-05-09T11:24:38Z" level=info msg="Deleting pod busybox-wget-failure..."
time="2024-05-09T11:24:43Z" level=info msg="Pod busybox-wget-failure has been successfully deleted within 60s"
time="2024-05-09T11:24:43Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
time="2024-05-09T11:24:43Z" level=info msg="Do test kbs key release"
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
time="2024-05-09T11:25:29Z" level=info msg="Success to get key.bin This is my cluster name: "
    assessment_runner.go:415: Output when execute test commands:This is my cluster name: 
time="2024-05-09T11:25:29Z" level=info msg="Deleting pod busybox-wget..."
time="2024-05-09T11:25:34Z" level=info msg="Pod busybox-wget has been successfully deleted within 60s"
--- PASS: TestLibvirtKbsKeyRelease (101.70s)
    --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test (51.19s)
        --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed (6.15s)
    --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test (50.50s)
        --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful (5.46s)
PASS
ok      github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e 197.629s

I'll check your log later @stevenhorsman

huoqifeng commented 1 month ago

@stevenhorsman can you help cat your libvirt.properties file and the the commit in trustee repo?

stevenhorsman commented 1 month ago

@stevenhorsman can you help cat your libvirt.properties file and the the commit in trustee repo?

I think my settings are the same as yours. The only difference I've made is update the test-e2e target to only run the KBS test:

# cat libvirt.properties
libvirt_uri="qemu+ssh://root@10.250.129.5/system?no_verify=1"
libvirt_ssh_key_file="id_rsa"
CLUSTER_NAME = "peer-pods"
KBS_IMAGE = "ghcr.io/confidential-containers/staged-images/kbs"
KBS_IMAGE_TAG = "dc01f454264fb4350e5f69eba05683a9a1882c41"
root@sh-libvirt-switch-main:~/go/src/github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e/trustee# git log -1 --oneline
dc01f45 (HEAD) k8s-config: Add support for NodePort service type
export TEST_PROVISION_FILE="$PWD/libvirt.properties"
export TEST_PROVISION="no"
export CLOUD_PROVIDER=libvirt
export DEPLOY_KBS=true
export TEST_INSTALL_CAA=yes
export TEST_TEARDOWN=no
export KUBECONFIG=$HOME/.kcli/clusters/peer-pods/auth/kubeconfig
make test-e2e
go test -v --run TestLibvirtKbsKeyRelease  -tags=libvirt -timeout 50m -count=1 ./test/e2e
time="2024-05-09T11:50:27Z" level=info msg="Do setup"
time="2024-05-09T11:50:27Z" level=info msg="Deploying kbs"
time="2024-05-09T11:50:27Z" level=info msg="creating key.bin"
/root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e
time="2024-05-09T11:50:27Z" level=info msg="Creating kbs install overlay"
time="2024-05-09T11:50:27Z" level=info msg="Customize the overlay yaml file"
time="2024-05-09T11:50:27Z" level=info msg="Updating kbs image with \"ghcr.io/confidential-containers/staged-images/kbs\""
time="2024-05-09T11:50:27Z" level=info msg="Updating kbs image tag with \"dc01f454264fb4350e5f69eba05683a9a1882c41\""
time="2024-05-09T11:50:27Z" level=info msg="Creating kbs install overlay"
time="2024-05-09T11:50:27Z" level=info msg="Install Kbs"
Wait for the kbs deployment be available
time="2024-05-09T11:50:32Z" level=info msg="KBS PARAMS: cc_kbc::http://192.168.122.76:30992"
time="2024-05-09T11:50:32Z" level=info msg="Podvm uploading"
time="2024-05-09T11:50:32Z" level=trace msg="UploadPodvm()"
time="2024-05-09T11:50:35Z" level=info msg="Install Cloud API Adaptor"
time="2024-05-09T11:50:35Z" level=info msg="Deploy the Cloud API Adaptor"
time="2024-05-09T11:50:35Z" level=info msg="Install the controller manager"
time="2024-05-09T11:50:37Z" level=trace msg="/usr/local/bin/kubectl apply -k github.com/confidential-containers/operator/config/release?ref=v0.8.0, output: namespace/confidential-containers-system configured\ncustomresourcedefinition.apiextensions.k8s.io/ccruntimes.confidentialcontainers.org configured\nserviceaccount/cc-operator-controller-manager unchanged\nrole.rbac.authorization.k8s.io/cc-operator-leader-election-role unchanged\nclusterrole.rbac.authorization.k8s.io/cc-operator-manager-role configured\nclusterrole.rbac.authorization.k8s.io/cc-operator-metrics-reader unchanged\nclusterrole.rbac.authorization.k8s.io/cc-operator-proxy-role unchanged\nrolebinding.rbac.authorization.k8s.io/cc-operator-leader-election-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cc-operator-manager-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cc-operator-proxy-rolebinding unchanged\nconfigmap/cc-operator-manager-config unchanged\nservice/cc-operator-controller-manager-metrics-service unchanged\ndeployment.apps/cc-operator-controller-manager unchanged\n"
Wait for the cc-operator-controller-manager deployment be available
time="2024-05-09T11:50:42Z" level=info msg="Customize the overlay yaml file"
time="2024-05-09T11:51:05Z" level=trace msg="/usr/local/bin/kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/peer-pods?ref=v0.8.0, output: ccruntime.confidentialcontainers.org/ccruntime-peer-pods configured\n"
time="2024-05-09T11:51:05Z" level=info msg="Install the cloud-api-adaptor"
Wait for the cc-operator-daemon-install DaemonSet be available
Wait for the pod cc-operator-daemon-install-8hc2n be ready
Wait for the cloud-api-adaptor-daemonset DaemonSet be available
Wait for the pod cloud-api-adaptor-daemonset-r2ktq be ready
Wait for the kata-remote runtimeclass be created
time="2024-05-09T11:51:30Z" level=info msg="Installing peerpod-ctrl"
time="2024-05-09T11:51:32Z" level=trace msg="/usr/bin/make -C ../peerpod-ctrl deploy, output: make[1]: Entering directory '/root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/peerpod-ctrl'\ntest -s /root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/peerpod-ctrl/bin/controller-gen || GOBIN=/root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/peerpod-ctrl/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.14.0\n/root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/peerpod-ctrl/bin/controller-gen rbac:roleName=manager-role crd webhook paths=\"./...\" output:crd:artifacts:config=config/crd/bases\ncd config/manager && /root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/peerpod-ctrl/bin/kustomize edit set image controller=quay.io/confidential-containers/peerpod-ctrl:latest\n/root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/peerpod-ctrl/bin/kustomize build config/default | kubectl apply -f -\n# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.\nnamespace/confidential-containers-system configured\ncustomresourcedefinition.apiextensions.k8s.io/peerpods.confidentialcontainers.org unchanged\nserviceaccount/peerpod-ctrl-controller-manager unchanged\nrole.rbac.authorization.k8s.io/peerpod-ctrl-leader-election-role unchanged\nclusterrole.rbac.authorization.k8s.io/peerpod-ctrl-manager-role unchanged\nclusterrole.rbac.authorization.k8s.io/peerpod-ctrl-metrics-reader unchanged\nclusterrole.rbac.authorization.k8s.io/peerpod-ctrl-proxy-role unchanged\nrolebinding.rbac.authorization.k8s.io/peerpod-ctrl-leader-election-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/peerpod-ctrl-manager-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/peerpod-ctrl-proxy-rolebinding unchanged\nservice/peerpod-ctrl-controller-manager-metrics-service unchanged\ndeployment.apps/peerpod-ctrl-controller-manager unchanged\nmake[1]: Leaving directory '/root/go/src/github.com/confidential-containers/cloud-api-adaptor/src/peerpod-ctrl'\n"
time="2024-05-09T11:51:32Z" level=info msg="Wait for the peerpod-ctrl deployment to be available"
time="2024-05-09T11:51:37Z" level=info msg="Creating namespace 'coco-pp-e2e-test-bf40ab10'..."
time="2024-05-09T11:51:37Z" level=info msg="Wait for namespace 'coco-pp-e2e-test-bf40ab10' be ready..."
time="2024-05-09T11:51:42Z" level=info msg="Wait for default serviceaccount in namespace 'coco-pp-e2e-test-bf40ab10'..."
time="2024-05-09T11:51:42Z" level=info msg="default serviceAccount exists, namespace 'coco-pp-e2e-test-bf40ab10' is ready for use"
=== RUN   TestLibvirtKbsKeyRelease
time="2024-05-09T11:51:42Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/deny_all.rego"
time="2024-05-09T11:51:42Z" level=trace msg="./kbs-client --url http://192.168.122.76:30992 config --auth-private-key ../../kbs/config/kubernetes/base/kbs.key set-resource-policy --policy-file ../../kbs/sample_policies/deny_all.rego, output: Set resource policy success \n policy: CnBhY2thZ2UgcG9saWN5CgpkZWZhdWx0IGFsbG93ID0gZmFsc2UK\n"
=== PAUSE TestLibvirtKbsKeyRelease
=== CONT  TestLibvirtKbsKeyRelease
time="2024-05-09T11:51:42Z" level=info msg="Do test kbs key release failure case"
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget-failure are ready
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
time="2024-05-09T11:52:27Z" level=error msg="Failed to faliure case as: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \\\"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\\\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
    assessment_runner.go:417: Command [wget -q -O- http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin] running in container busybox produced unexpected output on stdout: rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
time="2024-05-09T11:52:27Z" level=info msg="Deleting pod busybox-wget-failure..."
time="2024-05-09T11:52:32Z" level=info msg="Pod busybox-wget-failure has been successfully deleted within 60s"
time="2024-05-09T11:52:32Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
time="2024-05-09T11:52:32Z" level=trace msg="./kbs-client --url http://192.168.122.76:30992 config --auth-private-key ../../kbs/config/kubernetes/base/kbs.key set-resource-policy --policy-file ../../kbs/sample_policies/allow_all.rego, output: Set resource policy success \n policy: CnBhY2thZ2UgcG9saWN5CgpkZWZhdWx0IGFsbG93ID0gdHJ1ZQoK\n"
time="2024-05-09T11:52:32Z" level=info msg="Do test kbs key release"
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
time="2024-05-09T11:53:17Z" level=error msg="Failed to access key.bin: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \\\"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\\\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
    assessment_runner.go:417: Command [wget -q -O- http://127.0.0.1:8006/cdh/resource/reponame/workload_key/key.bin] running in container busybox produced unexpected output on stdout: rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: get token failed: Attestation Agent token provider error: cal ttrpc failed: rpc status: Status { code: INTERNAL, message: \"[ERROR:attestation-agent] AA-KBC get token failed: Unsupported token type: Matching variant not found\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
time="2024-05-09T11:53:17Z" level=info msg="Deleting pod busybox-wget..."
time="2024-05-09T11:53:22Z" level=info msg="Pod busybox-wget has been successfully deleted within 60s"
--- FAIL: TestLibvirtKbsKeyRelease (100.43s)
    --- FAIL: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test (50.22s)
        --- FAIL: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed (5.18s)
    --- FAIL: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test (50.20s)
        --- FAIL: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful (5.16s)
FAIL
FAIL    github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e 175.820s
FAIL
make: *** [Makefile:95: test-e2e] Error 1

Let me know if there is anything else you need, or I could give you access to my test box to try yourself?

huoqifeng commented 1 month ago

It's identified that the PeerPod VM image from quay.io/confidential-containers/podvm-generic-ubuntu-amd64:e5a6fb8fdb34943caceea738770f79b9db87faa1 works well but the image from quay.io/confidential-containers/podvm-generic-ubuntu-amd64:v0.8.2 failed the test. It's interesting. v0.8.2 and latest has same guest-components commit 277617af60c32661819c1132ffbf3db8dc6e1b9f, I'll investigate the reason...

huoqifeng commented 1 month ago 
# journalctl -t kata-agent |grep kbc
May 11 00:06:28 podvm-busybox-92b37f81 kata-agent[828]: {"msg":"announce","level":"INFO","ts":"2024-05-11T00:06:28.531813269Z","subsystem":"root","version":"0.1.0","source":"agent","pid":"828","name":"kata-agent","api-version":"0.0.1","agent-commit":"3.2.0-rc0-d0df91935b8840036c2891b1f93dd8059ebe486a","config":"AgentConfig { debug_console: false, dev_mode: false, log_level: Info, hotplug_timeout: 3s, debug_console_vport: 0, log_vport: 0, container_pipe_size: 0, server_addr: \"unix:///run/kata-containers/agent.sock\", unified_cgroup_hierarchy: false, tracing: false, endpoints: AgentEndpoints { allowed: {\"ExecProcessRequest\", \"GetMetricsRequest\", \"OnlineCPUMemRequest\", \"RemoveContainerRequest\", \"StartContainerRequest\", \"GuestDetailsRequest\", \"AddSwapRequest\", \"MemHotplugByProbeRequest\", \"CloseStdinRequest\", \"ReseedRandomDevRequest\", \"GetOOMEventRequest\", \"ResumeContainerRequest\", \"StopTracingRequest\", \"TtyWinResizeRequest\", \"UpdateContainerRequest\", \"SignalProcessRequest\", \"WaitProcessRequest\", \"CreateContainerRequest\", \"CopyFileRequest\", \"DestroySandboxRequest\", \"ListInterfacesRequest\", \"PullImageRequest\", \"CreateSandboxRequest\", \"AddARPNeighborsRequest\", \"UpdateRoutesRequest\", \"StartTracingRequest\", \"ListRoutesRequest\", \"PauseContainerRequest\", \"UpdateInterfaceRequest\", \"SetGuestDateTimeRequest\", \"StatsContainerRequest\", \"ReadStreamRequest\", \"SetPolicyRequest\", \"WriteStreamRequest\"}, all_allowed: false }, supports_seccomp: true, container_policy_path: \"\", aa_kbc_params: \"cc_kbc::http://192.168.122.123:32601\", rest_api: \"\", https_proxy: \"\", no_proxy: \"\", data_integrity: false, enable_signature_verification: false, image_policy_file: \"\", image_registry_auth_file: \"\", simple_signing_sigstore_config: \"\" }","agent-version":"3.2.0-rc0"}
May 11 00:06:30 podvm-busybox-92b37f81 kata-agent[925]: [2024-05-11T00:06:30Z WARN  kms::plugins::kbs::cc_kbc] KBS_CERT get failed: NotPresent. Use no KBS public key certs.
May 11 00:06:33 podvm-busybox-92b37f81 kata-agent[876]: [2024-05-11T00:06:33Z ERROR attestation_agent::rpc::getresource::ttrpc] Call AA-KBC to get resource failed: AA does not support the given KBC module! Module: cc_kbc
May 11 00:06:35 podvm-busybox-92b37f81 kata-agent[876]: [2024-05-11T00:06:35Z ERROR attestation_agent::rpc::getresource::ttrpc] Call AA-KBC to get resource failed: AA does not support the given KBC module! Module: cc_kbc

Seems cc_kbc was not enabled in kata-agent/cdh/asr in the PodVM image in v0.8.2.

huoqifeng commented 1 month ago

Updated the readme. I'll add this AA_KBC=cc_kbc parameter when build the PodVM image when run libvirt_e2e GHA in a new PR.

stevenhorsman commented 1 month ago

I've added the test_e2e_libvirt label now that there is a podvm change to ensure we don't cause issues to existing tests, but am happy that the libvirt workflow changes to deploy the KBS and therefore test the new functions can be done in a subsequent PR.

stevenhorsman commented 1 month ago

When I re-built the podvm image with the cc_kbs the tests passed:

=== RUN   TestLibvirtKbsKeyRelease
time="2024-05-13T13:45:40Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/deny_all.rego"
time="2024-05-13T13:45:40Z" level=trace msg="./kbs-client --url http://192.168.122.242:31597 config --auth-private-key ../../kbs/config/kubernetes/base/kbs.key set-resource-policy --policy-file ../../kbs/sample_policies/deny_all.rego, output: Set resource policy success \n policy: CnBhY2thZ2UgcG9saWN5CgpkZWZhdWx0IGFsbG93ID0gZmFsc2UK\n"
=== PAUSE TestLibvirtKbsKeyRelease
=== CONT  TestLibvirtKbsKeyRelease
time="2024-05-13T13:45:40Z" level=info msg="Do test kbs key release failure case"
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget-failure are ready
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
time="2024-05-13T13:46:51Z" level=info msg="Pass failure case as: rpc status: Status { code: INTERNAL, message: \"[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: request unautorized\", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }"
    assessment_runner.go:415: Output when execute test commands:rpc status: Status { code: INTERNAL, message: "[CDH] [ERROR]: Get Resource failed: get resource failed: get rersource failed: Kbs client error: get resource failed: request unautorized", details: [], special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }
time="2024-05-13T13:46:51Z" level=info msg="Deleting pod busybox-wget-failure..."
time="2024-05-13T13:46:56Z" level=info msg="Pod busybox-wget-failure has been successfully deleted within 60s"
time="2024-05-13T13:46:56Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
time="2024-05-13T13:46:56Z" level=trace msg="./kbs-client --url http://192.168.122.242:31597 config --auth-private-key ../../kbs/config/kubernetes/base/kbs.key set-resource-policy --policy-file ../../kbs/sample_policies/allow_all.rego, output: Set resource policy success \n policy: CnBhY2thZ2UgcG9saWN5CgpkZWZhdWx0IGFsbG93ID0gdHJ1ZQoK\n"
time="2024-05-13T13:46:56Z" level=info msg="Do test kbs key release"
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: busybox-wget are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
time="2024-05-13T13:47:41Z" level=info msg="Success to get key.bin This is my cluster name: "
    assessment_runner.go:415: Output when execute test commands:This is my cluster name:
time="2024-05-13T13:47:41Z" level=info msg="Deleting pod busybox-wget..."
time="2024-05-13T13:47:46Z" level=info msg="Pod busybox-wget has been successfully deleted within 60s"
--- PASS: TestLibvirtKbsKeyRelease (126.63s)
    --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test (76.31s)
        --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed (6.26s)
    --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test (50.31s)
        --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful (5.27s)
huoqifeng commented 1 month ago

azure e2e test action https://github.com/confidential-containers/cloud-api-adaptor/blob/main/.github/workflows/azure-e2e-test.yml#L224 put trustee repo in ${cloud-api-adaptor-repo-dir}/src/cloud-api-adaptor/test in stead of ${cloud-api-adaptor-repo-dir}/src/cloud-api-adaptor/test/e2e, make correspond change in readme and TRUSTEE_REPO_PATH.