search

confidential-containers / cloud-api-adaptor

Ability to create Kata pods using cloud provider APIs aka the peer-pods approach
Apache License 2.0
48 stars 88 forks source link

failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Address is not allowed #2138

Closed gaussye closed 1 week ago

gaussye commented 3 weeks ago

Describe the bug

I follow the guide https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/aws/README.md to deploy CAA on EKS(1.30), i got the following error message. It blocks the remaining steps.

clusterrolebinding.rbac.authorization.k8s.io/peer-pods-webhook-manager-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/peer-pods-webhook-proxy-rolebinding created configmap/peer-pods-webhook-manager-config created service/peer-pods-webhook-controller-manager-metrics-service created service/peer-pods-webhook-webhook-service created deployment.apps/peer-pods-webhook-controller-manager created mutatingwebhookconfiguration.admissionregistration.k8s.io/peer-pods-webhook-mutating-webhook-configuration created Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Address is not allowed Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Address is not allowed make[1]: [deploy] Error 1 make[1]: Leaving directory `/kata-container-testing/cloud-api-adaptor/src/webhook' make: [deploy] Error 2


kubectl get pods -A
NAMESPACE                        NAME                                                    READY   STATUS              RESTARTS   AGE
calico-system                    calico-kube-controllers-68474795c6-szz25                1/1     Running             0          4h51m
calico-system                    calico-node-p2cpc                                       1/1     Running             0          4h51m
calico-system                    calico-node-srfr9                                       1/1     Running             0          4h51m
calico-system                    calico-typha-c7995bf7b-nh9bl                            1/1     Running             0          4h51m
calico-system                    csi-node-driver-fjx7s                                   2/2     Running             0          4h51m
calico-system                    csi-node-driver-h28lf                                   2/2     Running             0          4h51m
cert-manager                     cert-manager-756d54fb98-swdk8                           1/1     Running             0          3h14m
cert-manager                     cert-manager-cainjector-7d96c69dbf-7b692                1/1     Running             0          3h14m
cert-manager                     cert-manager-webhook-778c78f68c-qhffs                   1/1     Running             0          3h14m
confidential-containers-system   cc-operator-controller-manager-557b5cbdc5-wvznz         2/2     Running             0          18m
confidential-containers-system   cloud-api-adaptor-daemonset-4dz2s                       1/1     Running             0          12m
confidential-containers-system   cloud-api-adaptor-daemonset-q2rn5                       1/1     Running             0          12m
confidential-containers-system   peerpod-ctrl-controller-manager-6b5fc64766-xdgmv        2/2     Running             0          4m18s
kube-system                      coredns-586b798467-25hb5                                1/1     Running             0          5h5m
kube-system                      coredns-586b798467-rv8gd                                1/1     Running             0          5h5m
kube-system                      kube-proxy-qqkbl                                        1/1     Running             0          4h52m
kube-system                      kube-proxy-xmrq8                                        1/1     Running             0          4h52m
peer-pods-webhook-system         peer-pods-webhook-controller-manager-6d7848fd45-sctqk   0/2     ContainerCreating   0          4m12s
peer-pods-webhook-system         peer-pods-webhook-controller-manager-6d7848fd45-twxjc   0/2     ContainerCreating   0          4m12s
tigera-operator                  tigera-operator-76ff79f7fd-rdgj2                        1/1     Running             0          5h2m
kubectl get svc -A
calico-system                    calico-kube-controllers-metrics                        ClusterIP   None            <none>        9094/TCP                 4h51m
calico-system                    calico-typha                                           ClusterIP   172.20.22.94    <none>        5473/TCP                 4h52m
cert-manager                     cert-manager                                           ClusterIP   172.20.138.63   <none>        9402/TCP                 3h15m
cert-manager                     cert-manager-webhook                                   ClusterIP   172.20.86.205   <none>        443/TCP                  37m
confidential-containers-system   cc-operator-controller-manager-metrics-service         ClusterIP   172.20.206.56   <none>        8443/TCP                 19m
confidential-containers-system   peerpod-ctrl-controller-manager-metrics-service        ClusterIP   172.20.38.249   <none>        8443/TCP                 5m5s
default                          kubernetes                                             ClusterIP   172.20.0.1      <none>        443/TCP                  5h10m
kube-system                      kube-dns                                               ClusterIP   172.20.0.10     <none>        53/UDP,53/TCP,9153/TCP   5h6m
peer-pods-webhook-system         peer-pods-webhook-controller-manager-metrics-service   ClusterIP   172.20.105.1    <none>        8443/TCP                 4m59s
peer-pods-webhook-system         peer-pods-webhook-webhook-service                      ClusterIP   172.20.91.117   <none>        443/TCP                  4m59s

How to reproduce

follow the guide https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/aws/README.md

CoCo version information

quay.io/confidential-containers/operator:v0.10.0

What TEE are you seeing the problem on

None

Failing command and relevant log output

No response

gaussye commented 3 weeks ago

duplicated with issue https://github.com/cert-manager/cert-manager/issues/3645

bpradipt commented 2 weeks ago

@gaussye as you already found it there is an issue with deploying cert-manager on EKS when not using the VPC CNI. I tried the following sequence to overcome the error

  1. Setup EKS cluster
  2. Allow port 10260 in the EKS security group 
    aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 10260 --cidr "$EKS_VPC_CIDR"
  3. Setup cert-manager using helm 
    helm install \  
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.16.1 \
--set crds.enabled=true \
--set webhook.hostNetwork=true \
--set webhook.securePort=10260
  4. Deploy cloud-api-adaptor
  5. Patch peer-pods-webhook-controller-manager deployment to use hostNetwork: true 
    kubectl patch deployment peer-pods-webhook-controller-manager -n peer-pods-webhook-system --type='json' -p='[{"op": "add", "path": "/spec/template/spec/hostNetwork", "value": true}]'
  6. Deploy your pod.

I'll add it to the README. Also you can use default AWS VPC CNI. It works now with latest code. I'll create a PR soon to update the README

bpradipt commented 1 week ago

Documentation updated via - https://github.com/confidential-containers/cloud-api-adaptor/pull/2148

Chasing1020 commented 3 days ago

@bpradipt I want to deploy the CAA on my local k8s (v1.31.2, not EKS) cluter, and I use cilium (v1.16.3) as the CNI.

I still have the same bug on the latest doc.

The detailed output of make deploy ```bash make deploy ... namespace/peer-pods-webhook-system created serviceaccount/peer-pods-webhook-controller-manager created role.rbac.authorization.k8s.io/peer-pods-webhook-leader-election-role created clusterrole.rbac.authorization.k8s.io/peer-pods-webhook-manager-role configured clusterrole.rbac.authorization.k8s.io/peer-pods-webhook-metrics-reader unchanged clusterrole.rbac.authorization.k8s.io/peer-pods-webhook-proxy-role unchanged rolebinding.rbac.authorization.k8s.io/peer-pods-webhook-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/peer-pods-webhook-manager-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/peer-pods-webhook-proxy-rolebinding unchanged configmap/peer-pods-webhook-manager-config created service/peer-pods-webhook-controller-manager-metrics-service created service/peer-pods-webhook-webhook-service created deployment.apps/peer-pods-webhook-controller-manager created mutatingwebhookconfiguration.admissionregistration.k8s.io/peer-pods-webhook-mutating-webhook-configuration unchanged Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": EOF Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": EOF make[1]: *** [Makefile:103: deploy] Error 1 make[1]: Leaving directory '/home/zjc/go/src/github.com/confidential-containers/cloud-api-adaptor/src/webhook' make: *** [Makefile:130: deploy] Error 2 ``` ```bash kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE default nginx-676b6c5bbc-98sd9 1/1 Running 0 54m kube-system cilium-8n577 1/1 Running 1 (62m ago) 101m kube-system cilium-envoy-tcqrp 1/1 Running 1 (62m ago) 101m kube-system cilium-operator-b4bfbfd9c-zjhmv 1/1 Running 1 (62m ago) 101m kube-system coredns-7c65d6cfc9-6n592 1/1 Running 1 (62m ago) 107m kube-system coredns-7c65d6cfc9-xgxv7 1/1 Running 1 (62m ago) 107m kube-system etcd-archlinux 1/1 Running 1 (62m ago) 107m kube-system ext-res-updater-q5sgf 1/1 Running 0 32m kube-system kube-apiserver-archlinux 1/1 Running 1 (62m ago) 107m kube-system kube-controller-manager-archlinux 1/1 Running 1 (62m ago) 107m kube-system kube-proxy-hlsx7 1/1 Running 1 (62m ago) 107m kube-system kube-scheduler-archlinux 1/1 Running 1 (62m ago) 107m peer-pods-webhook-system peer-pods-webhook-controller-manager-db564cfcd-tkvsv 0/2 ContainerCreating 0 2m42s peer-pods-webhook-system peer-pods-webhook-controller-manager-db564cfcd-zgf5s 0/2 ContainerCreating 0 2m42s ```

The commands of #3645 were also tried

kubectl delete -A ValidatingWebhookConfiguration cert-manager-webhook
kubectl delete -A mutatingwebhookconfigurations cert-manager-webhook

If I use make deploy to install the CAA by operators, do I need to install the cert-manager by helm manually? If so, how to allow port 10260 for my local cluster?

Thank you for your time.