search

kubernetes-sigs / security-profiles-operator

The Kubernetes Security Profiles Operator
Apache License 2.0
715 stars 107 forks source link

e2e tests are broken due to failing cert-manager certificate creation #220

Closed saschagrunert closed 3 years ago

saschagrunert commented 3 years ago

The e2e tests are currently not able to deploy the operator, because cert-manager seems to fail to create the certificate in kind.

The operator deployment gets stuck in ContainerCreating state:

> k describe pod security-profiles-operator-844b669589-fxvvm
Name:           security-profiles-operator-844b669589-fxvvm
Namespace:      security-profiles-operator
Priority:       0
Node:           spo-e2e-1611320144-control-plane/172.18.0.2
Start Time:     Fri, 22 Jan 2021 14:06:49 +0100
Labels:         app=security-profiles-operator
                name=security-profiles-operator
                pod-template-hash=844b669589
Annotations:    seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/security-profiles-operator-844b669589
Containers:
  security-profiles-operator:
    Container ID:
    Image:         security-profiles-operator:latest
    Image ID:
    Port:          9443/TCP
    Host Port:     0/TCP
    Args:
      manager
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      RELATED_IMAGE_OPERATOR:          security-profiles-operator:latest
      RELATED_IMAGE_NON_ROOT_ENABLER:  bash:5.0
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from security-profiles-operator-token-p4wvt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-server-cert
    Optional:    false
  security-profiles-operator-token-p4wvt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  security-profiles-operator-token-p4wvt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                 node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    17s               default-scheduler  Successfully assigned security-profiles-operator/security-profiles-operator-844b669589-fxvvm to spo-e2e-1611320144-control-plane
  Warning  FailedMount  1s (x6 over 16s)  kubelet            MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found

The cert-manager-cainjector pod reported the following error:

E0122 13:03:23.740344       1 sources.go:114] cert-manager/certificate/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io \"webhook-cert\" not found" "certificate"={"Namespace":"security-profiles-operator","Name":"webhook-cert"} "resource_kind"="MutatingWebhookConfiguration" "resource_name"="spo-mutating-webhook-configuration" "resource_namespace"="" "resource_version"="v1beta1"
I0122 13:03:23.740375       1 controller.go:167] cert-manager/certificate/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="spo-mutating-webhook-configuration" "resource_namespace"="" "resource_version"="v1beta1"

It may be possible that we have to add additional configuration to make cert-manager work in kind.

cc @cmurphy

cmurphy commented 3 years ago

Strange that the tests passed when the change was introduced...

It may be possible that we have to add additional configuration to make cert-manager work in kind.

I've been using kind in my development and never ran into this

saschagrunert commented 3 years ago

Hm, let's close this issue for now and revisit if the issue occurs again /close

k8s-ci-robot commented 3 years ago

@saschagrunert: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/security-profiles-operator/issues/220#issuecomment-766636841): >Hm, let's close this issue for now and revisit if the issue occurs again >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
cmurphy commented 3 years ago

/reopen

This seems to still be popping up, I'll try to dig into it. At a minimum I can split out the webhook deployment so that the e2e tests don't have to deploy cert-manager while we figure it out.

k8s-ci-robot commented 3 years ago

@cmurphy: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/security-profiles-operator/issues/220#issuecomment-767719939): >/reopen > >This seems to still be popping up, I'll try to dig into it. At a minimum I can split out the webhook deployment so that the e2e tests don't have to deploy cert-manager while we figure it out. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.