VPA E2E tests are failing

[jbartosik]

VPA e2e tests are consistently failing. They were consistently passing (flake once every few days) until recently.

For full suite:

• Last green run was on 2021-12-20 at 16:15 CET prow link,
• First red run was on 2021-12-20 at 18:15 CET prow link.

For admission-controller:

• Last green run was on 2021-12-20 at 16:14 CET prow link,
• First red run was on 2021-12-20 at 18:14 CET prow link.

For actuation:

• Last green run was on 2021-12-20 at 16:14 CET prow link,
• First red run was on 2021-12-20 at 18:14 CET prow link.

It looks like admission-controller is broken. I suspect that the culprit is #4537

[jbartosik]

4537 merged on 2021-12-20 17:02 CET and affects admission controller, that's why I suspect it causes the problem.

[jbartosik]

When I run e2e tests in my cluster from current master in logs of API server I see logs like:

"failed calling webhook "vpa.k8s.io": expected webhook response of admission.k8s.io/v1, Kind=AdmissionReview, got /, Kind="

[adriananeci]

What do you have configured for admissionReviewVersions in vpa webhook in your cluster? Was the VPA webhook manually created and vpa admission-controller started with--register-webhook=False ?

[adriananeci]

Is there a way I can run the E2E tests suite locally, on my laptop?

[jbartosik]

Is there a way I can run the E2E tests suite locally, on my laptop?

I don't know. I create a cluster and use ./hack/deploy-for-e2e.sh admission-controller to deploy admission controller from what's in my repo, then ./hack/run-e2e-tests.sh admission-controller to run e2e test.

[jbartosik]

What do you have configured for admissionReviewVersions in vpa webhook in your cluster? Was the VPA webhook manually created and vpa admission-controller started with--register-webhook=False ?

$ kubectl get MutatingWebhookConfiguration vpa-webhook-config -oyaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: {{scrubbed}}
  generation: 1
  name: vpa-webhook-config
  resourceVersion: {{scrubbed}}
  uid: {{scrubbed}}
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: {{scrubbed}}
    service:
      name: vpa-webhook
      namespace: kube-system
      port: 443
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: vpa.k8s.io
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  - apiGroups:
    - autoscaling.k8s.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - verticalpodautoscalers
    scope: '*'
  sideEffects: None
  timeoutSeconds: 30