admission webhook unable to handle the request

[fentas]

I am installing voyager via helm as a subchart

voyager:
  enableAnalytics: false
  ingressClass: voyager
  cloudProvider: baremetal
  rbac:
    create: true
  apiserver:
    # enableValidatingWebhook is used to configure apiserver as ValidationWebhook for Voyager CRDs
    # TODO check out if this is needed?
    enableValidatingWebhook: true
    # enable for k8s 1.11+
    # If true, uses status sub resource for Voyager crds.
    # enableStatusSubresource: true

like this helm upgrade --tls --install deploynmae -f values.yaml --namespace somenamespace mychart/folder

I create voyager ingresses via a service account within different namespaces. But if I enable enableValidatingWebhook I get

Internal error occurred: failed calling admission webhook "admission.voyager.appscode.com": the server is currently unable to handle the request

For my understanding, the webhook is only for validation porpuses?

[tamalsaha]

@fentas, sorry for the slow response. Are you still having this issue?

What is your kubectl version --short? That error usually means that Kubernetes aggregate api server is not setup correctly.

[mcasperson]

I've experienced this issue myself quite a bit in AKS with Kubernetes 1.11.5 and Voyager 8.01. Falling back to Voyager 7.40 resolves the issue.

This error happens in a virgin cluster created with the following scripts.

Create AKS

   az aks create `
      --name "k8s" `
      --generate-ssh-keys `
      --resource-group $resourceGroup `
      --service-principal $clientId`
      --client-secret $password`
      --node-vm-size Standard_A1_v2 `
      --node-count 2 `
      --enable-rbac `
      --kubernetes-version 1.11.5

Install Helm

# https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/aks/kubernetes-helm.md

Set-Content -Path helm-rbac.yaml -Value @"
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
"@
kubectl apply -f helm-rbac.yaml
helm init --upgrade --service-account tiller

At this point installing Voyager 8.01 via helm and deploying an ingress rule results in the failed calling admission webhook "admission.voyager.appscode.com" error.

[mkozjak]

Same here:

Error from server (InternalError): error when creating "ingress.yaml": 
Internal error occurred: failed calling admission webhook "admission.voyager.appscode.com":
the server is currently unable to handle the request

$ kubectl get crd -l app=voyager
NAME                                CREATED AT
certificates.voyager.appscode.com   2019-01-15T20:24:31Z
ingresses.voyager.appscode.com      2019-01-15T20:24:31Z

$ kubectl version --short
Client Version: v1.13.2
Server Version: v1.11.5-gke.5

This one makes the error go away, but I guess it shouldn't really be used:

kubectl delete validatingwebhookconfiguration -l app=voyager

# or install voyager with --enable-validating-webhook=false
# if installing via helm, use --set apiserver.enableValidatingWebhook=false

https://banzaicloud.com/blog/k8s-admission-webhooks

[pawelpabich]

This prevents us from upgrading to 8.01.

[mazzy89]

I was able to reproduce it and at the same time go over this inconvenient.

Try to run with

apiserver:
  # enableValidatingWebhook is used to configure apiserver as ValidationWebhook for Voyager CRDs
  enableValidatingWebhook: true

It will fail by showing that error.

Now run again but this time disabling the validation

apiserver:
  # enableValidatingWebhook is used to configure apiserver as ValidationWebhook for Voyager CRDs
  enableValidatingWebhook: false

It will fail.

Enable again the validation and this time it will pass

[diptadas]

I have tested Voyager v9.0.0 with AKS k8s v1.11.6. Everything working as expected. Here are the steps I followed:

Create azure resource-group and service-principal

$ az group create -l westus -n voyager-test-1203
$ az ad sp create-for-rbac -n voyager-test-1203 --skip-assignment

$ export resourceGroup=voyager-test-1203
$ export clientId=...
$ export password=...

Create cluster and configure kubectl

$ az aks create  \
      --name "voyager-test-1203"  \
      --generate-ssh-keys  \
      --resource-group $resourceGroup  \
      --service-principal $clientId \
      --client-secret $password \
      --node-vm-size Standard_A1_v2  \
      --node-count 2  \
      --enable-rbac  \
      --kubernetes-version 1.11.6

$ az aks get-credentials --resource-group voyager-test-1203 --name voyager-test-1203

Configure helm

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
EOF

$ helm init --service-account tiller --upgrade

Install voyager via helm

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ helm install appscode/voyager --name voyager-operator --version 9.0.0 \
  --namespace kube-system \
  --set cloudProvider=aks

Create test-server

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-server
  namespace: default
  labels:
    app: test-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-server
  template:
    metadata:
      labels:
        app: test-server
    spec:
      containers:
        - name: server
          image: appscode/test-server:2.3
          imagePullPolicy: Always
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          ports:
            - name: http-1
              containerPort: 8080
            - name: http-2
              containerPort: 8989
---
apiVersion: v1
kind: Service
metadata:
  name: test-server
  namespace: default
  labels:
    app: test-server
spec:
  ports:
  - port: 8080
    name: http-1
    protocol: TCP
    targetPort: 8080
  - port: 8989
    name: http-2
    protocol: TCP
    targetPort: 8989
  selector:
    app: test-server
EOF

Create ingress

$ cat <<EOF | kubectl apply -f -
apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
spec:
  rules:
  - host: aa.appscode.test
    http:
      port: 8080
      paths:
      - backend:
          serviceName: test-server
          servicePort: 8080
  - host: bb.appscode.test
    http:
      port: 8989
      paths:
      - backend:
          serviceName: test-server
          servicePort: 8989
EOF

Check response

$ kubectl get service voyager-test-ingress
NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)                         AGE
voyager-test-ingress   LoadBalancer   10.0.90.210   137.117.20.153   8080:31198/TCP,8989:30302/TCP   2m

$ curl -vv -H "Host: aa.appscode.test:8080" 'http://137.117.20.153:8080'
* Rebuilt URL to: http://137.117.20.153:8080/
*   Trying 137.117.20.153...
* TCP_NODELAY set
* Connected to 137.117.20.153 (137.117.20.153) port 8080 (#0)
> GET / HTTP/1.1
> Host: aa.appscode.test:8080
> User-Agent: curl/7.61.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 25 Feb 2019 08:11:31 GMT
< Content-Length: 249
< Content-Type: text/plain; charset=utf-8
< 
{"type":"http","host":"aa.appscode.test:8080","podName":"test-server-68c4d6bf58-9ff5v","serverPort":":8080","path":"/","method":"GET","headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["curl/7.61.0"],"X-Forwarded-For":["10.244.1.1"]}}
* Connection #0 to host 137.117.20.153 left intact

Try to create an invalid ingress

$ cat <<EOF | kubectl apply -f -
apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
spec:
  rules:
  - host: aa.appscode.test
    http:
      port: 8080
      paths:
      - backend:
          serviceName: test-server
          servicePort: 8080
  - host: aa.appscode.test
    http:
      port: 8080
      paths:
      - backend:
          serviceName: test-server
          servicePort: 8080
EOF

Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"voyager.appscode.com/v1beta1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"test-ingress\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"aa.appscode.test\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"test-server\",\"servicePort\":8080}}],\"port\":8080}},{\"host\":\"aa.appscode.test\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"test-server\",\"servicePort\":8080}}],\"port\":8080}}]}}\n"}},"spec":{"rules":[{"host":"aa.appscode.test","http":{"paths":[{"backend":{"serviceName":"test-server","servicePort":8080}}],"port":8080}},{"host":"aa.appscode.test","http":{"paths":[{"backend":{"serviceName":"test-server","servicePort":8080}}],"port":8080}}]}}
to:
Resource: "voyager.appscode.com/v1beta1, Resource=ingresses", GroupVersionKind: "voyager.appscode.com/v1beta1, Kind=Ingress"
Name: "test-ingress", Namespace: "default"
Object: &{map["metadata":map["namespace":"default" "resourceVersion":"6472" "selfLink":"/apis/voyager.appscode.com/v1beta1/namespaces/default/ingresses/test-ingress" "uid":"77098396-38d4-11e9-b30f-3e47562d18b6" "annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"voyager.appscode.com/v1beta1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"test-ingress\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"aa.appscode.test\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"test-server\",\"servicePort\":8080}}],\"port\":8080}},{\"host\":\"bb.appscode.test\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"test-server\",\"servicePort\":8989}}],\"port\":8989}}]}}\n"] "creationTimestamp":"2019-02-25T08:07:59Z" "generation":'\x02' "name":"test-ingress"] "spec":map["rules":[map["host":"aa.appscode.test" "http":map["paths":[map["backend":map["serviceName":"test-server" "servicePort":'\u1f90']]] "port":'\u1f90']] map["host":"bb.appscode.test" "http":map["paths":[map["backend":map["serviceName":"test-server" "servicePort":'\u231d']]] "port":'\u231d']]]] "status":map["loadBalancer":map["ingress":[map["ip":"137.117.20.153"]]]] "apiVersion":"voyager.appscode.com/v1beta1" "kind":"Ingress"]}
for: "STDIN": admission webhook "admission.voyager.appscode.com" denied the request: spec.rules[1].http.paths[0] is reusing path  for addr aa.appscode.test:8080, also used in spec.rules[0].http.paths[0]

[mazzy89]

Still same issue here

2019/03/22 18:41:55 INFO: applying Helmsman labels to [ haproxy ] in namespace [ kube-system ]
2019/03/22 18:42:06 Command returned with exit code: . And error message: Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling admission webhook "admission.voyager.appscode.com": the server is currently unable to handle the request

with version 9.0.0

[tamalsaha]

@mazzy89, are you using AKS? What is your k8s version? What is your AKS version? Which region are you using? If you look at @diptadas comment from above, this worked in our testing. Can you please give us reproducible steps for this issue?

In the mean time, you can disable the webhook via https://github.com/appscode/voyager/blob/master/chart/voyager/values.yaml#L89

[mazzy89]

I'm using helmsman 1.7.2, Helm 2.3.0, K8s 1.11.5 on AWS (no EKS). Region does not affect anyhow a webhook.

[tamalsaha]

How are you provisioning your k8s cluster?

[mazzy89]

kops 1.11.0

[mazzy89]

Also once enableValidatingWebhook: false then I get this error:

2019/03/22 21:47:04 Command returned with exit code: . And error message: Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling admission webhook "admission.voyager.appscode.com": the server could not find the requested resource

[tamalsaha]

Please run the following commands and share the output:

kubectl get validatingwebhookconfiguration
kubectl get mutatingwebhookconfiguration

[mazzy89]

➜ kubectl get validatingwebhookconfiguration
NAME                             AGE
admission.voyager.appscode.com   3h

➜ kubectl get mutatingwebhookconfiguration --all-namespaces
No resources found.

[tamalsaha]

Run

kubectl delete validatingwebhookconfiguration admission.voyager.appscode.com

That should fix the issue.

[mazzy89]

Oh I see where are we heading. I've heard about this. this validationwebhook usually are very bad and just mess around with the cluster. 😞I discourage to use them. they can literally fuck up an entire cluster.

removed and same issue again

2019/03/22 22:05:46 INFO: applying Helmsman labels to [ haproxy ] in namespace [ kube-system ]
2019/03/22 22:05:58 Command returned with exit code: . And error message: Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling admission webhook "admission.voyager.appscode.com": the server is currently unable to handle the request

[mazzy89]

I've solved it removing it and disabling the validation in the Helm config

[tamalsaha]

That should do it.

[mazzy89]

Thank you for the support @tamalsaha. very appreciate it. all solved

[tamalsaha]

Validating webhooks have been generally very useful to protect against various of bad user input. This has been fairly stable in the last 6-9 months (since 1.11 release). We know there is some issue in AKS that causes problem and AKS has been unresponsive to user bug reports. I have not seen any issues in Kops in this regard.

[mazzy89]

Validation webhooks are a pain in kubernetes as indeed this is another case I've seen in the last months. Nothing related with AKS or kops. Simply kubernetes is affected by this.