Can't register any broker, the webhook request times out

[jiayihu]

Bug Report

What happened: I've installed the catalog chart but I can't register any broker. I've tried with both a custom broker and ups-broker. This is the error no matter how I try it (both with svcat and YAML)

Error: register request failed (Internal error occurred: failed calling webhook "mutating.clusterservicebrokers.servicecatalog.k8s.io": Post "https://catalog-catalog-webhook.catalog.svc:443/mutating-clusterservicebrokers?timeout=30s": context deadline exceeded)

What you expected to happen: Being able to register the broker

How to reproduce it (as minimally and precisely as possible):

helm install catalog ./catalog-0.3.1.tgz --namespace catalog
helm install ups-broker ./ups-broker-0.3.1.tgz --namespace ups-broker
svcat register ups-broker --url http://ups-broker-ups-broker.ups-broker --scope cluster

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.20 via k3s versoin v1.20.0+k3s2

• service-catalog version: 0.3.1

• Cloud provider or hardware configuration:

• Do you have api aggregation enabled?

  • Do you see the configmap in kube-system? Yes, I can see extension-apiserver-authentication
  • Does it have all the necessary fields? Yes
  • kubectl get cm -n kube-system extension-apiserver-authentication -o yaml and look for requestheader-XXX fields
  • This is what I get:

  • apiVersion: v1
    data:
      client-ca-file: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      requestheader-allowed-names: '["system:auth-proxy"]'
      requestheader-client-ca-file: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      requestheader-extra-headers-prefix: '["X-Remote-Extra-"]'
      requestheader-group-headers: '["X-Remote-Group"]'
      requestheader-username-headers: '["X-Remote-User"]'
    kind: ConfigMap
    metadata:
      creationTimestamp: "2020-12-31T19:24:17Z"
      managedFields:
      - apiVersion: v1
        fieldsType: FieldsV1
        fieldsV1:
          f:data:
            .: {}
            f:client-ca-file: {}
            f:requestheader-allowed-names: {}
            f:requestheader-client-ca-file: {}
            f:requestheader-extra-headers-prefix: {}
            f:requestheader-group-headers: {}
            f:requestheader-username-headers: {}
        manager: k3s
        operation: Update
        time: "2020-12-31T19:24:17Z"
      name: extension-apiserver-authentication
      namespace: kube-system
      resourceVersion: "28"
      uid: a96a8b17-1665-42e5-af65-e25508e8c857

• Install tools:

  • Did you use helm? What were the helm arguments? Did you --set any extra values? I've changed the image urls adding -arm64 in values.yml, directly inside the chart package.

  • helm install catalog ./catalog-0.3.1.tgz --namespace catalog
    helm install ups-broker ./ups-broker-0.3.1.tgz --namespace ups-broker
    svcat register ups-broker --url http://ups-broker-ups-broker.ups-broker --scope cluster

• Are you trying to use ALPHA features? Did you enable them? No

[jiayihu]

I have checked the broker ups-broker is live and reachable via DNS from the catalog namespace. For instance I can k exec -ti dnsutils -n catalog -- wget --header="X-Broker-API-Version: 2.16" -O - http://ups-broker-ups-broker.ups-broker/v2/catalog

To be precise, from my custom OSB I can see that the actually no request is even reaching the broker.

The webhook and controller seem also to be live and fine. I'll attach the logs just as info.

webhook.log controller.log

[jiayihu]

Okay I was able to solve the issue, it was an issue in my node hosting the catalog. I noticed the node wasn't able to send back the responses to the master because it was missing the ip route to handle packets with master's IP. The flannel.1 interface was missing for the master IP.

These are the steps I've taken to debug. Your issue might be different, but I hope this helps:

• Check if the pods are running okay, both using kubectl and reading the logs
• Check if the services are reachable by DNS from other pods in the namespace. Dump TCP packets flowing through the network device responsible for the k8s networking. In my case it's flannel and I dumped all TCP packets going in/out flannel.1. You can filter by source IP or host IP. You should see TCP packets going in/out when a request reaches the catalog service.
• Check if the master can reach the catalog service by DNS or by using the service ClusterIP.

In my case I noticed that the master was able to send the packet to the service but I only saw Start packets, without ACK or anything going back to the master. That's when I got suspicious and I noticed the missing network interface for the master. Re-installing the cluster and rebooting solved the issue.