search

kubermatic / machine-controller

Apache License 2.0
305 stars 125 forks source link

Can't create MachineDeployment: Post "https://machine-controller-webhook.kube-system.svc:443/machinedeployments?timeout=10s": context deadline exceeded #1730

Closed MagnusR closed 10 months ago

MagnusR commented 10 months ago

Trying to create MachineDeployment with OpenStack provider

kubectl apply -f md.yaml Error from server (InternalError): error when creating "md.yaml": Internal error occurred: failed calling webhook "machinedeployments.machine-controller.kubermatic.io": failed to call webhook: Post "https://machine-controller-webhook.kube-system.svc:443/machinedeployments?timeout=10s": context deadline exceeded

md.yaml

apiVersion: cluster.k8s.io/v1alpha1 kind: MachineDeployment metadata: annotations: cluster.k8s.io/cluster-api-autoscaler-node-group-max-size: "2" cluster.k8s.io/cluster-api-autoscaler-node-group-min-size: "2" k8c.io/operating-system-profile: "" creationTimestamp: null name: kubeone-cluster-pool1 namespace: kube-system spec: minReadySeconds: 0 replicas: 2 selector: matchLabels: workerset: kubeone-cluster-pool1 strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate template: metadata: creationTimestamp: null labels: workerset: kubeone-cluster-pool1 namespace: kube-system spec: metadata: creationTimestamp: null labels: workerset: kubeone-cluster-pool1 providerSpec: value: cloudProvider: openstack cloudProviderSpec: flavor: m1.small image: jammy-server-cloudimg-amd64 network: kubeone-cluster-cluster securityGroups:

==============

kubectl logs machine-controller-webhook-5958595865-n29cs -n kube-system

W1102 11:58:31.378132 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. {"level":"info","time":"2023-11-02T11:58:31.407Z","caller":"manager/plugin.go:98","msg":"Looking for plugin","os":"amzn2","plugin":"machine-controller-userdata-amzn2"} {"level":"info","time":"2023-11-02T11:58:31.407Z","caller":"manager/plugin.go:139","msg":"Found plugin","os":"amzn2","filename":"/usr/local/bin/machine-controller-userdata-amzn2"} {"level":"info","time":"2023-11-02T11:58:31.407Z","caller":"manager/plugin.go:98","msg":"Looking for plugin","os":"centos","plugin":"machine-controller-userdata-centos"} {"level":"info","time":"2023-11-02T11:58:31.407Z","caller":"manager/plugin.go:139","msg":"Found plugin","os":"centos","filename":"/usr/local/bin/machine-controller-userdata-centos"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:98","msg":"Looking for plugin","os":"flatcar","plugin":"machine-controller-userdata-flatcar"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:139","msg":"Found plugin","os":"flatcar","filename":"/usr/local/bin/machine-controller-userdata-flatcar"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:98","msg":"Looking for plugin","os":"rhel","plugin":"machine-controller-userdata-rhel"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:139","msg":"Found plugin","os":"rhel","filename":"/usr/local/bin/machine-controller-userdata-rhel"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:98","msg":"Looking for plugin","os":"ubuntu","plugin":"machine-controller-userdata-ubuntu"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:139","msg":"Found plugin","os":"ubuntu","filename":"/usr/local/bin/machine-controller-userdata-ubuntu"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:98","msg":"Looking for plugin","os":"rockylinux","plugin":"machine-controller-userdata-rockylinux"} {"level":"info","time":"2023-11-02T11:58:31.408Z","caller":"manager/plugin.go:139","msg":"Found plugin","os":"rockylinux","filename":"/usr/local/bin/machine-controller-userdata-rockylinux"} {"level":"info","time":"2023-11-02T11:58:31.409Z","caller":"webhook/main.go:151","msg":"Listening","address":"0.0.0.0:9876"} {"level":"error","time":"2023-11-02T12:00:17.187Z","caller":"admission/admission.go:175","msg":"Failed to write admissionResponse","error":"context canceled"} {"level":"error","time":"2023-11-02T12:01:56.536Z","caller":"admission/admission.go:175","msg":"Failed to write admissionResponse","error":"context canceled"}

kubectl describe pod machine-controller-webhook-5958595865-n29cs -n kube-system

Name: machine-controller-webhook-5958595865-n29cs Namespace: kube-system Priority: 0 Service Account: machine-controller-webhook Node: kubeone-cluster-cp-2/192.168.1.154 Start Time: Thu, 02 Nov 2023 11:58:12 +0000 Labels: app=machine-controller-webhook pod-template-hash=5958595865 Annotations: cni.projectcalico.org/containerID: df9ef2767e246690a3911fb5f45071b170ee17966c2a749c7b942d29b4df777d cni.projectcalico.org/podIP: 10.244.2.3/32 cni.projectcalico.org/podIPs: 10.244.2.3/32 kubeone.k8c.io/cabundle-hash: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 kubeone.k8c.io/credentials-hash: 9fd2e3602e1918ba8aa2c29ec160a28d41fbf8429e127788c2e0c86e29060210 Status: Running IP: 10.244.2.3 IPs: IP: 10.244.2.3 Controlled By: ReplicaSet/machine-controller-webhook-5958595865 Containers: machine-controller-webhook: Container ID: containerd://8e206becd8f0a9fa57da7a2482ac07832216f48df182f075f659592d862ed4c9 Image: quay.io/kubermatic/machine-controller:v1.57.3 Image ID: quay.io/kubermatic/machine-controller@sha256:ac441b435f4c2f95f764f04d7a445119b3b1bfb014f83fd5cfdcd0d98a257317 Port: Host Port: Command: /usr/local/bin/webhook Args: -listen-address=0.0.0.0:9876 -namespace=kube-system -tls-cert-path=/tmp/cert/cert.pem -tls-key-path=/tmp/cert/key.pem -node-external-cloud-provider -use-osm State: Running Started: Thu, 02 Nov 2023 11:58:31 +0000 Ready: True Restart Count: 0 Liveness: http-get https://:9876/healthz delay=5s timeout=1s period=5s #success=1 #failure=3 Readiness: http-get https://:9876/healthz delay=0s timeout=1s period=5s #success=1 #failure=3 Environment: HTTPS_PROXY:
NO_PROXY:
OS_APPLICATION_CREDENTIAL_ID: <set to the key 'OS_APPLICATION_CREDENTIAL_ID' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_APPLICATION_CREDENTIAL_SECRET: <set to the key 'OS_APPLICATION_CREDENTIAL_SECRET' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_AUTH_URL: <set to the key 'OS_AUTH_URL' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_DOMAIN_NAME: <set to the key 'OS_DOMAIN_NAME' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_PASSWORD: <set to the key 'OS_PASSWORD' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_REGION_NAME: <set to the key 'OS_REGION_NAME' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_TENANT_ID: <set to the key 'OS_TENANT_ID' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_TENANT_NAME: <set to the key 'OS_TENANT_NAME' in secret 'kubeone-machine-controller-credentials'> Optional: false OS_USER_NAME: <set to the key 'OS_USER_NAME' in secret 'kubeone-machine-controller-credentials'> Optional: false Mounts: /tmp/cert from machinecontroller-webhook-serving-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7ssph (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: machinecontroller-webhook-serving-cert: Type: Secret (a volume populated by a Secret) SecretName: machinecontroller-webhook-serving-cert Optional: false kube-api-access-7ssph: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/control-plane= Tolerations: CriticalAddonsOnly op=Exists node-role.kubernetes.io/control-plane:NoSchedule op=Exists node-role.kubernetes.io/master:NoSchedule op=Exists node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 5m36s default-scheduler Successfully assigned kube-system/machine-controller-webhook-5958595865-n29cs to kubeone-cluster-cp-2 Normal Pulling 5m35s kubelet Pulling image "quay.io/kubermatic/machine-controller:v1.57.3" Normal Pulled 5m17s kubelet Successfully pulled image "quay.io/kubermatic/machine-controller:v1.57.3" in 17.86819966s (17.868314368s including waiting) Normal Created 5m17s kubelet Created container machine-controller-webhook Normal Started 5m17s kubelet Started container machine-controller-webhook

MagnusR commented 10 months ago

Fixed with MTU setting