multus kubeconfig file not populated correctly after node reboot

What happend: Apiserver IP address was empty in multus kubeconfig file

/etc/cni/net.d/multus.d/multus.kubeconfig (server: https://[])
`apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: https://[]:
    certificate-authority-data: [snipped]==
users:
- name: multus
  user:
    token: "[snipped]=="
contexts:
- name: multus-context
  context:
    cluster: local
    user: multus
current-context: multus-context`

Pods What you expected to happen: Pods were not scheduled and the below errors were appearing in /var/log/messages

May 20 14:13:20 ace-func1-3n1 kubelet[1683]: E0520 14:13:20.399208 1683 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"d02259dca386dfd70b0d6433452e99a880b57e849f79f74993ace14d90e55fb9\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: error loading config file \\\"/etc/cni/net.d/multus.d/multus.kubeconfig\\\": yaml: line 7: mapping values are not allowed in this context\"" pod="kube-system/coredns-58f4964b57-f8wkh" podUID="d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da"

How to reproduce it (as minimally and precisely as possible): Reboot a node in multi-node Kubernetes cluster

Anything else we need to know?:

Environment:

Multus version : ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
Kubernetes version (use kubectl version): Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.1
Primary CNI for Kubernetes cluster: Calico
OS (e.g. from /etc/os-release): ` NAME="Red Hat Enterprise Linux" VERSION="9.3 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.3" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.3 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.3" `

File of '/etc/cni/net.d/'
File of '/etc/cni/multus/net.d'
NetworkAttachment info (use kubectl get net-attach-def -o yaml)
Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml) ` apiVersion: v1 kind: Pod metadata: creationTimestamp: "2024-05-20T09:02:12Z" generateName: kube-multus-ds- labels: app: multus controller-revision-hash: 789c4467b8 name: multus pod-template-generation: "1" tier: node name: kube-multus-ds-hxtkt namespace: kube-system ownerReferences:
- apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: DaemonSet name: kube-multus-ds uid: cc6cd4f8-540a-4593-b099-4be99d0088bd resourceVersion: "2107922" uid: 308c615b-8284-4c79-91f5-9c3ec976e0e6 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms:
  - matchFields:
    - key: metadata.name operator: In values:
    - ace-func1-3n1 containers:
- args:
- --cni-version=0.3.1
- --cni-conf-dir=/host/etc/cni/net.d
- --multus-autoconfig-dir=/host/etc/cni/net.d
- --multus-log-to-stderr=true
- --multus-log-level=verbose command:
- /thin_entrypoint image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imagePullPolicy: IfNotPresent name: kube-multus resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
- mountPath: /host/etc/cni/net.d name: cni
- mountPath: /host/opt/cni/bin name: cnibin
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-5wxv9 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostNetwork: true initContainers:
- args:
- --type
- thin command:
- /install_multus image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imagePullPolicy: IfNotPresent name: install-multus-binary resources: requests: cpu: 10m memory: 15Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
- mountPath: /host/opt/cni/bin mountPropagation: Bidirectional name: cnibin
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-5wxv9 readOnly: true nodeName: ace-func1-3n1 preemptionPolicy: PreemptLowerPriority priority: 2000001000 priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: multus serviceAccountName: multus terminationGracePeriodSeconds: 10 tolerations:
- effect: NoSchedule operator: Exists
- effect: NoExecute operator: Exists
- effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists
- effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists
- effect: NoSchedule key: node.kubernetes.io/disk-pressure operator: Exists
- effect: NoSchedule key: node.kubernetes.io/memory-pressure operator: Exists
- effect: NoSchedule key: node.kubernetes.io/pid-pressure operator: Exists
- effect: NoSchedule key: node.kubernetes.io/unschedulable operator: Exists
- effect: NoSchedule key: node.kubernetes.io/network-unavailable operator: Exists volumes:
- hostPath: path: /etc/cni/net.d type: "" name: cni
- hostPath: path: /opt/cni/bin type: "" name: cnibin
- name: kube-api-access-5wxv9 projected: defaultMode: 420 sources:
  - serviceAccountToken: expirationSeconds: 3607 path: token
  - configMap: items:
    - key: ca.crt path: ca.crt name: kube-root-ca.crt
  - downwardAPI: items:
    - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions:
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:13Z" status: "True" type: PodReadyToStartContainers
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:14Z" status: "True" type: Initialized
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:15Z" status: "True" type: Ready
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:15Z" status: "True" type: ContainersReady
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:12Z" status: "True" type: PodScheduled containerStatuses:
- containerID: containerd://fe1413443a789d09e25900c72cd3cfc2d33a6ccd5308bf75741284952ed58d18 image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760 lastState: {} name: kube-multus ready: true restartCount: 0 started: true state: running: startedAt: "2024-05-20T09:02:14Z" hostIP: 10.52.214.11 hostIPs:
- ip: 10.52.214.11 initContainerStatuses:
- containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3 image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760 lastState: {} name: install-multus-binary ready: true restartCount: 0 started: false state: terminated: containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3 exitCode: 0 finishedAt: "2024-05-20T09:02:13Z" reason: Completed startedAt: "2024-05-20T09:02:13Z" phase: Running podIP: 10.52.214.11 podIPs:
- ip: 10.52.214.11 qosClass: Burstable startTime: "2024-05-20T09:02:12Z" `
Other log outputs (if you use multus logging)

k8snetworkplumbingwg / multus-cni

multus kubeconfig file not populated correctly after node reboot #1283