Waiting for first consumer before binding

[maaft]

Somehow, the volume is not created in my setup. All logs seem to be OK / have no relevant output.

Setup

• k3s cluster with flannel and wireguard-native backend
• agents are a mix of cloud and bare-metal servers (the latter have the required instance.hetzner.cloud/is-root-server=true label set)

I applied this yaml to test PVCs. Note, how I specifically schedule the pod (and thus the PVC) on a cloud server via nodeName

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: hcloud-volumes
---
kind: Pod
apiVersion: v1
metadata:
  name: my-csi-app
spec:
  nodeName: node-that-runs-in-cloud
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
      - mountPath: "/data"
        name: my-csi-volume
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-csi-volume
      persistentVolumeClaim:
        claimName: csi-pvc

The volume is not being created. Both hcloud-csi-controller and hcloud-csi-node print no relevant ouputs / are silent.

Only my k3s logs show repeated:

Jan 02 14:34:41 node-that-runs-in-cloud k3s[1224]: E0102 14:34:41.626531    1224 desired_state_of_world_populator.go:299] "Error processing volume" err="error processing PVC kube-system/csi-pvc: PVC is not bound" pod="kube-system/my-csi-app" volumeName="my-csi-volume"

But I don't think that this is helpful.

kubectl get pods -o wide

NAME                                              READY   STATUS              RESTARTS       AGE     IP               NODE                                       NOMINATED NODE   READINESS GATES
cluster-autoscaler-8458f68856-9qctb               1/1     Running             0              12h     10.42.0.53       cp-node-1                                  <none>           <none>
coredns-597584b69b-nzqwf                          1/1     Running             0              12h     10.42.4.57       node-that-runs-in-cloud                    <none>           <none>
hcloud-cloud-controller-manager-dc59c4b46-fb7cd   1/1     Running             0              12h     159.69.181.217   node-that-runs-in-cloud                    <none>           <none>
hcloud-csi-controller-776cdc4b6d-8pkv7            5/5     Running             0              14m     10.42.4.70       node-that-runs-in-cloud                    <none>           <none>
hcloud-csi-node-5hvvh                             3/3     Running             0              11m     10.42.0.54       cp-node-1                                  <none>           <none>
hcloud-csi-node-cs4qr                             3/3     Running             0              10m     10.42.3.53       cp-node-2                                  <none>           <none>
hcloud-csi-node-dgn5j                             3/3     Running             0              10m     10.42.4.71       node-that-runs-in-cloud                    <none>           <none>
hcloud-csi-node-njbdw                             3/3     Running             0              11m     10.42.1.46       agent-1                                    <none>           <none>
hcloud-csi-node-nk96p                             3/3     Running             0              10m     10.42.2.49       cp-node-3                                  <none>           <none>
helm-install-traefik-kdmlj                        0/1     Completed           0              12h     10.42.4.58       node-that-runs-in-cloud                    <none>           <none>
kured-4fx8b                                       1/1     Running             0              11d     10.42.7.2        bm-worker2                                 <none>           <none>
kured-7xdgj                                       1/1     Running             12 (12h ago)   11d     10.42.0.50       cp-node-1                                  <none>           <none>
kured-dfjg7                                       1/1     Running             0              11d     10.42.5.2        bm-worker1                                 <none>           <none>
kured-lr4f4                                       1/1     Running             12 (11h ago)   11d     10.42.2.48       cp-node-3                                  <none>           <none>
kured-n4jt7                                       1/1     Running             0              11d     10.42.6.2        bm-worker3                                 <none>           <none>
kured-pzvhw                                       1/1     Running             12 (12h ago)   11d     10.42.4.55       node-that-runs-in-cloud                    <none>           <none>
kured-qlvld                                       1/1     Running             12 (12h ago)   11d     10.42.3.49       cp-node-2                                  <none>           <none>
kured-tplm7                                       1/1     Running             12 (12h ago)   11d     10.42.1.44       agent-1                                    <none>           <none>
metrics-server-5c8978b444-5fjt5                   1/1     Running             0              11h     10.42.3.51       cp-node-2                                  <none>           <none>
my-csi-app                                        0/1     ContainerCreating   0              8m45s   <none>           node-that-runs-in-cloud                    <none>           <none>
traefik-55974696bf-gmj7z                          1/1     Running             0              11h     10.42.3.52       cp-node-2                                  <none>           <none>

kubectl descrive pvc csi-pvc

Name:          csi-pvc
Namespace:     kube-system
StorageClass:  hcloud-volumes
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       my-csi-app
Events:
  Type    Reason                Age                  From                         Message
  ----    ------                ----                 ----                         -------
  Normal  WaitForFirstConsumer  104s (x42 over 11m)  persistentvolume-controller  waiting for first consumer to be created before binding

I also tried to set HCLOUD_DEBUG=true for both csi-controller and csi-node to maybe get more log outputs. But logs are still silent about any potential errors...

Any other logs that I need to check?

[apricote]

At first glance this sounds like a problem coming from k3s, the csi-driver is only participating after a PersistentVolume was created.

Can you check the list of persistentvolumes in your cluster, as well as the logs from the k3s server?

[github-actions[bot]]

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.