Backup data upload fails with `data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/lib/kubelet: no such file or directory`

[e3b0c442]

What happened: A clear and concise description of what the bug is.

When attempting to back up a VM with the CSI snapshot data mover, the data upload fails with the error data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/lib/kubelet: no such file or directory

What you expected to happen: A clear and concise description of what you expected to happen.

The backup completes successfully

How to reproduce it (as minimally and precisely as possible): Steps to reproduce the behavior.

1. Create a VM
2. Install velero with the EnableCSI feature flag, and the kubevirt-velero-plugin and velero-plugin-for-aws plugins enabled. (In my case, all are latest [velero 1.14.0, kubevirt-velero-plugin 0.7.0, velero-plugin-for-aws 1.10.0]. I also tried with velero 1.13 and the appropriate plugin versions with no change in behavior)

Additional context: While attempting to troubleshoot this issue, I created a pod with identical mounts to the velero node-agent pod on the node containing the VM, including volumeMount /host_pods which mount's the host's /var/lib/kubelet folder. Investigating the volume's contents, I found a file trying to link back to /var/lib/kubelet, which I think is the root of this failure, as the path /var/lib/kubelet doesn't exist in the node-agent pod.

# ls -alR /host_pods/POD_UUID
# ...
./volumeDevices/kubernetes.io~csi:
total 0
drwxr-x--- 2 root root  54 Jul  1 20:48 .
drwxr-x--- 3 root root  31 Jul  1 20:48 ..
lrwxrwxrwx 1 root root 142 Jul  1 20:48 pvc-eecd1d33-cefd-42bf-a99b-c45f6cae5759 -> /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-eecd1d33-cefd-42bf-a99b-c45f6cae5759/88e93be8-befb-4a7f-a670-1a87b081aedb
# ...

I am using the minimalist Talos Linux, though I don't think that should have any special bearing here.

I can provide Velero configs as needed but there is very little modified from default.

Environment:

• KubeVirt version (use virtctl version):

  Client Version: version.Info{GitVersion:"1.2.2", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.22.4", Compiler:"gc", Platform:"darwin/arm64"}
  Server Version: version.Info{GitVersion:"v1.2.2", GitCommit:"1466b658f78b9b8bb9517ffb6dafd4b777f33fe6", GitTreeState:"clean", BuildDate:"2024-06-06T05:46:26Z", GoVersion:"go1.21.8 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes version (use kubectl version): N/A

  Client Version: v1.30.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.29.6

• VM or VMI specifications: N/A

  apiVersion: kubevirt.io/v1
  kind: VirtualMachine
  metadata:
  annotations:
  kubevirt.io/latest-observed-api-version: v1
  kubevirt.io/storage-observed-api-version: v1
  creationTimestamp: "2024-06-26T14:25:42Z"
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  generation: 8
  labels:
  app.kubernetes.io/instance: vms
  kubevirt.io/vm: one
  name: one
  namespace: vms
  resourceVersion: "5517501"
  uid: 5e880baa-accd-40c8-8276-3f6f862a2987
  spec:
  dataVolumeTemplates:
  - metadata:
    creationTimestamp: null
    labels:
      kubevirt.io/vm: one
    name: one-rootdisk
    namespace: vms
  spec:
    pvc:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 40Gi
      storageClassName: ceph-block
      volumeMode: Block
    source:
      pvc:
        name: fedora-40-amd64
        namespace: vms
  instancetype:
  kind: virtualmachineclusterinstancetype
  name: o1.large
  revisionName: one-o1.large-3ae509a7-5446-4119-945d-1eb2a00d00cb-1
  preference:
  kind: virtualmachineclusterpreference
  name: fedora
  revisionName: one-fedora-f2c9d38a-8f77-44a4-bdd6-108ac0282713-1
  runStrategy: Always
  template:
  metadata:
    creationTimestamp: null
    labels:
      kubevirt.io/vm: one
  spec:
    architecture: amd64
    domain:
      cpu:
        model: Westmere
      devices:
        disks:
        - name: rootdisk
        - name: cloudinitdisk
        interfaces:
        - bridge: {}
          macAddress: 02:e3:b0:00:00:01
          name: multus
      machine:
        type: q35
      resources: {}
    evictionStrategy: LiveMigrate
    networks:
    - multus:
        networkName: vms/vlan-176
      name: multus
    terminationGracePeriodSeconds: 0
    volumes:
    - dataVolume:
        name: one-rootdisk
      name: rootdisk
    - cloudInitNoCloud:
        userData: |-
          #cloud-config
          users:
            - name: nick
              groups: wheel
              passwd: REDACTED
              lock_passwd: false
              ssh_authorized_keys:
                - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILG8LVJNXtypT9BARc1NKKB4E0mMZB4nIjSeoeNYufJo
          growpart:
            mode: auto
      name: cloudinitdisk

• Cloud provider or hardware configuration: on-prem bare metal, various configurations. Shouldn't be important here.
• OS (e.g. from /etc/os-release):

  NAME="Talos"
  ID=talos
  VERSION_ID=v1.7.5
  PRETTY_NAME="Talos (v1.7.5)"
  HOME_URL="https://www.talos.dev/"
  BUG_REPORT_URL="https://github.com/siderolabs/talos/issues"
  VENDOR_NAME="Sidero Labs"
  VENDOR_URL="https://www.siderolabs.com/"

• Kernel (e.g. uname -a): Linux node-agent-debug 6.6.33-talos #1 SMP Tue Jun 18 14:02:42 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: talosctl
• Others: N/A

[e3b0c442]

Adding an extra volume mount to mount /var/lib/kubelet in the pod at the same location as the host allowed the backup to succeed, confirming my hypothesis. This doesn't seem like it should be normal practice though.

[e3b0c442]

Closing this issue as it is not specifically related to KubeVirt or kubevirt-velero-plugin; this issue is present on any pod with a Block mode PVC attached via volumeDevices spec.