search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.57k stars 1.39k forks source link

k3d default storage-class can't backup #8123

Open liyin37 opened 3 weeks ago

liyin37 commented 3 weeks ago

we have two clusters: cluster A 

[root@giskg ~]# kubectl version
Client Version: v1.28.8+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8+k3s1

clusterB 

[root@k8s-node1 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.9", GitCommit:"d15213f69952c79b317e635abff6ff4ec81475f8", GitTreeState:"clean", BuildDate:"2023-12-19T13:41:13Z", GoVersion:"go1.20.12", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.9", GitCommit:"d15213f69952c79b317e635abff6ff4ec81475f8", GitTreeState:"clean", BuildDate:"2023-12-19T13:32:15Z", GoVersion:"go1.20.12", Compiler:"gc", Platform:"linux/amd64"}

clusterA backup one namespace，the namespace resource have PVC

[root@giskg ~]# kubectl -n test get all 
NAME          READY   STATUS    RESTARTS       AGE
pod/minio-0   1/1     Running   1 (145m ago)   2d19h

NAME                     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                         AGE
service/minio-headless   ClusterIP   None          <none>        9000/TCP,9001/TCP               2d19h
service/minio            NodePort    10.43.36.40   <none>        9000:30087/TCP,9001:30086/TCP   2d19h

NAME                     READY   AGE
statefulset.apps/minio   1/1     2d19h
[root@giskg ~]# kubectl -n test get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-minio-0   Bound    pvc-fd9ac663-68a4-455c-bdea-057117f43956   100Gi      RWO            local-path     2d19h

the sc name is ： local-path

when I install the verelo ，the version is: 1.14.0

[root@giskg ~]# velero version
Client:
    Version: v1.14.0
    Git commit: 2fc6300f2239f250b40b0488c35feae59520f2d3
<error getting server version: namespaces "velero" not found>
[root@giskg ~]# kubectl -n velero-system get pod|grep -E "node|velero"
node-agent-rbrmm                                                  1/1     Running     2 (147m ago)   3d21h
velero-7887649466-qm2kn                                           1/1     Running     2 (147m ago)   3d21h
node-agent-scbv2                                                  1/1     Running     3 (147m ago)   3d21h
node-agent-vjb5q                                                  1/1     Running     3 (147m ago)   3d21h
node-agent-bqhl5                                                  1/1     Running     2 (147m ago)   3d21h

then,I execute below cmd:

velero debug  backup create test-backup-2024081902 \
--include-cluster-resources=true \
--include-namespaces test \
--default-volumes-to-fs-backup \
--namespace velero-system

check the logs：

time="2024-08-19T04:21:49Z" level=info msg="Summary for skipped PVs: [{\"name\":\"pvc-4300ea91-9064-4dec-aa33-7ac5baaf29d8\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]},{\"name\":\"pvc-728c03ff-1c6f-403b-b06c-aa22346d3287\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]},{\"name\":\"pvc-74b80bab-f044-4a70-897d-1f203ca0f31d\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]},{\"name\":\"pvc-7c774522-e0ea-487d-b679-c593006b79ff\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]},{\"name\":\"pvc-9445a3c5-f197-488f-a2e1-32ca04b4e28f\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]},{\"name\":\"pvc-bff44368-f0c3-403f-9ef3-8ba9dc33c99a\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]},{\"name\":\"pvc-f563f48f-dc63-4964-9972-e069f9db46f0\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]},{\"name\":\"pvc-fd9ac663-68a4-455c-bdea-057117f43956\",\"reasons\":[{\"approach\":\"volumeSnapshot\",\"reason\":\"not satisfy the criteria for VolumePolicy or the legacy snapshot way\"}]}]" backup=velero-system/test-backup-2024081902 logSource="pkg/backup/backup.go:494"
time="2024-08-19T04:21:49Z" level=info msg="Backed up a total of 266 items" backup=velero-system/test-backup-2024081902 logSource="pkg/backup/backup.go:498" progress=

the pvc doesnt backupd

blackpiglet commented 3 weeks ago 
velero debug  backup create test-backup-2024081902 \
--include-cluster-resources=true \
--include-namespaces test \
--default-volumes-to-fs-backup \
--namespace velero-system

This command doesn't look right. The velero debug CLI is used to collect the debug information for backup or restore. The velero backup create is used to create a Velero backup.

Your command mixes them.

not satisfy the criteria for VolumePolicy or the legacy snapshot way This means the backup doesn't backup the volume data in the snapshot way. Please check whether those volumes are backed up by the File-System way.

liyin37 commented 3 weeks ago

@blackpiglet sorry, I writed wrong cmd，right cmd is below:

velero  backup create test-backup-2024081902 \
--include-cluster-resources=true \
--include-namespaces test \
--default-volumes-to-fs-backup \
--namespace velero-system

but the pvc data dont backup succussfully

liyin37 commented 3 weeks ago 
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    local.path.provisioner/selected-node: k8s-node3
    pv.kubernetes.io/provisioned-by: rancher.io/local-path
  creationTimestamp: "2024-08-19T16:20:38Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-89a57302-33a4-426d-ad54-f5c5a259e471
  resourceVersion: "61852"
  uid: b7a99f1a-c11b-4221-8e91-f387e132d894
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 100Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: data-postgis-md-0
    namespace: test
    resourceVersion: "61835"
    uid: 89a57302-33a4-426d-ad54-f5c5a259e471
  hostPath:
    path: /opt/local-path-provisioner/pvc-89a57302-33a4-426d-ad54-f5c5a259e471_test_data-postgis-md-0
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node3
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard
  volumeMode: Filesystem
status:
  phase: Bound
blackpiglet commented 3 weeks ago

Please help collect the debug info by running the command velero debug and uploading the generated tarball.

liyin37 commented 3 weeks ago

bundle-2024-08-20-03-52-23.tar.gz

liyin37 commented 3 weeks ago

@blackpiglet

blackpiglet commented 3 weeks ago

@liyin37 What is the type of the PV? It seems it's HostPath.

liyin37 commented 3 weeks ago

@blackpiglet hostPath，but use rancher.io/local-path storage

blackpiglet commented 2 weeks ago

Please notice the file system backup doesn't support HostPath volumes. https://velero.io/docs/v1.14/file-system-backup/#limitations