search

vmware-tanzu / velero-plugin-for-vsphere

Plugin to support Velero on vSphere
Other
59 stars 49 forks source link

Velero install failed with the latest dev build of plugin on k8s provisioned by tkgm #398

Closed ywk253100 closed 1 year ago

ywk253100 commented 3 years ago

Trying to install velero with the plugin dpcpinternal/velero-plugin-for-vsphere:main-1e7aa17-22.Sep.2021.20.19.31 on the Kubernetes provisioned by tkgm 1.3.1, the backup-driver and datamgr-for-vsphere-plugin cannot start up

yinw@yinw-a02 ci % k --kubeconfig ~/Downloads/vsphere-kubeconfig get pod -n velero
NAME                               READY   STATUS             RESTARTS   AGE
backup-driver-7c88cb7f44-smkhh     0/1     CrashLoopBackOff   5          5m31s
datamgr-for-vsphere-plugin-d9cxv   0/1     CrashLoopBackOff   5          4m31s
datamgr-for-vsphere-plugin-hj6z8   0/1     CrashLoopBackOff   5          4m31s
datamgr-for-vsphere-plugin-smcrn   0/1     CrashLoopBackOff   5          4m31s
velero-55454fbc59-6pjvf            0/1     Init:1/2           1          5m48s
yinw@yinw-a02 ci % k --kubeconfig ~/Downloads/vsphere-kubeconfig logs backup-driver-7c88cb7f44-smkhh -n velero
time="2021-09-24T03:30:50Z" level=info msg="Starting backup-driver server main-1e7aa17-22.Sep.2021.20.19.31 (1e7aa17a6a442c6bb08a8acc6f3d0b98ee7ccca2)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:116"
time="2021-09-24T03:30:50Z" level=info msg="backup-driver server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:191"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
time="2021-09-24T03:30:50Z" level=info msg="backup-driver detected cluster flavor: vSphere Kubernetes Cluster" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:245"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
time="2021-09-24T03:30:50Z" level=error msg="Unable to find installed CSI drivers while retrieving VC configuration Secret" error="Expected CSI driver images not found" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1251" error.function=github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils.GetCSIVersionFromImage logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:105"
time="2021-09-24T03:30:50Z" level=error msg="Could not retrieve vsphere credential from k8s secret" error="Expected CSI driver images not found" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1251" error.function=github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils.GetCSIVersionFromImage logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:117"
An error occurred: Expected CSI driver images not found
yinw@yinw-a02 ci % k --kubeconfig ~/Downloads/vsphere-kubeconfig logs datamgr-for-vsphere-plugin-d9cxv -n velero
time="2021-09-24T03:32:48Z" level=info msg="Starting data manager server main-1e7aa17-22.Sep.2021.20.19.31 (1e7aa17a6a442c6bb08a8acc6f3d0b98ee7ccca2)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:111"
time="2021-09-24T03:32:48Z" level=info msg="data manager server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:214"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
time="2021-09-24T03:32:48Z" level=info msg="data-manager detected cluster flavor: vSphere Kubernetes Cluster" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:238"
time="2021-09-24T03:32:48Z" level=info msg="VC Configuration will be retrieved from cluster secret" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:242"
time="2021-09-24T03:32:48Z" level=info msg="No customized config map for vddk exists" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1336"
time="2021-09-24T03:32:48Z" level=info msg="Successfully retrieved vddk log level" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:256"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
time="2021-09-24T03:32:48Z" level=error msg="Unable to find installed CSI drivers while retrieving VC configuration Secret" error="Expected CSI driver images not found" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1251" error.function=github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils.GetCSIVersionFromImage logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:105"
An error occurred: Expected CSI driver images not found
time="2021-09-24T03:32:48Z" level=error msg="Could not retrieve vsphere credential from k8s secret" error="Expected CSI driver images not found" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1251" error.function=github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils.GetCSIVersionFromImage logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:117"

After creating the configamp, it is still failed

apiVersion: v1
kind: ConfigMap
metadata:
  name: velero-vsphere-plugin-config
data:
  cluster_flavor: "VANILLA"
yinw@yinw-a02 ci % k --kubeconfig ~/Downloads/vsphere-kubeconfig logs backup-driver-7c88cb7f44-cd8zd -n velero
time="2021-09-24T03:54:27Z" level=info msg="Starting backup-driver server main-1e7aa17-22.Sep.2021.20.19.31 (1e7aa17a6a442c6bb08a8acc6f3d0b98ee7ccca2)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:116"
time="2021-09-24T03:54:27Z" level=info msg="backup-driver server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:191"
time="2021-09-24T03:54:27Z" level=info msg="backup-driver detected cluster flavor: vSphere Kubernetes Cluster" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:245"
An error occurred: Expected CSI driver images not found
time="2021-09-24T03:54:27Z" level=error msg="Unable to find installed CSI drivers while retrieving VC configuration Secret" error="Expected CSI driver images not found" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1251" error.function=github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils.GetCSIVersionFromImage logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:105"
time="2021-09-24T03:54:27Z" level=error msg="Could not retrieve vsphere credential from k8s secret" error="Expected CSI driver images not found" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1251" error.function=github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils.GetCSIVersionFromImage logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:117"
xing-yang commented 3 years ago

Have you created the secret?

deepakkinni commented 3 years ago

Assuming velero is installed on "velero" namespace please follow the instructions precisely:

Steps:

  1. Bring down velero, backup-driver, data-manager
kubectl -n velero delete deploy/backup-driver
kubectl -n velero delete daemonset datamgr-for-vsphere-plugin
kubectl -n velero scale deploy/velero --replicas=0
  1. Create VC credential Secret Note: You can retrieve these values from Secret vsphere-config-secret in vmware-system-csi namespace for 2.3 vSphere CSI driver OR Secret vsphere-config-secret in kube-system namespace for 2.2 vSphere CSI driver 
    
dkinni@dkinni-a02 ~ % cat csi-vsphere.conf 
[Global]
cluster-id = "cluster1"

[VirtualCenter "10.182.1.133"] user = "Administrator@vsphere.local" password = "password" port = "443"

kubectl -n velero create secret generic velero-vsphere-config-secret --from-file=csi-vsphere.conf


3. Create the velero-vsphere-plugin-config

apiVersion: v1 kind: ConfigMap metadata: name: velero-vsphere-plugin-config namespace: velero data: cluster_flavor: VANILLA vsphere_secret_name: velero-vsphere-config-secret vsphere_secret_namespace: velero

kubectl apply -f velero-vsphere-plugin-config.yaml


4. Edit velero-vsphere-plugin-feature-states configmap to enable feature "decouple-vsphere-csi-driver"

dkinni@dkinni-a02 ~ % kubectl -n velero get cm velero-vsphere-plugin-feature-states -o yaml apiVersion: v1 data: decouple-vsphere-csi-driver: "true" local-mode: "false" kind: ConfigMap metadata: name: velero-vsphere-plugin-feature-states namespace: velero


5. scale up the velero deployment

kubectl -n velero scale deploy/velero --replicas=1

xing-yang commented 2 years ago

@ywk253100 This issue should be fixed by v1.3.0. Please verify.

ywk253100 commented 2 years ago

@xing-yang Still get the same error with vSphere plugin 1.3.0

yinw@yinw-a02 velero-develop-tool % k --kubeconfig ~/Downloads/vsphere-kubeconfig get pod -n velero
NAME                               READY   STATUS             RESTARTS   AGE
backup-driver-7469b784db-6cdnh     0/1     CrashLoopBackOff   3          87s
datamgr-for-vsphere-plugin-4cmzd   0/1     CrashLoopBackOff   1          26s
datamgr-for-vsphere-plugin-t74mj   0/1     CrashLoopBackOff   1          26s
datamgr-for-vsphere-plugin-zdmtq   0/1     CrashLoopBackOff   1          26s
restic-k8drq                       1/1     Running            0          2m5s
restic-vwccq                       1/1     Running            0          2m5s
restic-zg2l7                       1/1     Running            0          2m5s
velero-9995f6ddd-rt5bj             0/1     Init:1/2           0          2m5s
yinw@yinw-a02 velero-develop-tool % k --kubeconfig ~/Downloads/vsphere-kubeconfig version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4+vmware.1", GitCommit:"d475bbd9e7cd66c6db7069cb447766daada65e3b", GitTreeState:"clean", BuildDate:"2021-02-22T22:15:46Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
yinw@yinw-a02 velero-develop-tool % k --kubeconfig ~/Downloads/vsphere-kubeconfig logs backup-driver-7469b784db-6cdnh -n velero
time="2021-11-23T01:43:33Z" level=info msg="Starting backup-driver server v1.3.0 (179348de0db72a0a62649a16309efa296f368f37)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:116"
time="2021-11-23T01:43:33Z" level=info msg="backup-driver server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:191"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
time="2021-11-23T01:43:33Z" level=info msg="backup-driver detected cluster flavor: vSphere Kubernetes Cluster" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/backupdriver/cli/server/server.go:245"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
Failed to retrieve velero-vsphere-plugin-config configuration, using defaults, err: configmaps "velero-vsphere-plugin-config" not found.
An error occurred: secrets "velero-vsphere-config-secret" not found
time="2021-11-23T01:43:33Z" level=info msg="Skipping k8s secret velero-vsphere-config-secret as it does not exist" error="secrets \"velero-vsphere-config-secret\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:125"
time="2021-11-23T01:43:33Z" level=error msg="failed to retrieve vSphere credentials Secret, [velero-vsphere-config-secret]" error="secrets \"velero-vsphere-config-secret\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:130"
time="2021-11-23T01:43:33Z" level=error msg="Could not retrieve vsphere credential from k8s secret" error="secrets \"velero-vsphere-config-secret\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:117"
yinw@yinw-a02 velero-develop-tool % k --kubeconfig ~/Downloads/vsphere-kubeconfig logs datamgr-for-vsphere-plugin-4cmzd -n velero
time="2021-11-23T01:43:44Z" level=info msg="Starting data manager server v1.3.0 (179348de0db72a0a62649a16309efa296f368f37)" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:111"
time="2021-11-23T01:43:44Z" level=info msg="data manager server is started" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:214"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
time="2021-11-23T01:43:44Z" level=info msg="data-manager detected cluster flavor: vSphere Kubernetes Cluster" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:238"
time="2021-11-23T01:43:44Z" level=info msg="VC Configuration will be retrieved from cluster secret" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:242"
time="2021-11-23T01:43:44Z" level=info msg="No customized config map for vddk exists" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:1336"
time="2021-11-23T01:43:44Z" level=info msg="Successfully retrieved vddk log level" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/cmd/datamgr/cli/server/server.go:256"
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment
Failed to retrieve velero-vsphere-plugin-config configuration, using defaults, err: configmaps "velero-vsphere-plugin-config" not found.
time="2021-11-23T01:43:44Z" level=info msg="Skipping k8s secret velero-vsphere-config-secret as it does not exist" error="secrets \"velero-vsphere-config-secret\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:125"
time="2021-11-23T01:43:44Z" level=error msg="failed to retrieve vSphere credentials Secret, [velero-vsphere-config-secret]" error="secrets \"velero-vsphere-config-secret\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/utils/utils.go:130"
time="2021-11-23T01:43:44Z" level=error msg="Could not retrieve vsphere credential from k8s secret" error="secrets \"velero-vsphere-config-secret\" not found" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/snapshotmgr/snapshot_manager.go:117"
An error occurred: secrets "velero-vsphere-config-secret" not found
xing-yang commented 2 years ago

@ywk253100, you need to create a Configmap. See details in the PR description section here: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/pull/391

xing-yang commented 2 years ago

I also opened a PR for @deepakkinni to update the install doc in Github: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/issues/418

deepakkinni commented 2 years ago

Follow the same instructions as https://github.com/vmware-tanzu/velero-plugin-for-vsphere/issues/398#issuecomment-926914563 , will be updating the official documentation shortly.