velero backup describe hangs after migration to 1.9.1

[GiuseppeLoRe]

What steps did you take and what happened:

Hi, I have upgraded my velero instance from v.17 to 1.9. Most of it went fine, backup operations are still working. The only main issue is now that when I type

• "velero backup describe --details" or "velero backup logs" it hangs, like if there was something misconfigured relative to the S3 backend. Could someone help in debugging/fixing this?

What did you expect to happen:

Command not hanging.

The following information will help us better understand what's going on:

I have tried to use the "velero debug" command, but it hangs as well at the point where it attempts to get the backup job logs.

~ > velero debug --backup velero-snapshot-daily-7days-20220919050100 2022/09/19 09:25:35 Collecting velero resources in namespace: velero 2022/09/19 09:25:41 Collecting velero deployment logs in namespace: velero 2022/09/19 09:25:49 Collecting log and information for backup: velero-snapshot-daily-7days-20220919050100

Attaching the velero deployment logs and velero backup describe. velero-backup-describe.txt velero-deploy-logs.txt

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

The "velero version" command has also a wrong behaviour.

~ > velero version Client: Version: v1.9.1 Git commit: - Server: Version:

WARNING: the client version does not match the server version. Please update server

~ > velero client config get features features: EnableCSI

~ > kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:27:51Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

The k8s is a bare metal one installed with KubeSpray,

~ > kubectl get nodes NAME STATUS ROLES AGE VERSION fulen-m001.cscs.ch Ready control-plane,master 235d v1.22.5 fulen-m002.cscs.ch Ready control-plane,master 235d v1.22.5 fulen-m003.cscs.ch Ready control-plane,master 235d v1.22.5 fulen-w001.cscs.ch Ready 235d v1.22.5 fulen-w002.cscs.ch Ready 235d v1.22.5 fulen-w003.cscs.ch Ready 235d v1.22.5 fulen-w004.cscs.ch Ready 235d v1.22.5 fulen-w005.cscs.ch Ready 235d v1.22.5 fulen-w006.cscs.ch Ready 235d v1.22.5 fulen-w007.cscs.ch Ready 235d v1.22.5 fulen-w008.cscs.ch Ready 89d v1.22.5 fulen-w009.cscs.ch Ready 89d v1.22.5 fulen-w010.cscs.ch Ready 89d v1.22.5 fulen-w011.cscs.ch Ready 89d v1.22.5

root@fulen-m001:~# cat /etc/os-release NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[ywk253100]

Do you upgrade the CRDs of Velero as well?

[GiuseppeLoRe]

Yes, that's done automatically by the helm chart as I have upgradeCRDs: true Do you know how I can double check if the CRDs upgrade was done correctly?

[ywk253100]

It's very likely that the CRDs aren't upgraded. You can try to get the CRD kubectl get crd downloadrequests.velero.io -o yaml and check whether it contains the subresources https://github.com/vmware-tanzu/velero/blob/release-1.7/config/crd/v1/bases/velero.io_downloadrequests.yaml#L87 section. In v1.9, it shouldn't contain that part

[GiuseppeLoRe]

Hi, I attempted to update the CRDs as described here https://velero.io/docs/v1.9/upgrade-to-1.9/. So: ~ > velero install --crds-only --dry-run -o yaml | kubectl apply -f - customresourcedefinition.apiextensions.k8s.io/backups.velero.io configured customresourcedefinition.apiextensions.k8s.io/backupstoragelocations.velero.io configured customresourcedefinition.apiextensions.k8s.io/deletebackuprequests.velero.io configured customresourcedefinition.apiextensions.k8s.io/downloadrequests.velero.io configured customresourcedefinition.apiextensions.k8s.io/podvolumebackups.velero.io configured customresourcedefinition.apiextensions.k8s.io/podvolumerestores.velero.io configured customresourcedefinition.apiextensions.k8s.io/resticrepositories.velero.io configured customresourcedefinition.apiextensions.k8s.io/restores.velero.io configured customresourcedefinition.apiextensions.k8s.io/schedules.velero.io configured customresourcedefinition.apiextensions.k8s.io/serverstatusrequests.velero.io configured customresourcedefinition.apiextensions.k8s.io/volumesnapshotlocations.velero.io configured But still, the CRDs don't seem to get updated: ~ > kubectl get crd downloadrequests.velero.io -o yaml |grep subresources {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.7.0"},"creationTimestamp":null,"labels":{"app.kubernetes.io/name":"velero","component":"velero"},"name":"downloadrequests.velero.io"},"spec":{"group":"velero.io","names":{"kind":"DownloadRequest","listKind":"DownloadRequestList","plural":"downloadrequests","singular":"downloadrequest"},"scope":"Namespaced","versions":[{"name":"v1","schema":{"openAPIV3Schema":{"description":"DownloadRequest is a request to download an artifact from backup object storage, such as a backup log file.","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"DownloadRequestSpec is the specification for a download request.","properties":{"target":{"description":"Target is what to download (e.g. logs for a backup).","properties":{"kind":{"description":"Kind is the type of file to download.","enum":["BackupLog","BackupContents","BackupVolumeSnapshots","BackupItemSnapshots","BackupResourceList","RestoreLog","RestoreResults"],"type":"string"},"name":{"description":"Name is the name of the kubernetes resource with which the file is associated.","type":"string"}},"required":["kind","name"],"type":"object"}},"required":["target"],"type":"object"},"status":{"description":"DownloadRequestStatus is the current status of a DownloadRequest.","properties":{"downloadURL":{"description":"DownloadURL contains the pre-signed URL for the target file.","type":"string"},"expiration":{"description":"Expiration is when this DownloadRequest expires and can be deleted by the system.","format":"date-time","nullable":true,"type":"string"},"phase":{"description":"Phase is the current state of the DownloadRequest.","enum":["New","Processed"],"type":"string"}},"type":"object"}},"type":"object"}},"served":true,"storage":true,"subresources":{"status":{}}}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} subresources: The subresources part is still there.

[ywk253100]

Could you double-check the version of the Velero CLI you use to do the CRD upgrade?

[GiuseppeLoRe]

Installed with brew on a macos:

~ > velero version --client-only Client: Version: v1.9.1 Git commit: -

[ywk253100]

Please download it in the release page

[GiuseppeLoRe]

Hi, I did, but the problem was not that. We manage the velero deployment via ArgoCD, I did upgrade the velero/aws-pligin/csi-plugin image version, but unfortunately I forgot to upgrade the helm chart version. After the velero upgrade, ArgoCD was rolling back to the previous CDRs.... Sorry for the noise.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closing the stale issue.