search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.69k stars 1.4k forks source link

insecureSkipTLSVerify not working for client commands #7547

Open abh opened 7 months ago

abh commented 7 months ago

What steps did you take and what happened:

I have a backup location defined as

    config:
      insecureSkipTLSVerify: "true"
      region: minio
      s3ForcePathStyle: "true"
      s3Url: https://kube-backup-store.tailscale.svc.cluster.local:9000

It validates okay according to velero, but when I start a backup it errors out with this error:

  <error getting backup volume info: Get "https://kube-backup-store.tailscale.svc.cluster.local:9000/velero/backups/bidaily-20240321004728/bidaily-20240321004728-volumeinfo.json.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=G7sHZ6yEbvG0ZUBLGNAe%2F20240321%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20240321T005003Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=f7cf073fd63fffcdf421558e7c2a13fb088838ec477403fb896b739fac3a6e80": tls: failed to verify certificate: x509: certificate is valid for kube-backup-store.ntp.ts.net, not kube-backup-store.tailscale.svc.cluster.local>

What did you expect to happen:

The following information will help us better understand what's going on:

Bundle attached: [removed as it was unclear what was shared and maybe it wasn't relevant]

blackpiglet commented 7 months ago

Suggest to clean your environment before proceeding with backup and restore, because I found both ResticRepository and BackupRespository in the debug bundle. They are not compatible. Please uninstall Velero(velero uninstall. This command will delete the staled Velero CRD.) environment first, then install the Velero.

Please find more information about deprecation of the ResticRepository in the Velero v1.10.0 release note. https://github.com/vmware-tanzu/velero/releases/tag/v1.10.0

abh commented 7 months ago

Thanks @blackpiglet

I had gone through all the upgrade instructions (including for 1.10.0). The restic and backup repositories were going to different targets.

I did as suggested all the same and abandoned the old target in case the upgrade process had left old cruft.

The backups are working now (and I think actually they were before too). I think the issue I ran into was the confusing (but documented!) behavior that the client will talk to the valero / kube API and also talk directly to the object store; and when talking directly to the object store it's not using the "insecure" configuration from the backup location manifest.

Now I understand that, but it's still a pretty terrible UI because for example velero backups logs ... will error out if --insecure-skip-tls-verify is added (because it's not relevant there) but velero backups describe ... requires it for me.

When not specified the output mostly look like things are working except a final confusing error message that I mistook as coming from the server.

blackpiglet commented 7 months ago

velero backups logs ... will error out if --insecure-skip-tls-verify is added (because it's not relevant there)

Could you give more information about this command in your environment? AFAIK, the velero backup log CLI needs to download the backup log file from the backup repository, so, for the --insecure-skip-tls-verify parameter, the velero backup log and velero backup describe CLI should have the same result.

By the way, I agree that it could be confusing different versions of the Velero works slightly differently.