search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.64k stars 1.39k forks source link

backup cannot be synchronized normally #8040

Closed ShyunnY closed 2 months ago

ShyunnY commented 2 months ago

What steps did you take and what happened: I am a novice using velero. I followed the steps below:

What did you expect to happen:

I don't know what caused my Backup to be lost, which makes me very upset. I saw in the documentation: Velero's Backup will be synchronized with S3, but I don't know why Velero doesn't do this. I hope to be able to back up the data normally.

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

ShyunnY commented 2 months ago

One more thing: I found that Backup was deleted in about 30s

blackpiglet commented 2 months ago

There are several possible reasons:

ShyunnY commented 2 months ago

There are several possible reasons:

  • Please check whether any GitOps is governing your k8s environment. It's possible the generated backup was deleted by the GitOps tools
  • If you use Velero with an on-premise Object Storage service, please check whether the BackupStorageLocation's s3ForcePathStyle parameter is set to true. https://velero.io/docs/v1.14/contributions/minio/#set-up-server

Thanks for your reply! Unfortunately, I have tried both solutions above and it seems that the problem cannot be solved.

I set up minio in my local cluster and tried the quick-start example. This is normal, and my Backup runs as expected. I don't know what happened, but I'm eager to solve this problem.

In addition, the s3 I use is Qiniu Cloud Kodo storage

blackpiglet commented 2 months ago

Let me sum up the information:

IMO, this is dedicated to Qiniu Kodo storage. Does the Kodo storage support using the path-style URL? https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html

ShyunnY commented 2 months ago

@blackpiglet

yeah, we can see from the documentation that it does support this feature: docs

blackpiglet commented 2 months ago

This issue should be caused by the Velero backup sync controller cannot read the backup from the object storage. Please set the Velero server's log level to debug, then check whether there is such log printed.

    spec:
      containers:
      - args:
        - server
        - --features=EnableCSI
        - --uploader-type=kopia
        - --log-level=debug
Deleted orphaned backup from cluster

And what is the version of the AWS plugin you are using?

ShyunnY commented 2 months ago

Thanks :)

I solved this problem. After carefully analyzing the workflow of BackupController, I found that: Since BackupController has a cycle (seemingly called the synchronization cycle) to obtain the Backup in S3 storage, and since BackupController cannot obtain the corresponding resources from S3 within a cycle, the Backup CR is deleted during reconcile.

Based on this idea: I finally found that it was because my S3 URL was not a complete PathStyle style, so BackupController could not obtain the Backup resource as expected. (But I was confused, even if the S3 URL was wrong, the Backup could still be stored in S3)

blackpiglet commented 2 months ago

Upload object is different from download. Please check my analysis on another issue related to this topic. https://github.com/vmware-tanzu/velero/issues/7268#issuecomment-1878338996