search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.4k stars 1.37k forks source link

Duplicated backup files are increasing the memory usage #7889

Open ricardomiguel-os opened 2 weeks ago

ricardomiguel-os commented 2 weeks ago

Describe the problem/challenge you have

Velero supports API group versions and for backward compatibility purposes, the backup files are being duplicated - 1 folder with version and another without version.

Here you can find that duplication.

Do you know if this duplication can be removed? The backup file size is increasing and the Velero pods are consuming a lot of memory on big clusters, triggering an OOMKilled when not configured with enough resources.

Describe the solution you'd like

Remove this condition or add it behind a feature flag.

Anything else you would like to add:

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

Lyndon-Li commented 2 weeks ago

Could you provide below information for your case?

  1. The number of resources to be backed up in your cluster
  2. What is the backup size
  3. How much memory is consumed for your backup
ricardomiguel-os commented 1 week ago

Hi @Lyndon-Li , yes sure.

  1. 66776 items
  2. 151,8 MB zipped and 1,32 GB unzipped
  3. At least 2GB, its what we have configured for the limits and then its killed

For now we will increase the resources requests and limit

Lyndon-Li commented 1 week ago

I've opened another issue for memory usage optimization in this case, see #7902

Lyndon-Li commented 1 week ago

The behavior for preferred version has been there for a long time, so we will not change it unless there are many requirements. On the other hand, we will work on this issue on the original cause --- when there are many resources, the memory usage of velero backup looks unnecessarily high.

ricardomiguel-os commented 1 week ago

Hi @Lyndon-Li, thanks for the update. But do you think we still need those duplicated files for backward compatibility since this was added more than one year ago?