During the data backup process, the current instance lacks relevant parameters for throttling on S3. If a user has a large amount of data, it is easy to trigger throttling on S3 requests.
For large instances that exclusively occupy a bucket, this can affect the instance's own interface access to the S3 interface.
For small instances that share a bucket with other instances, if multiple instances access it simultaneously, it may result in abnormal data access for multiple instances.
Expected Behavior
The backup tool has added a new parameter to control the upper limit of S3 requests. The platform can directly implement throttling based on the results. Currently, the backup tool can only control concurrent parameters during the backup creation process, and the platform cannot assess the resulting volume of S3 requests from these parameters. Once the backup tool adjusts its internal logic in the future, the platform will also be unaware of these changes.
Steps To Reproduce
1. There is an instance with a corresponding large data volume (data backed up from a production instance totals 1.8 TiB, with a total of 444,791 files).
2. Initiate the backup.
3. Throttling on S3 is triggered during the backup process.
https://grafana.op.zillizcloud.com/explore?orgId=1&left=%7B%22datasource%22:%22Loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bpod%3D%5C%22bp-in01-83afda439e722ec-kxqxx3k%5C%22%7D%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D
Current Behavior
During the data backup process, the current instance lacks relevant parameters for throttling on S3. If a user has a large amount of data, it is easy to trigger throttling on S3 requests.
Expected Behavior
The backup tool has added a new parameter to control the upper limit of S3 requests. The platform can directly implement throttling based on the results. Currently, the backup tool can only control concurrent parameters during the backup creation process, and the platform cannot assess the resulting volume of S3 requests from these parameters. Once the backup tool adjusts its internal logic in the future, the platform will also be unaware of these changes.
Steps To Reproduce
Environment
No response
Anything else?
No response