peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.72k stars 243 forks source link

command/rm: introduce --max-batch option #579

Closed andriytk closed 1 year ago

andriytk commented 1 year ago

The default value of 1000 objects in one DeleteObjects request can be too high for some S3 storage providers, the deletions of such amount of objects can take some time which can be interpreted as a failure by timeout. Retries would generate even more DeleteObjects requests that would only worsen the situation.

Solution: allow users to specify the number of objects in a single DeleteObjects requests with --max-batch option.