This PR adds the ability for users to list and evaluate all versioned objects in the bucket. The existing behavior was using ListObjectsV2 API which only listed current/latest objects on a versioned bucket. As a result all the metrics did not account for older versions of objects which were present on the bucket and taking up space. For my use case this was unacceptable, as I need to know when I'm approaching the storage limit of my bucket.
In order to address this a few changes were introduced:
Provide a new flag --s3.with-versions which changes the behavior of the API calls and is purely opt in (default is false) which maintains backwards compatibility.
Abstract away the counters into a new ItemAggregator struct which keeps track of statistics and define a separate parallel method on how to evaluate all objects in a bucket using ListObjectVersions API.
Select between 2 different implementations (CountViaListObjectsV2 and CountViaListObjectVersions) the appropriate method based on the flag.
Extend test case semantics to support the new API usage and write additional unit tests to exercise them.
Further things of note:
Right now, the flag is global and not per bucket. This is ok, because ListObjectVersions is backwards compatible with non-versioned buckets and will function just as well with any bucket. Otherwise, we would also have to query the bucket versioning status.
In integ testing, on my S3 Provider (E2 idrive) the ListObjectVersions API performs about 10-15% slower for the same non-versioned bucket. Obviously for a versioned bucket the difference can be much much bigger. This matters if the evaluation time is close to the job timeout in Prometheus (i.e. you have a huge bucket with a lot of objects).
Tested on my account end to end across 3 different buckets (versioned, non-versioned, empty non-versioned).
Happy to consider any changes you want me to make or any other suggestions you might have.
Summary:
This PR adds the ability for users to list and evaluate all versioned objects in the bucket. The existing behavior was using ListObjectsV2 API which only listed current/latest objects on a versioned bucket. As a result all the metrics did not account for older versions of objects which were present on the bucket and taking up space. For my use case this was unacceptable, as I need to know when I'm approaching the storage limit of my bucket.
In order to address this a few changes were introduced:
--s3.with-versionswhich changes the behavior of the API calls and is purely opt in (default is false) which maintains backwards compatibility.ItemAggregatorstruct which keeps track of statistics and define a separate parallel method on how to evaluate all objects in a bucket using ListObjectVersions API.Further things of note:
Tested on my account end to end across 3 different buckets (versioned, non-versioned, empty non-versioned).
Happy to consider any changes you want me to make or any other suggestions you might have.