This adds support for Azure blob storage to up space billing get. It also refactors the GCS and S3 implementations to use provider-generic code for generating the report.
Other things worth noting:
All provider implementations support reading gzipped data.
S3 events can now be grouped by arbitrarily sized windows, like the other providers.
The number of S3 objects that can be read in a time window is now unlimited, up from 5000.
I gave no consideration to optimizing download speed with concurrency. I'll leave this for a later PR.
This is a big PR, so see commits for logical groupings of changes.
Description of your changes
This adds support for Azure blob storage to
up space billing get. It also refactors the GCS and S3 implementations to use provider-generic code for generating the report.Other things worth noting:
This is a big PR, so see commits for logical groupings of changes.
I have:
make reviewableto ensure this PR is ready for review.backport release-x.ylabels to auto-backport this PR, as appropriate.How has this code been tested
I ran
up space billing getagainst sample data with each storage provider. I'll update this with more detail shortly.