Cannot export metrics where the prefix matches more than 1000 objects

avanier commented 3 years ago

Since we are using the ListObjectsV2 call, the S3 API will limit the response to 1000 objects. Adding to this, objects are listed in no guaranteed order, so once you reach 1000 objects, the metrics a pretty much baloney.

To address this, I believe there are two options which start with using the ListObjectsV2Pages API call instead:

We can recurse through all the pages, which yields good metrics at the cost of API calls and compute time.
We can list the pages, and just read the last. This would potentially lead to inaccurate metrics, but have more or less the same compute and API call cost as the current implementation.

We might need this feature, so I'll likely submit a PR.

ribbybibby commented 3 years ago

Interesting. Have you seen it failing for >1000 objects? I was under the impression that we could use the continuation token to keep listing until we'd listed everything, which is what we do: https://github.com/ribbybibby/s3_exporter/blob/master/s3_exporter.go#L93-L118

avanier commented 3 years ago

Hrm, I think you might be right. Lemme dig some more. I'll be glad to be wrong. I might have improperly configured my probes.

I'll close the issue with more details if that's the case.

avanier commented 3 years ago

Yep, you were totally right. I both didn't read the code far enough, and I was watching the wrong prefix.

Thanks a lot for indulging me. :stuck_out_tongue: Your exporter is awesome. :muscle:

ribbybibby / s3_exporter

Cannot export metrics where the prefix matches more than 1000 objects #22