Gateway: add a custom endpoint to retrieve total object count

storj / edge

Storj edge services (including multi-tenant, S3-compatible server to interact with the Storj network)

GNU Affero General Public License v3.0

53 stars 18 forks source link

Gateway: add a custom endpoint to retrieve total object count #373

Open mobyvb opened 11 months ago

mobyvb commented 11 months ago

The endpoint should return values for:

total object count (includes encrypted + decrypted objects) - this can be done with iteratively calling "libuplink list" with the "encrypted" flag
encrypted object count

It should also be limited in some way (e.g. if there are 10s of millions of objects). Add a config value.

See "list buckets with attribution" for an example of a custom endpoint on gateway: https://github.com/storj/gateway-mt/blob/267c4fc8f7fdbb1953a8636ea305368a503d5506/pkg/minio/api-response.go#L25

Context for underlying issue: https://github.com/storj/storj/issues/6080

AC:

new endpoint added to gateway for listing total object counts on the bucket level
add configuration for restricting this functionality in situations where the object count is very high

storj-gerrit[bot] commented 11 months ago

Change pkg/{server, minio}: custom endpoint to return total object count mentions this issue.

halkyon commented 11 months ago

Is there any reason we decided not to add something to the satellite API instead of going through gateway with ListObjects calls? I may be missing some context, but just looking at the gateway change it seems like calling ListObjects might not be an efficient way of doing it.

BTW: it would be helpful if background info is added to the issue and the commit message to aid in reviewing.

storj-gerrit[bot] commented 11 months ago

Change pkg/{server, minio}: custom endpoint to return total object count mentions this issue.

ferristocrat commented 11 months ago

@halkyon - The context is in https://github.com/storj/storj/issues/6080 (added to description as well)

halkyon commented 11 months ago

@ferristocrat Unfortunately it doesn't have everything, there's at least a couple of Slack threads with more details I had to go hunt down. Part of my problem generally is our reliance on Slack too much (it's horrible as a documentation source, and it's exclusive to open source contributions), but that's for another discussion ;)

Anyway, judging by some discussions there's still some work to figure out what to do on this.

ferristocrat commented 11 months ago

GitHub Thread discussing options: https://storj.slack.com/archives/CAQV0AE2Z/p1698676617000929