search

storj / gateway-st

Single-tenant, S3-compatible server to interact with the Storj network
Apache License 2.0
71 stars 19 forks source link

Make it possible for object listing actions to be fully S3-compliant (lexicographical ordering) #49

Closed amwolff closed 2 years ago

amwolff commented 2 years ago

Currently, we are fully S3-compliant when we fall back to the exhaustive listing (listing the entire bucket and filtering it gateway-side, which is slow, but it's better than nothing!) when the user specifies custom prefix and/or delimiter. We lack lexicographical ordering to be fully S3-complaint on the optimal listing path (non-exhaustive listing).

We should make a configuration attribute/flag to make it possible to always fall back to exhaustive listing as some use cases require ListObjects/ListObjectsV2 actions to be fully S3-compliant.

Implementation of this enhancement must ensure that the listObjectsExhaustive method does not depend on the location from where it's called.

storjBuildBot commented 2 years ago

Change https://review.dev.storj.io/c/storj/gateway/+/6748 mentions this issue.

wthorp commented 2 years ago

Note that this fix would not realistically be deployed in most production scenarios. Storj does List objects in a reliably sorted order, it's ordered by the encrypted object path. This fix proves that we could remedy this issues in the Gateway-MT tier, however the potential performance penalties make this solution untenable. Other solutions may exists deeper into the Storj product - notably not encrypting paths. Not encrypting paths might take some work and would compromise on our expressed security posture.