search

IntelLabs / vdms

VDMS: Your Favorite Visual Data Management System
MIT License
83 stars 31 forks source link

S3 connection layer doesn't support buckets with large number of files #179

Open ifadams opened 1 month ago

ifadams commented 1 month ago

Issue: The basic AWS S3 connection interface doesn't support listing files in an AWS bucket when the number of files exceeds a threshold (possibly 1000).

Solution: Need to implement continuation tokens for AWS request responses to handle this scenario.

ifadams commented 1 month ago

In practice, current version of VDMS never does a full listing of the bucket, but this is still an issue we should look into as we add more resiliency and recovery mechanisms over time which may wish to do so.