EMCECS / presto-s3-connector

Apache License 2.0
8 stars 3 forks source link

S3TableDescriptionSupplier will not scale for buckets with many objects - need to refactor #4

Closed chipmaurer closed 2 years ago

chipmaurer commented 3 years ago

The code is calling listObjects for each bucket, which will not work for buckets with large numbers of objects. For bucket metadata queries, we need to be more like ODBC queries, and translate SQL into metadata search REST calls.