sphuber / aiida-s3

AiiDA plugin that provides various storage backends that allow using cloud data storage services, such as AWS S3 and Azure Blob Storage.
MIT License
3 stars 2 forks source link

`S3RepositoryBackend`: Do not rely on `list_objects` for `open` #14

Closed rikigigi closed 1 year ago

rikigigi commented 1 year ago

"file exist" operation now is performed with boto3's head_object before it was using list_object, that by default returns 1000 objects at maximum. The requested (existing) key may not be in the returned list. In any case the maximum number of objects must be known in advance, and listing all objects is a waste of resources.

solves https://github.com/sphuber/aiida-s3/issues/11

Some points to discuss on my side:

rikigigi commented 1 year ago

Thank you @sphuber for the explanation. I agree on everything. If the method list_objects is used we should definitevely find a better solution. Anyway I rebased on master, removed the modification to has_files, and checked that everything worked as expected for my workflow that triggered this issue.

sphuber commented 1 year ago

Thanks a lot for your contribution @rikigigi ! Much appreciated