Closed rikigigi closed 1 year ago
Thank you @sphuber for the explanation. I agree on everything. If the method list_objects is used we should definitevely find a better solution. Anyway I rebased on master, removed the modification to has_files, and checked that everything worked as expected for my workflow that triggered this issue.
Thanks a lot for your contribution @rikigigi ! Much appreciated
"file exist" operation now is performed with boto3's head_object before it was using list_object, that by default returns 1000 objects at maximum. The requested (existing) key may not be in the returned list. In any case the maximum number of objects must be known in advance, and listing all objects is a waste of resources.
solves https://github.com/sphuber/aiida-s3/issues/11
Some points to discuss on my side: