sphuber / aiida-s3

AiiDA plugin that provides various storage backends that allow using cloud data storage services, such as AWS S3 and Azure Blob Storage.
MIT License
3 stars 2 forks source link

`S3RepositoryBackend`: Do not explicitly check existence of objects before deleting #21

Open sphuber opened 1 year ago

sphuber commented 1 year ago

See #20 for a discussion. The current implementation calls the super, which performs a preliminary check using has_objects, which in turn calls list_objects which will be very expensive for larger repositories. If the interface of the AbstractRepositoryBackend allows it, we should simply call boto3.delete_objects which will delete all files that exist and return an error for keys that didn't exist.