has_objects() compares the provided keys with the result of a call to the boto3 function list_objects()
list_objects() returns by default a maximum of 1000 items. In general the maximum number of returned items is required. This call can be time consuming for the s3 server because the metadata of all objects in the response is returned.
the key may not be in the object list and the code raises an error
Proposed solution: if necessary, the "file exists" operation can be implemented with the head_object call.
The open() call should not check that the file exist by calling super().open(key)
https://github.com/sphuber/aiida-s3/blob/a7eb601b50367954efe8cbd9e059e55adbfe4192/src/aiida_s3/repository/aws_s3.py#L119 before calling the download_fileobj in boto3 library, the code checks in a wrong way that the file exist. It performs the following steps:
Proposed solution: if necessary, the "file exists" operation can be implemented with the head_object call. The open() call should not check that the file exist by calling super().open(key)