Closed LouisAuneau closed 7 months ago
Thank you for reporting the issue. I have a few ideas and could need your help to check them out and report back:
We had to implement a custom credential cache that might be to blame. Could you please try to await the first put_object
task before gathering the others?
await tasks[0]
await asyncio.gather(*tasks[1:])
If that helps, I will attempt to improve the caching logic.
According to docs for PutObject:
When you use this operation with a directory bucket, you must use virtual-hosted-style requests in the format Bucket_name.s3express-az_id.region.amazonaws.com.
Have you tried providing the bucket name in this format?
Thank you for your quick feedback. I tested your 2 points:
my-bucket--eun1-az1--x-s3.s3express-eun1-az1.eu-north-1.amazonaws.com
in my example), I get a NoSuchBucket
error surprisingly.
- Awaiting the first task before the other are executed indeed works. Let me know if I can be of any help for caching logic investigation.
Great! I will prepare a fix ASAP.
- When providing the full endpoint (
my-bucket--eun1-az1--x-s3.s3express-eun1-az1.eu-north-1.amazonaws.com
in my example), I get aNoSuchBucket
error surprisingly.
Well, it was worth a shot.
@jakob-keller ah I missed that, ya it needs to use an asyncio.Lock
Hello,
Describe the bug We are aiming at improving our uploading speed using S3 Express One Zone. Running the following code using botocore, I manage to upload files in a reasonable time:
However, I wanted to improve perfomance by running async with the following code:
But it seems when running multiple
put_object
call asynchronously, theCreateSession
endpoint is called at each call, leading to the followingClientError
:Checklist
pip check
passes without errorspip freeze
resultspip freeze results
Environment:
Additional context Add any other context about the problem here.