Open all4one-max opened 3 months ago
I tried implementing using the below code but it did not work I was getting Runtime error, Event loop is closed. I am executing operations such as download, upload in my celery task.
class AsyncS3ClientSingleton(metaclass=AWSMeta):
_async_s3_client = None
@classmethod
async def get_s3_client(cls) -> Any:
if not cls._async_s3_client:
# Lazy initialization of the S3 client
session = aioboto3.Session()
ctx = session.client(
"s3",
aws_access_key_id=AWS_ACCESS_KEY,
aws_secret_access_key=AWS_ACCESS_KEY_SECRET,
region_name="us-east-1",
)
cls._async_s3_client = await ctx.__aenter__()
return cls._async_s3_client
But when I create the session and client for every operation as shown below it works fine, though it defeats the purpose of reusing the client.
session = aioboto3.Session()
async with session.client(
"s3",
aws_access_key_id=AWS_ACCESS_KEY,
aws_secret_access_key=AWS_ACCESS_KEY_SECRET,
region_name="us-east-1",
) as async_s3_client:
await async_s3_client.upload_file(file_name, bucket, key)
Please help me fix this. I feel it has something to do with session being tied to an event loop and everytime a new celery task is initiated, new event loop is created. Only speculation though
you need to track the lifetimes of your event loops. The client is only valid for the life of the run loop
sorry it took so long to get back, somehow missed this
Is tracking an obligation? Especially in runtimes that are shortlived (e.g. AWS Lambda), this shouldn't really be a concern. I also use (and am looking for an equivalent) the singleton pattern (i.e. initialising one client and using it for everything) in Kotlin and other languages. Is it possible to achieve it with aiobotocore or should we just initialize these connections with the async enter/exit pattern?
Discussed in https://github.com/aio-libs/aiobotocore/discussions/1110