Feature Request: Opening aioboto3 config to increase pool thread

robm1158 commented 10 months ago

Async AWS SDK for Python version:
Python version: 3.11
Operating System: Windows 11

Description

I am attempting to increase the max_pool_size for aioboto3 Session. I have attempted to use the boto3 version as aioboto3 wraps boto3 i thought but with no success. I am making an async API request(s) that will eventually be sending off 300k requests so a way to increase the pool size so that I dont get banned from the website would be great.

What I Did

I attempted doing these things: config = botocore.config.Config(max_pool_connections=20) self.session = aioboto3.Session()

boto_config = aioboto3.Config(max_pool_connections=20) self.session = aioboto3.Session(botocore_session=boto_config)

self.connector = aiohttp.TCPConnector(limit_per_host=20) self.aiohttp_session = aiohttp.ClientSession(connector=self.connector)

Paste the command(s) you ran and the output.
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: dynamodb.us-east-1.amazonaws.com. Connection pool size: 10

terricain commented 10 months ago

Aioboto3 doesn't really care about session, you're probably better off dealing with that in aiobotocore it'll then apply to aioboto3.

robm1158 commented 10 months ago

ah gotcha is there any documentation for how to mess with that?

terricain commented 9 months ago

Not really, I'd head over to their repo and read the code 🤣

robm1158 commented 9 months ago

Haha fair enough cheers!

terricain / aioboto3