Aioboto3 blocks when calling Session or creating a resource.

[jpazarzis]

• Async AWS SDK for Python version: aioboto3 9.3.1
• Python version: python3.6
• Operating System: linux

Description

While load testing and profiling aioboto3 we suspect that we found a bug where an async code is calling a synchronous function. This problem blocks the entire ioloop (i.e non boto calls are affected)

What I Did

Please see the sample code that demonstrates the issue.

Hi,

While load testing and profiling aioboto3 we suspect that we found a bug where an async code is calling a synchronous function. This problem blocks the entire ioloop (i.e non boto calls are affected)

Libraries used:

The main environment we use for testing (python3.6)

• aioboto3 9.0.0
• aiobotocore 1.3.1
• boto3 1.17.49
• botocore 1.20.49

Trivial example to demonstrate the issue:

"""Demonstrates the blocking issue with aioboto3."""

import asyncio
import datetime
import aioboto3

counter = 0
lock = asyncio.Lock()

async def retrieval_process():
    global counter
    session = aioboto3.Session()
    async with session.resource('dynamodb', region_name='us-east-1',
                                endpoint_url='http://localhost:8000'
                                ) as dynamo_resource:
        pass
    async with lock:
        counter += 1

async def main(N):
    t1 = datetime.datetime.now()
    tasks = [retrieval_process() for _ in range(N)]
    await asyncio.gather(*tasks)
    while True:
        async with lock:
            if counter >= N:
                break
        await asyncio.sleep(0.1)
    t2 = datetime.datetime.now()
    print(f"N={N} total duration= {(t2 - t1).total_seconds()}")

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(2))
    loop.run_until_complete(main(50))

Similar behaviour when debugging aiobotocore directly (python 3.8.10)

• aiobotocore 2.1.0
• botocore 1.23.24

The block is happening because aiobotocore calls synchronous functions of boto3.

As an example, think of the coroutine named _create_client (implemented in aiobotocoro/session) which calls self._get_internal_component('endpoint_resolver')

def _get_internal_component(self, name):
        # While this method may be called by botocore classes outside of the
        # Session, this method should **never** be used by a class that lives
        # outside of botocore.
        return self._internal_components.get_component(name)

which eventually hits the get_component method in botocore.session which executes the deffered (sync) function as can be seen here:

class ComponentLocator(object):
    """Service locator for session components."""
    def __init__(self):
        self._components = {}
        self._deferred = {}

    def get_component(self, name):
        if name in self._deferred:
            factory = self._deferred[name]
            self._components[name] = factory()
            # Only delete the component from the deferred dict after
            # successfully creating the object from the factory as well as
            # injecting the instantiated value into the _components dict.
            del self._deferred[name]
        try:
            return self._components[name]
        except KeyError:
            raise ValueError("Unknown component: %s" % name)

The question

How we can assure the asynchronicity of both the Session and the create_client to eliminate the blocking.

[terricain]

This should be raised in the aiobotocore repo. That being said, it can be worked around for now by caching the client/resource by using something like an AsyncExitStack