await response['Payload'].read() got stuck and never finishes when invoking lambda functions

[DataNoob0723]

• Async AWS SDK for Python version: aioboto3==9.0.0, aiobotocore==1.3.1
• Python version: 3.6.8
• Operating System: MacOS Catalina

Description

We are using aioboto3 to build an application with aiohttp==3.6.2, which invokes two AWS Lambda functions remotely. The core part of the code is something like:

    async with boto3_session.client(
            'lambda',
            aws_access_key_id=AWS_ACCESS_KEY_ID,
            aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
            region_name=REGION
    ) as lambda_client:
        response = await lambda_client.invoke(
            FunctionName=LAMBDA_FUNCTION_NAMES['xxx_function'],
            InvocationType="RequestResponse",
            Payload=json.dumps(data)
        )
    if response['StatusCode'] == 200:
        result = await response['Payload'].read()
        result = json.loads(result)  # Convert to Dict
        if result['flag'] == 1:
            result1, result2 = result.get('result1'), result.get('result2')
            return result1, result2
        else:
            raise Exception(result.get('message'))
    else:
        raise Exception(f'Task failed with ResponseMetadata: {response["ResponseMetadata"]}.')

Where boto3_session = aioboto3.Session(), imported from a shared module. We were using asyncio.gather to invoke two different lambda functions in parallel. The issue is that sometimes it works fine, but sometimes it got stucked. It will always return a response in the above code snipet, but sometimes it will got stuck due to result = await response['Payload'].read() never finishes. We are not sure why the Streamingbody object's read method can get stuck and never finishes. And this issue happened sometimes, not sure how to fix it. Please help!

What I Did

We tried to search online, but couldn't get any clue about how to resolve this issue. Help would be highly appreciated.

[hassanj-576]

Did anyone find a solution to this ? I am facing the same issue

[luis8]

Did anyone find a solution to this ? I am facing the same issue

In case someone else have this problem in the future.

You need to await the read inside the context manager otherwise the connection will be closed.

Using the OP example

Wrong:

    async with boto3_session.client(
            'lambda',
            aws_access_key_id=AWS_ACCESS_KEY_ID,
            aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
            region_name=REGION
    ) as lambda_client:
        response = await lambda_client.invoke(
            FunctionName=LAMBDA_FUNCTION_NAMES['xxx_function'],
            InvocationType="RequestResponse",
            Payload=json.dumps(data)
        )
    if response['StatusCode'] == 200:
        result = await response['Payload'].read() # this is outside!

Right

    async with boto3_session.client(
            'lambda',
            aws_access_key_id=AWS_ACCESS_KEY_ID,
            aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
            region_name=REGION
    ) as lambda_client:
        response = await lambda_client.invoke(
            FunctionName=LAMBDA_FUNCTION_NAMES['xxx_function'],
            InvocationType="RequestResponse",
            Payload=json.dumps(data)
        )
        if response['StatusCode'] == 200:
            result = await response['Payload'].read() # this now lives inside the context manager

[terricain]

I must have missed this, @luis8 is correct, if you try and read the response outside the context manager the underlying connection and resources for the API call will have been closed/freed