aioboto3 import error on aws lambda based image

[cmflynn]

• Async AWS SDK for Python version: 10.1.0
• Python version: 3.9
• Operating System: FROM public.ecr.aws/lambda/python:3.9

Description

aws lambda fails to import aioboto3 with the following error: [ERROR] ImportError: cannot import name 'apply_request_checksum' from 'botocore.client' (/var/runtime/botocore/client.py)

What I Did

Using aioboto3<=9.3.0 is a workaround.

It appears that AWS uses their own versions of boto3 and botocore on their lambda image exec environment, that do match what pip freeze thinks should be installed on the actual docker image.

as a result, aiobotocore.client fails on

from botocore.client import (
    BaseClient,
    ClientCreator,
    ClientEndpointBridge,
    PaginatorDocstring,
    S3ArnParamHandler,
    S3EndpointSetter,
    apply_request_checksum,
    logger,
    resolve_checksum_context,
)

this is because botocore==1.23.32 does not have apply_request_checksum in its client.

looking deeper at package version...

pip freeze says: boto3==1.24.59 botocore==1.27.59

(app.py, I run this lambda handler in aws env, to see whats actually installed at run time)

import importlib_metadata

def lambda_handler(event, context):
    print(event)
    print(context)
    print(importlib_metadata.version("aioboto3"))
    print(importlib_metadata.version("aiobotocore"))
    print(importlib_metadata.version("boto3"))
    print(importlib_metadata.version("botocore"))

    import aioboto3
    assert aioboto3

if __name__ == '__main__':
    lambda_handler("a", "b")

BUT in the aws lambda logs I get this:

2022-10-03T12:28:21.554-04:00 | 10.1.0 (aioboto3)
  | 2022-10-03T12:28:21.607-04:00 | 2.4.0 (aiobotocore)
  | 2022-10-03T12:28:21.608-04:00 | 1.20.32 ( boto3)
  | 2022-10-03T12:28:21.610-04:00 | 1.23.32 ( botocore )

pip freeze:

(venv) drivedigital git:cflynn/DRIV-604 ❯ docker run -it --entrypoint /bin/bash updatereconciliationdetailsfunction:python3.9-v1                                                                                                  ➜ ✹ ✭
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
bash-4.2# python app.py 
a
b
10.1.0
2.4.0
1.24.59
1.27.59
bash-4.2# pip freeze
aioboto3==10.1.0
aiobotocore==2.4.0
aiohttp==3.8.3
aioitertools==0.11.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==22.1.0
boto3==1.24.59
botocore==1.27.59
charset-normalizer==2.1.1
drivedigital-setup==1.0
frozenlist==1.3.1
idna==3.4
importlib-metadata==5.0.0
jmespath==1.0.1
multidict==6.0.2
python-dateutil==2.8.2
s3transfer==0.6.0
six==1.16.0
typing_extensions==4.3.0
urllib3==1.26.12
wrapt==1.14.1
yarl==1.8.1
zipp==3.8.1
bash-4.2# 

my requirements.txt for this image:

importlib-metadata
aioboto3

[cmflynn]

ended up resolving this by doing:

...
import sys

# aws lambda come pre-installed with boto libs that are not the same version as the one we package. this ensures
# we import the version we package.
sys.path.insert(0, "/app")
import aioboto3
...

where /app is the root of your lambda

[terricain]

Yeah, there's nothing much I can do about this as this is really a lambda problem.

Apparently in /var/runtime/bootstrap.py LAMBDA_TASK_ROOT should set that as the first in the path, though im not sure where that would get set.

[simlmx]

I had the same error and I fixed it by choosing versions of dependencies that matched the ones in the python lambda runtime.

The versions in the runtime are:

boto3 = '1.20.32'
botocore = '1.23.32'

I ended up using

botocore ='~1.23.24'
aiobotocore = '~2.1.2'

as this was the closest fit I could get that poetry was happy with.