Boto3 and botocore need upgrading

[mdavis-xyz]

The versions of the AWS SDK in the python3.8 image are outdated. (I suspect the same is true for all python 3.x images)

The docs say boto3-1.18.55 and botocore-1.21.55 for all python3.x runtimes. These get updated every few months.

This image uses:

$ docker run --rm -it lambci/lambda:build-python3.8 ls /var/runtime/ -l | grep boto
drwxr-xr-x 10 root root   210 Jan 13  2021 boto3
drwxr-xr-x  2 root root   148 Jan 13  2021 boto3-1.16.31.dist-info
drwxr-xr-x  7 root root  4096 Jan 13  2021 botocore
drwxr-xr-x  2 root root   148 Jan 13  2021 botocore-1.19.31.dist-info

The consequence of this is that when I pip install libraries that have the same dependencies as boto/botocore, I get dependency clashes. Either pip takes an eternity downloading every combination of a few libraries to figure out mutual dependencies, or it claims to work but simple scripts fail because of incompatable interfaces between libraries.

[thomas-icomplai]

is there a way to upgrade this yourself?

[mdavis-xyz]

There's probably a way. Note that the normal image doesn't include pip. The build one does though.

Note also that the version of botocore and boto3 in lambda tends to lag behind the latest version of the libraries. So you can't simply upgrade to the latest, you must pin the versions. I was hoping that's something that this project would handle for me, by just copy-pasting the files from the Lambda runtime regularly.

[thomas-icomplai]

I worked around this now with the wonderful blog post from https://www.mandsconsulting.com/lambda-functions-with-newer-version-of-boto3-than-available-by-default/

effectively just boils down to prioritize the pip installed packages before importing them:

if "LAMBDA_TASK_ROOT" in os.environ:
    envLambdaTaskRoot = os.environ["LAMBDA_TASK_ROOT"]
    sys.path.insert(0, "/var/lang/lib/python3.9/site-packages")

import boto3
import botocore

It's a real shame that the runtime does not do this itself before executing the script.

[Homas]

You can add a layer with updated packages On Linux:

mkdir -p lambda_layer/python
cd lambda_layer/python
pip install boto3 -t ./
pip install botocore -t ./
cd ..
zip -r ~/lambda_layer.zip .

after that upload lambda_layer.zip as a layer to your function (it is relatively small so no need to store on S3)

[mdavis-xyz]

From memory boto3 and botocore and not relatively small. They're like 60MB, right? So that's 1/8th of the size quota gone just to the AWS SDK. If you're trying to also squeeze in some other library like matplotlib or awswrangler, that's going to pose a challenge for real deployments.

Although this project is for unit testing. so size isn't an issue. the issue is just that the docker container doesn't accurately reflect the environment its emulating. Installing the latest version of boto3 and botocore won't either, since real lambda tends to lag the latest SDK versions.

Also, you should probably install boto3 and botocore in one single pip command, to ensure the two versions are compatible with each other.