Maintenance: Support for Python 3.11

heitorlessa commented 1 year ago

Why is this needed?

Python 3.11 official docker image is available in preview: https://github.com/aws/aws-lambda-base-images/issues/62#issuecomment-1569890315

Which area does this relate to?

Other

Solution

Add support in CI. We have some customers using Python 3.11 using OCI images in Lambda.

Acknowledgment

[X] This request meets Powertools for AWS Lambda (Python) Tenets
[ ] Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

leandrodamascena commented 1 year ago

Initial findings:

1 - We need to remove python-snappy from the dev dependencies and refactor the example that uses it. This library does not support Python 3.11. There is an issue adding support for Python 3.11 (https://github.com/andrix/python-snappy/issues/124), but I have no idea when.

• Installing python-snappy (0.6.1): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel

  /tmp/tmpuenu801o/.venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'cffi_modules'
    warnings.warn(msg)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-311
  creating build/lib.linux-x86_64-cpython-311/snappy
  copying src/snappy/snappy_formats.py -> build/lib.linux-x86_64-cpython-311/snappy

2 - I got this error on the LRU cache test and I couldn't understand why we have this test. Do you have any historical data to help us @heitorlessa? Looks like we don't need this test.

    def test_lru_pop_failing():
        cache = LRUDict()
        key = "test"
        cache[key] = "value"
        try:
            cache.pop(key, None)
>           pytest.fail("GitHub #300: LRUDict pop bug has been fixed :)")
E           Failed: GitHub #300: LRUDict pop bug has been fixed :)

tests/unit/test_lru_cache.py:67: Failed
================================================================================================================================================ short test summary info ================================================================================================================================================
FAILED tests/unit/test_lru_cache.py::test_lru_pop_failing - Failed: GitHub #300: LRUDict pop bug has been fixed :)

3 - If we remove the python-snappy library and this LRU test, our CI works fine and we have no compatibility issues with our code.

4 - I will start some exploratory tests using the image available here: https://gallery.ecr.aws/lambda/python.

Thanks

leandrodamascena commented 1 year ago

For transparency, here's our playbook for releasing a new version of Powertools to add support for the AWS Lambda Python 3.11 runtime, when it's released in GA.

Steps

1. Prerequisites

The new AWS Lambda Python runtime needs to be supported by CDK
The new AWS Lambda Python runtime needs to be supported by SAM

2. Testing locally

Must install the new Python runtime version and run all the tests and linters locally to see if anything is breaking, by running make pr

3. Bumping the supported runtimes in the layer construct

On the CDK Layer repository, do:

Bump the CDK version to support the new AWS Lambda Python runtime version
Change the supported runtimes to include the new runtime
Make sure tests still pass by running yarn projen test
Submit and merge a PR with the changes
Execute the release workflow to generate a new version of the construct from the main branch
- Example: this example PR added support to 3.10

4. Updating workflows and templates

Update all files inside the .github directory to include the new runtime version. This includes the e2e workflow
- Example: this example PR added support to 3.10

5. Layer

Bump the CDK version on the on the root of the repository, by editing package.json and running npm install
Inside the layer directory: bump the Layer construct version by editing the pyproject.toml and running poetry lock.

6. SAR

Add the new runtime to the layer/sar/template.txt

7. Change the PyPi metadata

Edit the pyproject.toml and update the poetry classifiers to include the new runtime.

8. Open a PR and run E2E tests

Open a new PR with the changes made in steps 4, 5, 6 and 7.
Wait for the automated tests to pass (so the tests run against the new runtime)
Manually schedule an E2E workflow on that branch (so the automated tests run against the new runtime)
- Example: this example PR added support to 3.10

9. Examples

Update the SAM/Terraform examples to use the latest version of the supported runtime.

10. Merge and release

If everything is green you should be able to merge the PR and release a new version, which will generate and publish the new Layer and SAR template.
Have a cold beverage (don't forget that) and celebrate! 🍺 🎉

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

github-actions[bot] commented 1 year ago

This is now released under 2.22.0 version!

github-actions[bot] commented 1 year ago

This is now released under 2.23.0 version!

aws-powertools / powertools-lambda-python