Add support for Python 3.11

[michael-k]

Latest official update [2023-01-09]:

We are currently working on Py3.10 support (see https://github.com/aws/aws-lambda-base-images/issues/31). Py3.11 will ship separately. We don't currently have a timeline to share on Py3.11 support. We will we post updates on this issue when we have further information to share.

And another official update from a different thread [2023-02-28]:

We don't have a ETA to share as yet.

---

Python 3.11 ~will be~ was released on 2022-10-24 (the final release candidate was released on 2022-09-12)

• Announcements:
  • Release Candidate 2: https://pythoninsider.blogspot.com/2022/09/python-3110rc2-is-now-available.html
  • 3.11.0: https://pythoninsider.blogspot.com/2022/10/python-3110-is-now-available.html
  • 3.11.1: https://pythoninsider.blogspot.com/2022/12/python-3111-3109-3916-3816-3716-and.html
  • 3.11.2: https://pythoninsider.blogspot.com/2023/02/python-3112-python-31010-and-3120-alpha.html
  • 3.11.3: https://pythoninsider.blogspot.com/2023/04/its-time-for-another-set-of-python.html
• Release schedule: https://peps.python.org/pep-0664/
• What’s new in Python 3.11: https://docs.python.org/3.11/whatsnew/3.11.html

(I know that 3.10 isn't available yet on Lambda; but AWS could add both at the same time and not let us wait for > 1 year until they add 3.11 :pray:)

[jtuliani]

Thank you for creating this ticket. We are currently working on Python 3.10 support, with Python 3.11 to follow later. We are also working to improve our runtime release process to enable us to release new runtimes more quickly.

[mrlonis]

Any timeframe on this? 3.11 is critical for us.

They still don't have Python 3.10 ready and it's been over a year since its release. You might be waiting a LONG time

[rafrafek]

We are also working to improve our runtime release process to enable us to release new runtimes more quickly.

Maybe start with an alpha version like 3.12.0a2 then it would take less time to add a new stable version when it releases.

[rafrafek]

@fox91 @jorwoods why thumbs down? It is a standard approach to prepare with early builds before final release to speed up the adaption process and catch all the issues earlier. AWS can do this too.

[fox91]

@fox91 @jorwoods why thumbs down? It is a standard approach to prepare with early builds before final release to speed up the adaption process and catch all the issues earlier. AWS can do this too.

Open another issue, this is for 3.11 only. What does 3.12 have to do with it here? Unrelated messages are just spam for all people who subscribe and expect relevant messages.

Thanks

[keithly]

It's possible to run 3.11 with a custom runtime, for example: https://github.com/hb2638/aws_lambda_python_3.11 https://github.com/keithly/lambda-python-custom

[CharlesPerrotMinotHCHB]

True, but a custom runtime isn't an official one, hence will always present a security threat potential. Unless you import it, redo it yourself, and many companies won't really consider it worth the work compared to the gains of the update.

[jtuliani]

To clarify, I believe @baotran2207 's comment above reflects a desire for when s/he would like AWS Lambda to support Python 3.11. It is not a statement from AWS on when Python 3.11 will be supported.

We are currently working on Py3.10 support (see #31). Py3.11 will ship separately. We don't currently have a timeline to share on Py3.11 support. We will we post updates on this issue when we have further information to share.

[dorinclisu]

Given that 3.11 is so much faster (10–60%), I'm curious what would be the take of aws managment on prioritizing this runtime, considering that the faster execution time will directly translate into smaller lambda bills.

Of course, the rational thing to do as customers here is to re-calculate the cost of alternative compute systems and perhaps use lambda less.

[kamikaze]

Given that 3.11 is so much faster (10–60%), I'm curious what would be the take of aws managment on prioritizing this runtime, considering that the faster execution time will directly translate into smaller lambda bills.

now you have the answer :) they need larger bills, not smaller. So for 3.11 they need to port all the slowdowns back 🤭

[stewartsmith]

Amazon Linux 2023 is now GA, and contains Python 3.11 packages as an option (system python in AL2023 will remain as 3.9, but 3.11 is fully functional with pip)

[j616]

Amazon Linux 2023 is now GA, and contains Python 3.11 packages as an option (system python in AL2023 will remain as 3.9, but 3.11 is fully functional with pip)

Should we expect this to roll out as a lambda runtime option soon? I don't see it in the list of options currently

[dorinclisu]

So let's see the history a bit:	Version	GA Release	AWS Lambda Release	Delay
3.7	2018-06-27	2018-11-19	5 months
3.8	2019-10-14	2019-11-18	1 month
3.9	2020-10-05	2021-08-16	10 months
3.10	2021-10-04	?	17+ months
3.11	2022-10-24	?	5+ months

[mariogalic]

Amazon Linux 2023 is now GA, and contains Python 3.11 packages as an option (system python in AL2023 will remain as 3.9, but 3.11 is fully functional with pip)

Making use of Amazon Linux 2023 I was able to deploy python 3.11 AWS Lambda via the following Dockerfile:

FROM amazonlinux:2023
RUN yum update -y
RUN yum groupinstall "Development Tools" -y
RUN yum install python3.11 python3.11-pip python3.11-setuptools python3.11-wheel python3.11-devel -y
RUN pip3.11 install awslambdaric

ENV LAMBDA_TASK_ROOT=/var/task
RUN mkdir -p ${LAMBDA_TASK_ROOT}
WORKDIR ${LAMBDA_TASK_ROOT}

# Install application code
COPY requirements.txt  .
RUN pip3.11 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
COPY . .

ENTRYPOINT [ "python3.11", "-m", "awslambdaric" ]

CMD [ "app.lambda.handler" ]

[mat100payette]

To clarify, I believe @baotran2207 's comment above reflects a desire for when s/he would like AWS Lambda to support Python 3.11. It is not a statement from AWS on when Python 3.11 will be supported.

We are currently working on Py3.10 support (see #31). Py3.11 will ship separately. We don't currently have a timeline to share on Py3.11 support. We will we post updates on this issue when we have further information to share.

Blink twice if you're forced to post these generic updates.

More seriously though, if custom runtimes don't have free init, AWS management is probably milking 3.9 as long as possible and focusing on making the containers "easy to use" and "justified". Financially, AWS Lambda likely has to determine if the addition of python 3.10 and 3.11 as managed runtimes will bring in enough new big clients to offset the losses due to existing clients having to pay less from faster lambdas across the board. While this is obviously just speculation on my end, it doesn't sound too far fetched to believe that making people's lambdas significantly faster isn't their primary interest as long as they have decent client retention.

Google supports 3.11 since early january this year. Sorry to those subscribed to this (and the 3.10) issue; I'm of the opinion that publicly sharing justified displeasure towards a major player in the cloud providers sphere is a good thing. Their inability to provide any information whatsoever regarding the issues faced during this overly stretched integration of python runtimes is simply insulting. I still remain hopeful that their "improved release process" is a major improvement and not just a case of empty PR talk, but personally I'll move on to another provider in the meantime.

[tongclement]

Still waiting. Surely this can't be this complicated with so many talented engineers on hand.

[mmattice-arpio]

More seriously though, if custom runtimes don't have free init, AWS management is probably milking 3.9 as long as possible and focusing on making the containers "easy to use" and "justified". Financially, AWS Lambda likely has to determine if the addition of python 3.10 and 3.11 as managed runtimes will bring in enough new big clients to offset the losses due to existing clients having to pay less from faster lambdas across the board. While this is obviously just speculation on my end, it doesn't sound too far fetched to believe that making people's lambdas significantly faster isn't their primary interest as long as they have decent client retention.

There is another way to look at the situation. Making executions faster gives customers more room in their 15 minutes (for compute loads) or expands the capacity of the existing pile of infrastructure. Happier customers or less hardware deploy pressure respectively.

As an organization, we've already written all the code necessary to resume long running (15m+) processes, so we're not worried about that aspect.

The quicker execution allows us to provide better service to our customers.

[mbtx2]

My management often asks for reasons to use one cloud provider over another for different tasks. With the amount of Lambda time we purchase, this kind of stuff makes a good argument to move. AWS needs to stay current with their runtimes. I can make a custom one fairly easily, but that seems like something they should just support.

Interesting that in the link that @mat100payette references for GCP, they even list 3.11 as the recommended runtime. Being two versions behind GCP is not a good look, especially since AWS has historically been known to be a leading edge cloud provider. AWS, you should pay attention to this thread.

[skwashd]

@mbtx2 they're now only one version behind GCP - https://github.com/aws/aws-lambda-base-images/issues/31#issuecomment-1511878656.

With 3.10 out the door, that should free up some team members to work on 3.11 support.

[fdobrovolny]

https://aws.amazon.com/blogs/compute/python-3-10-runtime-now-available-in-aws-lambda/ https://gallery.ecr.aws/lambda/python

Preview base image released for 3.11. !

[jtuliani]

We have shipped a preview base container image for Python 3.11 support in AWS Lambda. The purpose of this image is to enable customers to get an early look at Python 3.11 support and provide any feedback. This image is subject to change and should not be used for production workloads.

At this time, we do not have a schedule to share on when Python 3.11 support will reach General Availability. However, we hear your feedback and acknowledge that the recent delays in supporting the latest runtimes has fallen short of customer expectations. We’re working hard to catch up.

[mbtx2]

Thank you, glad to see some progress! We'll start testing that preview image right away. I should add that one thing that would help is some sort of ballpark GA target. I understand your hesitancy in providing any timelines, but the reality is that your customers need to plan, and plans need some sort of expectation of timeline, so it would be very appreciated. Is this git repo the best channel for us to provide feedback on the 3.11 preview? Just submit issues/suggestions here?

[mbtx2]

What is the difference between: public.ecr.aws/lambda/python:3.11-preview.2023.04.17.20-x86_64 and public.ecr.aws/lambda/python:3.11-preview?

[fdobrovolny]

@mbtx2 As both tags have the same id/hash, I assume public.ecr.aws/lambda/python:3.11-preview will always point to the latest preview image and, therefore, will change in time. Where public.ecr.aws/lambda/python:3.11-preview.2023.04.17.20-x86_64 will stay the same image/binary. It is common practice with docker images.

[mbtx2]

Got it, thank you.

[jtuliani]

@mbtx2 We will share a timeline for Python 3.11 as soon as we can. You can provide feedback in this thread, or for specific topics it would be better to create a separate issue so we don't mix conversations and can track more clearly.

[mbtx2]

One issue we've encountered is that SAM CLI doesn't work to debug functions in VSCode, because SAM says it's not a valid runtime, as defined in the launch.json. It says:

2023-05-03 11:37:14 [ERROR]: SamLaunchRequestError: Invalid launch configuration: Run configurations for Image-based Lambdas require a valid Lambda runtime value, expected one of [python3.7, python3.8, python3.9, dotnet6, java11, dotnet5.0, dotnetcore3.1, java8.al2, go1.x, python3.10, nodejs12.x, java8, nodejs14.x, nodejs16.x, nodejs18.x] [BadLaunchConfig]

Maybe there is a preview version of SAM that allows you to use the preview runtimes? I can open another request in the sam-cli repo issues, but wanted to post this here as well for reference. It becomes a chicken-and-egg problem when AWS tooling doesn't have the ability to work with AWS preview runtimes. I'd like to test to provide feedback, but can't.

[jtuliani]

Thanks for flagging @mbtx2. The Python 3.11 image is currently in Preview. Using this image in SAM is not yet supported. We are working on it!

[mbtx2]

I should say that SAM works well with lambda containers built with 3.11 and Lambda RIC. Just doesn't work with the runtime.

[mt-ronkorving]

@jtuliani I noticed that the version of boto3 that ships with the runtime hasn't changed since 3.7 (including up to 3.10).

source: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Is that something that could be modernized with 3.11?

[DilLip-Chowdary-Codes]

@mt-ronkorving The version of boto3 will keep updated in the runtime updates (in all the supported runtimes), the version 1.26.90 was updated recently, previously it was an older version.

[abmohan]

@jtuliani - Thanks for your attention to this. It would be helpful to have a transparent release schedule from the Lambda team, similar to what the Python Software Foundation provides for python releases.

Python releases PEPs which outline the anticipated release schedule for minor and fixpack versions: https://peps.python.org/pep-0693/. Here is a a chart showing previous releases: https://devguide.python.org/versions/

Python 3.12 is expected to be generally available on Oct 02, 2023 (i.e., about 5 months from now). It looks like they've consistently been releasing one minor release every fall from 3.8. Is this something the AWS team can adequately resource to ensure that lambda images are reliably only a few months behind? Will the 3.11 lambda image be released before 3.12 is generally available?

Python has been aggressively adding non-trivial functionality in each minor release (e.g., async support, better type hints). If we're going to go through the trouble of rolling our own docker images and testing them, Lambda ceases to be as compelling an option and I'd likely shift our team to Google Cloud Functions (which has been on 3.11 since February) or Fargate.

[frichtarik]

will p3.11 get to the arm supported runtimes ? 3.10 did not get there yet https://docs.aws.amazon.com/lambda/latest/dg/foundation-arch.html

[jtuliani]

@frichtarik That's an oversight in the docs. All runtimes based on AL2 support both ARM and x86_64. I will get the docs updated, thank you for flagging.

@abmohan Thank you for your feedback. We recognize there is strong customer demand for timely runtime releases. We're working hard to catch up.

[mbtx2]

I still think it's appropriate for AWS to provide some expectation on timeline for 3.11 release for base image and 3.11 managed runtime. Your customers need to plan. If you wait too long, we'll plan on moving to another provider. Open-ended timelines work well for AWS and poorly for your customers. It is clear that AWS is putting more resources on this project, but there is still no clear timeline. Even if it's one that is heavily sandbagged, it's better than "we cannot provide a timeline at this time".

[lucaxchaves]

Are there any expectations for this?

The new code build runtime with AL2023 is already available with python 3.11:

https://github.com/aws/aws-codebuild-docker-images/pull/639

[jtuliani]

We are actively working on Python 3.11 support. We're not yet ready to share the container image or ETA, but we will do as soon as we can. As with Python 3.10, we recognize that we've fallen behind on runtime support and are working hard to catch up.

[mat100payette]

@jtuliani While it is understandable why you wouldn't want to make any promises on when it will be available, can't you at least make a rough statement on when it will not be available? I get that making such a statement could be detrimental if for example you say "at least 3 more months minimum probably" and you lose customers but then deliver it earlier than that. However at this point I honestly feel like it would be the bare minimum of transparency from AWS to tell us how long we should expect to be waiting. There is no way you guys don't have a rough planning of the milestones for 3.11, especially given that you just completed 3.10 and know what most of the challenges were.

I appreciate that you at the very least post here, but really there is no substance to most of these posts aside from saying sorry (which is good, but imho not enough for such a cloud leader). Thanks for the hard work.

[BTripp1986]

I get that making such a statement could be detrimental if for example you say "at least 3 more months minimum probably" and you lose customers but then deliver it earlier than that.

I assure you that AWS is not losing customers over release delays of Lambda runtimes(maybe there is some hobby dev somewhere who uses the free tier that is an exception to this), or from a comment in a Github issue.

They are going to give us the information they can, but AWS is a massive company with protocols about these things. AWS did a poor job with 3.10 runtime support, and honestly 3.11 should already be available, but it is what it is and they have acknowledged the problem.

As a community and customer base, let's please not make this issue into a repeat of the 3.10 issue. There is nothing we are going to do to move the understaffed behemoth that is AWS.

[mat100payette]

@BTripp1986 With all due respect, if your preferred solution is to wait silently, that's fine. That being said, please don't tell people not to be vocal about unexplained delays and a blatant lack of transparency from AWS. Sure, there is no point in repeating the same question over and over again, but sadly this is literally the only two-way communication channel we seem to have with AWS Lambda regarding this problematic. People have businesses and jobs, and while the move to 3.11 might not be important to you, it certainly can be for others. Not having the slightest idea of how long we have to wait is quite infuriating, and imo isn't justified by any lack of staff.

Not being able to say "it will be ready by X" is fine; they've got lots to do and probably unknowns to deal with. Not being able to say "it most likely won't be ready before X" is another thing... if they have any sort of decent planning, regardless of the unknowns, they should be able to know if 10 weeks or 10 months will be the strict minimum time for completion.

If the notifications from this issue bother you or anyone, I would recommend unsubcribing and maybe monitoring the What's New section of AWS (or their rss feed) instead.

[petarlishov]

I have to admit, I really like the idea of stating that we should not expect a solution until X amount of time into the future and to come back then regarding a status update. That achieves two benefits:

• it relaxes people from constantly checking on this issue and tells them that there is no need to even ask until X amount of time into the future for a status update
• it is easier on the AWS team working on this as well as it means they do not have to constantly message back like a robot saying it is being worked on. This of course is only the case if work has indeed actively begun on this

If AWS allows for it in its policies, it would be useful to share this information as we all know plans are always wrong but if we all admit that and keep on reiterating on them, it still allows us to get a shared picture of what to expect. With constant reiteration we will always get slightly closer to the truth and right now we are being provided with no plans.

I can also imagine a case where AWS does not allow these plans to be shared as they could be used as an incentive for competitors to beat AWS to certain features but in this case I think there is no concern over that, a decent amount of competitor serverless solutions support Python 3.11 out of the box already.

[Vladislav-Bartalevich]

We are actively working on Python 3.11 support. We're not yet ready to share the container image or ETA, but we will do as soon as we can. As with Python 3.10, we recognize that we've fallen behind on runtime support and are working hard to catch up.

Any updates so far? It's been a month :(

[Rogalek]

how is it going with this?

[jtuliani]

We have published an updated Lambda container base image for Python 3.11, for both x86_64 and arm64 architectures. This image is suitable for production workloads.

Instructions for how to use Lambda container base images for Python are given in the ‘Usage’ tab at https://gallery.ecr.aws/lambda/python.

This release also addresses the issue described in #8.

We expect to publish managed runtime support for Python 3.11 within 60 days.

[jtuliani]

We have completed the launch of Python 3.11 managed runtime support in AWS Lambda: https://aws.amazon.com/blogs/compute/python-3-11-runtime-now-available-in-aws-lambda/

Thanks all for your feedback and patience.

[JoelWR-Diligent]

Is there a timeline for this image to be available in govcloud?

[stewartsmith]

On Sep 12, 2023, at 4:04 AM, frichtarik @.***> wrote:

bit offtopic but related - will 3.11 get into amazon-linux-extras ? thx

It is available on Amazon Linux 2023, we have no plans to add it to AL2.