search

aws / aws-lambda-base-images

Apache License 2.0
671 stars 110 forks source link

Add support for Python 3.10 #31

Closed michael-k closed 1 year ago

michael-k commented 2 years ago

Latest official update for anyone who missed it in the wall of comments:

We have today [2023-02-28] published a Lambda container base image for Python 3.10, for both x86_64 and arm64 architectures. This image is suitable for production workloads.

Instructions for how to use Lambda container base images for Python are given in the ‘Usage’ tab at https://gallery.ecr.aws/lambda/python.

We expect to publish managed runtime support for Python 3.10 within 90 days.

Previous official update:

We recognize the extended delay in providing Python 3.10 support in AWS Lambda, and apologize for the inconvenience this has caused. Our plan is to publish a base container images for Python 3.10 this month [February 2023], with the managed runtime and tooling support (SAM CLI, CFN, CDK, AWS CLI, console, etc) support to follow. We will provide an ETA for the managed runtime when we publish the container image.

Python 3.10 was released on 2021-10-04

jtuliani commented 1 year ago

We have today published a Lambda container base image for Python 3.10, for both x86_64 and arm64 architectures. This image is suitable for production workloads.

Instructions for how to use Lambda container base images for Python are given in the ‘Usage’ tab at https://gallery.ecr.aws/lambda/python.

We expect to publish managed runtime support for Python 3.10 within 90 days.

hb2638 commented 1 year ago

Thx for this!!! Is there a guesstimate for 3.11? Currently using docker based images and I really want to go back to deploying lambas via zip files.

jtuliani commented 1 year ago

@bh2638 Python 3.11 is tracked in #62. We don't have a ETA to share as yet.

mwgamble commented 1 year ago

Thank you! You've made my day :D

konrad0101 commented 1 year ago

Out of curiosity, what was the reason for the release of 3.10 taking so long?

alexandervandekleutab commented 1 year ago

This is perfect, I just needed this today and was glad to see it was also released today.

mt-ronkorving commented 1 year ago

@konrad0101 Two theories mentioned here: https://www.lastweekinaws.com/blog/aws-is-asleep-at-the-lambda-wheel/ by @QuinnyPig

berzi commented 1 year ago

If this is done shouldn't the issue be closed?

HWiese1980 commented 1 year ago

If your definition of done is "it's announced", then this issue would never have been needed in the first place. The managed runtime is not available yet. So it is not done yet, and, as a consequence, no, the issue should not be closed. It can be closed when the managed runtime is actually available as in "we can use it and it works".

DilLip-Chowdary-Codes commented 1 year ago

We have today published a Lambda container base image for Python 3.10, for both x86_64 and arm64 architectures. This image is suitable for production workloads.

Instructions for how to use Lambda container base images for Python are given in the ‘Usage’ tab at https://gallery.ecr.aws/lambda/python.

We expect to publish managed runtime support for Python 3.10 within 90 days.

@jtuliani , we got this msg 2 days back.

Starting March 30, 2023, AWS CodeBuild will be moving these images (Ubuntu standard 5.0 [1] Docker Image, Amazon Linux 2 ARM standard 1.0 [2], or Amazon Linux 2 standard 3.0 [3]) to an unsupported status and they will not be cached on the build hosts anymore.

You may continue using these images for your builds, but you will notice an increase in provisioning latency after March 30, 2023.
These images will also not be getting any new updates. We recommend updating your Build Projects to use the latest build images in order to get the latest language runtimes and tools.

We need to update our codebuild projects to use standard6.0 images to avoid increase in provisioning latency.

but we have a dependency on AWS Lambda python3.10 (otherwise we need to install custom python runtime and use for our project) for this as 3.10 is the only runtime supported in standard6.0

fox91 commented 1 year ago

@DilLip-Chowdary-Codes how is your problem related to Lambda? Maybe you would like to open an issue in the appropriate repo?

michael-k commented 1 year ago

how is your problem related to Lambda?

Depends how you deploy your Lambda functions. When you run tests for that Python 3.9 Lambda code in CodeBuild, you might want to do that using Python 3.9. If you use Python 3.10 instead you might miss something, eg. syntax that's valid in Python 3.10, but not Python 3.9.

CodeBuild images supported multiple Python versions for a long time. But with the release of the image Ubuntu standard:6.0. As you can see in the list of available runtimes, Ubuntu standard:6.0 is the only image that supports Python 3.10, but Python 3.10 is also the only version of Python supported by Ubuntu standard:6.0.

The delay (of almost 1.5 years now) of Python 3.10 on Lambda is now causing issues when combing Lambda it with other AWS services.

Maybe you would like to open an issue in the appropriate repo?

In my experience, AWS is ignoring issues in that repository even more than runtime requests here. Issues on GitHub repositories doesn't seem to be a metric that AWS cares about.

Update

This is (part of) the answer I got from AWS' support:

To ensure an optimal experience for our customers, CodeBuild will NOT be deprecating the Ubuntu Standard 5.0 image. We will be sending an updated announcement to clarify this soon.

LouisAmon commented 1 year ago

Seems like this issue has been fixed https://hub.docker.com/r/amazon/aws-lambda-python/tags?page=1&name=3.10

Sighery commented 1 year ago

@LouisAmon https://github.com/aws/aws-lambda-base-images/issues/31#issuecomment-1455019435

HWiese1980 commented 1 year ago

@LouisAmon Not fixed! image

See! No Python 3.10! As long as there is no Python 3.10 in that very list and any lambda built with it works as expected, the issue is not fixed.

LouisAmon commented 1 year ago

Fair enough guys, sorry

In my company we only ever use Lambda containers

(And I do believe that it should be the best practice, especially in Python where you might have some C/C++ system dependencies every now and then)

In any case I tested the new 3.10 tag on graviton and it works wonders ✨

HWiese1980 commented 1 year ago

No need to apologize. We're all a little bit, let's say, concerned about and irritated by Amazon's complete negligence.

CGarces commented 1 year ago

(And I do believe that it should be the best practice, especially in Python where you might have some C/C++ system dependencies every now and then)

In any case I tested the new 3.10 tag on graviton and it works wonders ✨

If you not use .zip functions you are loosing a hidden gem on AWS lambda... https://bitesizedserverless.com/bite/when-is-the-lambda-init-phase-free-and-when-is-it-billed/

HWiese1980 commented 1 year ago

As long as Python 3.10 does not appear in this list, it is not supported by Lambda, regardless of whatever container is available. ZIP based Lambda still does not support 3.10 yet, and that's what we're asking for here.

image (Updated just now; still no Python 3.10).

Yes, there is indeed a base image for Lambda with Python 3.10. However, it is slower than ZIP Lambda, does not really fulfill the "managed" claim and for small Lambdas it is more like a workaround than an actual solution. Its main purpose, imho, is to run in bigger Lambdas that exceed some limitations of ZIP Lambdas and that do not have to run often and may take some time for orchestration. And they increase maintenance cost because you have to build and maintain a whole container instead of just a zipped source folder.

And last but not least, Lambda is sold primarily as managed service where you just upload your code and it gets executed in the cloud. Simple as that. Containerized Lambda does not keep this promise.

Hence, please stop asserting Python 3.10 was supported now just because there's a base image.

epierce78 commented 1 year ago

As long as Python 3.10 does not appear in this list, it is not supported by Lambda, regardless of whatever container is available. ZIP based Lambda still does not support 3.10 yet, and that's what we're asking for here.

image (Updated just now; still no Python 3.10).

Yes, there is indeed a base image for Lambda with Python 3.10. However, it is slower than ZIP Lambda, does not really fulfill the "managed" claim and for small Lambdas it is more like a workaround than an actual solution. Its main purpose, imho, is to run in bigger Lambdas that exceed some limitations of ZIP Lambdas and that do not have to run often and may take some time for orchestration. And they increase maintenance cost because you have to build and maintain a whole container instead of just a zipped source folder.

And last but not least, Lambda is sold primarily as managed service where you just upload your code and it gets executed in the cloud. Simple as that. Containerized Lambda does not keep this promise.

Hence, please stop asserting Python 3.10 was supported now just because there's a base image.

Amen. Not usually a big proponent of piling on, but it's utterly ridiculous we're in this place. When you look at timeline, etc. not having at least Python 3.10 available in that dropdown is ludicrous at this point. And our teams (myself included), are chomping to deploy 3.11 for I would think obvious reasons.

So, here's another plea: please AWS, get your stuff together and support these runtimes. I don't want to add building containers, etc. to our pipelines either. Not why I like using Lambdas.

sdaves commented 1 year ago

@jtuliani Thank you for the update on the 90 day delivery from Feb 28th.

I understand you are just one person within the biggest hosting company that has ever existed, and any information you can give to developers at large, must be guarded by expectations.

Do you personally think we are still looking at a publish on or before May 28, 2023?

jetttz commented 1 year ago

I think May 28, isn't feasible, might be 2x that

PierreKiwi commented 1 year ago

Halfway through the 90 day period, can we have a new update ? Surely some progress has been made and the GA date is now set in stone, right ?

GrahamCampbell commented 1 year ago

You can ask your TAM.

jtuliani commented 1 year ago

We remain on track to deliver Python 3.10 GA within the schedule communicated previously. These releases contain multiple components (runtime, SDK, CDK, SAM, CLI, CloudFormation, console, documentation, etc), which are delivered across multiple teams. As a result, it would not be appropriate to give a more specific date since plans can and do change when operating 24/7 services.

rcoops commented 1 year ago

We remain on track to deliver Python 3.10 GA within the schedule communicated previously.

@jtuliani to be clear, this schedule:

I think May 28, isn't feasible, might be 2x that

... Or the original one?

We expect to publish managed runtime support for Python 3.10 within 90 days.

Appreciate any eta you could give is just that: an estimate. Would be good to know if we're looking at roughly May/June or August/September

SimonSchick commented 1 year ago

Toasting in an epic thread 👌

berzi commented 1 year ago

Additional question: is it realistic to expect future versions after 3.10 to release faster?

HWiese1980 commented 1 year ago

At that rate we have Python 3.12 when Python 4.12 is already end of life.

Seriously, sorry to say so, but this is getting ridiculous.

tzmara commented 1 year ago

At that rate we have Python 3.12 when Python 4.12 is already end of life.

Seriously, sorry to say so, but this is getting ridiculous.

Keep in mind that as a startup they only have limited resources for professional and regular updates.... 🥸

robmoss2k commented 1 year ago

jtuliani commented 3 days ago We remain on track to deliver Python 3.10 GA within the schedule communicated previously. These releases contain multiple components (runtime, SDK, CDK, SAM, CLI, CloudFormation, console, documentation, etc), which are delivered across multiple teams. As a result, it would not be appropriate to give a more specific date since plans can and do change when operating 24/7 services.

@jtuliani - to quote Jeff Bezos:

Jeff Bezos stated 10 years ago: No! Communication is terrible!

The failure to deliver Python 3.10 (and 3.11) support for Lambda functions is a direct result of a failure to adhere to the guiding principles that made Amazon (and AWS) so successful in the first place. Define the interface between your team and the rest and cut out the social loafing. Other teams (Kendra, AppFlow, EKS) are iterating rapidly, and yours is not. I know you've introduced some nice features this year, such as response payload streaming, Amazon DocumentDB change streams as an event source, Lambda Powertools for .NET, the new Async metrics, runtime management controls, and setting Maximum Concurrency to the Amazon SQS event source, but the pace of new runtime support is utterly glacial across all languages. Lambda will become a useless service as the libraries leave its latest runtime version behind.

Solve the MIP: make sure Lambda can run our code.

GrahamCampbell commented 1 year ago

Maybe the plan is intentionally to make native support shit so that the community make their own stuff for free. That worked with PHP... https://github.com/brefphp.

josh-forum commented 1 year ago

I held back until this point about people who post irrelevant and been hurtful comments…. but the use of your that ugly word you just used made me finally comment.

I subscribed to this issue to get relevant updates. AWS have been more than gracious them.

@GrahamCampbell You on the other hand are making uncouth comments that are not only show up in my email box but I am sure others as well, that don’t appreciate getting them.

On top of that, I am sure it will make the people at aws reluctant to post updates and relevant info on the work in the future.

If you, and others like you, could please reframe from such loathsome comments I would be very appreciative.

GrahamCampbell commented 1 year ago

Truth hurts. As someone who gives many hours per week to the community for free, such as to the Bref project (not that I speak on behalf of anyone else involved with that project), and get nothing back from AWS in return, I am more than in a position to say this.

andrewCMU commented 1 year ago

Committing your free time to the community can in fact be done without using it as an excuse to be a jackass. Just my 2 cents, I’ll go back to lurking now

jbddc commented 1 year ago

I'm sorry, but Is it really that big of an ask to at least get some SLAs for a service we pay for??

chrissype commented 1 year ago

irrelevant and been hurtful comments

anger at the behaviour of amazon with regards to this issue, which in my view implicates them in conducting anti-competitive business practices, is a perfectly rational response.

chrissype commented 1 year ago

Article for the curious. We should be challenging behaviour like this, not saying 'poor amazon'.

https://www.theguardian.com/business/2023/apr/05/amazon-and-microsoft-face-referral-to-uk-regulator-over-cloud-services-market

dennybiasiolli commented 1 year ago

Under the hood it could be something "Debian stable"-related? The current stable version of Debian contains Python 3.9 by default, so "maybe" they are using Debian and they are stuck with 3.9 by default until the release of the next stable version (Bookworm)?

Info here:

systematicguy commented 1 year ago

Under the hood it could be something "Debian stable"-related? The current stable version of Debian contains Python 3.9 by default, so "maybe" they are using Debian and they are stuck with 3.9 by default until the release of the next stable version (Bookworm)?

Info here:

No matter the reason, this is why we pay for cloud services. They take on this risk, we pay more for those abstractions.

berzi commented 1 year ago

Article for the curious. We should be challenging behaviour like this, not saying 'poor amazon'.

https://www.theguardian.com/business/2023/apr/05/amazon-and-microsoft-face-referral-to-uk-regulator-over-cloud-services-market

I don't think anyone is saying that. It's clear that it's their own inefficiency and a problem to solve asap, but at this point we know they're on it and belittling or insulting them won't make the features come out any faster. It does however clutter about 80 people's inboxes unnecessarily. I think our motivations as users are clearly stated at this point, as is the fact that they're working on it, so how about we let some time pass before making the thread even longer and less useful to read? (This is not directed at anyone in particular.)

astuyve commented 1 year ago

image Looks like it just landed? At least in us-east-1.

jetttz commented 1 year ago

thank you it's live now

LucasHartridge commented 1 year ago

Thanks team 🙏, it is live now

dliwustl commented 1 year ago

Did they upgrade the manylinux platform version for 3.10? I believe it was manylinux2010_x86_64 in 3.9. Did they go to 2014? I'm not sure where to look to figure that out, but I want to know if I can change the platform when building Layers.

kinghuang commented 1 year ago

So, that's ~18 months~ 20 months to go from 3.9 to 3.10? I shudder to think how long 3.11 will take.

Reference: AWS Lambda adds support for Python 3.9 (Aug 16, 2021)

tongclement commented 1 year ago

Finally! 🎉 Now onto 3.11

mnapoli commented 1 year ago

Just a quick note since https://github.com/brefphp was mentioned. As the author of Bref, while I understand (and share) the frustration, I don’t condone aggressive comments. We can use better words to express that. Just wanted to clarify, sorry for the off-topic comment.

re-thc commented 1 year ago

Under the hood it could be something "Debian stable"-related?

It's Amazon Linux everywhere for AWS i.e. Redhat...

miketheman commented 1 year ago

Released: https://aws.amazon.com/blogs/compute/python-3-10-runtime-now-available-in-aws-lambda/