search

aws / aws-lambda-base-images

Apache License 2.0
648 stars 107 forks source link

Add support for Python 3.10 #31

Closed michael-k closed 1 year ago

michael-k commented 2 years ago

Latest official update for anyone who missed it in the wall of comments:

We have today [2023-02-28] published a Lambda container base image for Python 3.10, for both x86_64 and arm64 architectures. This image is suitable for production workloads.

Instructions for how to use Lambda container base images for Python are given in the ‘Usage’ tab at https://gallery.ecr.aws/lambda/python.

We expect to publish managed runtime support for Python 3.10 within 90 days.

Previous official update:

We recognize the extended delay in providing Python 3.10 support in AWS Lambda, and apologize for the inconvenience this has caused. Our plan is to publish a base container images for Python 3.10 this month [February 2023], with the managed runtime and tooling support (SAM CLI, CFN, CDK, AWS CLI, console, etc) support to follow. We will provide an ETA for the managed runtime when we publish the container image.

Python 3.10 was released on 2021-10-04

brett-matthews commented 1 year ago

I'd really love to know what the difficulties and obstacles are...

Agreed if AWS cant provide a rough timeframe, then at least a brief run down on why this has taken so long seems reasonable?

Rodriguevb commented 1 year ago

Any news?

rohanshiva commented 1 year ago

@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update fails on a self-hosted Gitlab.)

excellent idea: https://gist.github.com/jwgstr/4dcc9f16c52529486de0f6fa78c55aef

Thank you. I took your solution and copied over some files that I saw in AWS's 3.9 image as well as made the environment variabels match.

My POC is @ https://github.com/hb2638/aws_lambda_python_3.11

I tested it out an a simple AWS lambda that returns the event that was passed in. I tried in on some other code but got blocked because I can't pip install pyarrow (There are no wheels and there are just too many things I need to do to get it to work)

@hb2638 any way I can use this as a layer and then upload my code in another layer?

hb2638 commented 1 year ago

@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update fails on a self-hosted Gitlab.)

excellent idea: https://gist.github.com/jwgstr/4dcc9f16c52529486de0f6fa78c55aef

Thank you. I took your solution and copied over some files that I saw in AWS's 3.9 image as well as made the environment variabels match.

My POC is @ https://github.com/hb2638/aws_lambda_python_3.11

I tested it out an a simple AWS lambda that returns the event that was passed in. I tried in on some other code but got blocked because I can't pip install pyarrow (There are no wheels and there are just too many things I need to do to get it to work)

pyarrow released the wheels and all of our lambdas seem to be working on 3.11 (we have about 60 of them). The automated tests are also working on 3.11.. Had to make a few tweaks to the DOCKERFILE because urlibrequest wasn't working with some ssl certs and pandas was unable to import sqlite3.

I'm baffled why it's taking AWS so long to do similar work on their end to let use target python 3.10 and python 3.11 for zip files. I was hoping this excersice would have uncovered some blocker (E.x.: Some issue in awslambdaric) to justify why it's taking so long... but that hasn't happened.

mrlonis commented 1 year ago

Google Cloud Platform supports Python 3.10: https://cloud.google.com/functions/docs/concepts/python-runtime

keithly commented 1 year ago

Here's my own spin on using a custom runtime for a newer version of Python: https://github.com/keithly/lambda-python-custom The Dockerfile verifies PGP signatures for openssl and python.

tongclement commented 1 year ago

+1 I am in shock when I realised that AWS lambda doesn't support python310 today! Now I have to rewrite some of the FastAPI bits to make it work with python39. Will consider switching to GCP cloud run next time when I have more experience with writing more advanced GitHub actions.

ztane commented 1 year ago

Ok, seriously now, Node 18 already released as a lambda runtime?! even though it was released more than half a year later than Python 3.10. If only Python 3.10 was insignificant but it isn't, because of the match statement that is absolutely necessary when matching all these ridiculous nested dictionaries in lambda payloads.

ttamg commented 1 year ago

And I see that Azure Functions are now doing a Python 3.10 in preview. Come on AWS do you really want us to move to Azure? .... I don't really want to.

kneipp commented 1 year ago

Too late, release python 3.11!

HWiese1980 commented 1 year ago

Too late, release python 3.11!

Agreed! It does not make much sense anymore to focus on Python 3.10. Python 3.11 is out, production ready and mature enough. Python 3.10 could then be a simple by-product.

rafrafek commented 1 year ago

According to @baotran2207 https://github.com/aws/aws-lambda-base-images/issues/62#issuecomment-1375064131 Python 3.11 should be official supported in Feb.

Will it ship together with 3.10?

Sighery commented 1 year ago

According to @baotran2207 #62 (comment) Python 3.11 should be official supported in Feb.

Will it ship together with 3.10?

Are you sure about that? As far as I can tell, it doesn't say anywhere in his profile that he works for AWS. His LinkedIn, also available in his profile, doesn't mention working for AWS either

jtuliani commented 1 year ago

We are working on Py3.10 support and will share further updates as work progresses. Py3.11 will ship separately, this is tracked in #62.

realchandan commented 1 year ago

Today Recently, 3.12 (not stable) was released! :)

Tenzer commented 1 year ago

What makes you think that? 3.12a4 was released last week. They haven't started releasing beta version of 3.12 yet.

relsunkaev commented 1 year ago

We are working on Py3.10 support and will share further updates as work progresses. Py3.11 will ship separately, this is tracked in #62.

If it is possible, could you speak of the process for introducing new Python environments for AWS Lambda? What are some difficulties you are experiencing? It may seem "simple" to some, but I imagine there are a lot of issues people wouldn't even think of stemming from compliance alone. I think the insight would be really helpful for some.

It is totally understandable if you can't speak to this, of course.

realchandan commented 1 year ago

Here's how to use any version of python [I'm using 3.11.1), I'm writing this as It may be helpful for someone

from subprocess import run

def my_cute_lambda_handler(event, context):
    return f"Hello World, running on {run(['python', '--version'], capture_output=True, text=True).stdout.strip()}"
FROM python:3.11.1-bullseye

ENV APP_HOME="/app"
COPY . ${APP_HOME}
WORKDIR ${APP_HOME}

RUN pip install --target ${APP_HOME} awslambdaric

RUN curl -LO https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie && chmod +x aws-lambda-rie

ENTRYPOINT [ "./aws-lambda-rie", "python", "-m", "awslambdaric" ]

CMD [ "main.my_cute_lambda_handler" ]

now run your image locally and you can invoke lambda using curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

Remove RUN curl -LO https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie && chmod +x aws-lambda-rie and also update ENTRYPOINT ["python", "-m", "awslambdaric" ] when deploying to actual lambdas. (these code are for local emulator)

Note ->

relsunkaev commented 1 year ago

Here's how to use any version of python [I'm using 3.11.1), I'm writing this as It may be helpful for someone...

Thank you! This will definitely be helpful for some. Word of warning though, docker image lambdas have 2-3x slower cold starts than normal lambdas.

feoh commented 1 year ago

Folks this is becoming more urgent as time passes. Python 3.8 is becoming very difficult to install by default as it's being eliminated from the various Linux distributions. I know you support 3.9 but lots of folks are waiting to upgrade since 3.10 & 3.11 represent some fairly significant feature additions.

jakob-keller commented 1 year ago

Maybe the delay is related to today's announcement of AWS Lambda runtime management controls? I am just guessing, but it would make sense to focus effort on those management controls and THEN ship Python 3.10 and 3.11 runtimes with management support from the start. Any thoughts?

feoh commented 1 year ago

Having just recently left Amazon myself I can tell you that getting any given piece of work funded can be a real fight if the bizdev folks don't see the ROI.

HWiese1980 commented 1 year ago

Crazy, since the overall return goes to 0 if we can't use AWS Lambda anymore when every dev system ended support for 3.9 before Lambda started supporting 3.10.

Not to mention that there are other cloud providers...

SimonSchick commented 1 year ago

Also looking to move elsewhere due to this, we don't want to deal with maintaining our own docker images and the related perf impact, this wild to me, does AWS not see value in this, would be nice if a PM could respond..

feoh commented 1 year ago

I am quite sure that the rank-and-file engineers at AWS are very keenly aware of this, and have probably been bugging the crap out of the business people to try to get it funded. I've been in this exact same position myself in the past, and I can assure you it is not fun. Doesn't make the pain go away as a customer, but still…

mrtysn commented 1 year ago

It is unfortunate that we cannot even get a tentative date on something you would consider to be a rudimentary upgrade. We will have to separate our python processes to other cloud providers unless we can find a viable solution. What is more frustrating is the lack of a clear feedback cycle between its customers and the provider.

csmykay commented 1 year ago

This is now effecting us to be able to deploy consistent models with Lambda. We have other models that use Python 3.10 in a container which of course work fine. But when we switch to ZIP file for some of our models that use regex under the covers we get: [ERROR] Runtime.ImportModuleError: Unable to import module 'my_funtion_name': No module named 'regex._regex' Traceback (most recent call last):

Otherwise we have to switch to all containers which means greater memory constraint.

kadyrleev commented 1 year ago

Incredibly frustrating.

FredStober commented 1 year ago

FYI: The lack of a python 3.10 runtime prevents us in a project to move from AWS Glue 3.0 (python 3.7) to Glue 4.0 (python 3.10) since we want a consistent set of libraries in both lambda functions and glue jobs.

AhmedMRaihan commented 1 year ago

This situation of not being able to upgrade or use latest version and being hopeless, totally resembles the horrific old days of Vendor Locking.

ttamg commented 1 year ago

The question is ‘how do we get this concern on the radar of the right people at AWS?’. It’s clear a lot of us are not happy and it is making things difficult but I guess no-one with clout at AWS is reading our issue thread in GitHub.

Does anyone know how issues or feature requests get triaged within AWS? Then perhaps we can all bother them and see if that moves something forward?

As a side note … I’ve moved some toy examples to Azure Functions now and it is actually fine and nice to use … it’s just the broader Azure setup I don’t like as much. Sorry AWS but you are being too slow.

Matt

On 2 Feb 2023 at 07:23:53, Ahmed M Raihan @.***> wrote:

This situation of not being able to upgrade or use latest version and being hopeless, totally resembles the horrific old days of Vendor Locking.

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-lambda-base-images/issues/31#issuecomment-1413263974, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2ZJEXH47BETV5EZP3APADWVNOITANCNFSM5KQEVLWQ . You are receiving this because you commented.Message ID: @.***>

QuinnyPig commented 1 year ago

Let me take a whack at it…

tzmara commented 1 year ago

realchandan

Wow seriously thank you - didn't expect that to be THAT easy... I don't get where the problem for aws is now?

jtuliani commented 1 year ago

We recognize the extended delay in providing Python 3.10 support in AWS Lambda, and apologize for the inconvenience this has caused. Our plan is to publish a base container images for Python 3.10 this month, with the managed runtime and tooling support (SAM CLI, CFN, CDK, AWS CLI, console, etc) support to follow. We will provide an ETA for the managed runtime when we publish the container image.

acdha commented 1 year ago

@jtuliani thank you, it’s appreciated! I’m sure the team has a lot going on behind the scenes but this really helps at large organizations considering whether to rely on Lambda long-term.

QuinnyPig commented 1 year ago

ETAs are great, but it may be time for SLAs instead. This is the first time I've found myself refactoring code for an earlier language version to run in a supported non-custom Lambda runtime. A ball has been seriously dropped here.

Rizhiy commented 1 year ago

While we are on the topic, getting 3.11 runtime would be great as well.

HWiese1980 commented 1 year ago

While we are on the topic, getting 3.11 runtime would be great as well.

They are aware of that, there's another issue regarding Python 3.11: #62

jakob-keller commented 1 year ago

Mildly educated guess as to why Python 3.10 poses an engineering challenge for AWS: PEP 644 – Require OpenSSL 1.1.1 or newer.

Why? Lambda runtimes are based on Amazon Linux 2 since python3.8 and appear to be built around OpenSSL 1.0.2k-fips, which is incompatible with Python 3.10 due to PEP 644. (it's actually OpenSSL 1.1.1o according to https://github.com/aws/aws-lambda-base-images/issues/31#issuecomment-1425954955)

Amazon Linux 2022 ships up-to-date OpenSSL versions and would thus be a good OS candidate for future Lambda runtimes. It is still in preview and has been for quite a while. Maybe AWS is waiting for Amazon Linux 2022 to become GA before releasing new Lambda runtimes.

By the way, CodeBuild is suffering similar delays in supporting current runtimes, in particular the AArch64 flavour.

obataku commented 1 year ago

@jakob-keller the python3.9 runtime as of right now at least claims to ship with OpenSSL 1.1.1o 3 May 2022 (see ssl.OPENSSL_VERSION)

Bohynskyi commented 1 year ago

Hi, @jtuliani do you have any update on this topic?

kamikaze commented 1 year ago

skip 3.10, it is the time for 3.11 already!

jetttz commented 1 year ago

@jtuliani any ETA?

tzmara commented 1 year ago

I gave up and created my own 3.11 images. Very good experience. Faster bootup, smaller image, better performance. The explanation how to do this was a little bit above here in the thread. Took me about 30mins to adapt the image to 3.11 based on the official python image.

From: Jet @.> Sent: Monday, February 27, 2023 7:13:39 PM To: aws/aws-lambda-base-images @.> Cc: Torsten Zielke @.>; Comment @.> Subject: Re: [aws/aws-lambda-base-images] Add support for Python 3.10 (Issue #31)

any ETA?

— Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-lambda-base-images/issues/31#issuecomment-1446816827, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AY5WJT7HBB2YXRA4MYL55P3WZTVFHANCNFSM5KQEVLWQ. You are receiving this because you commented.Message ID: @.***>

mousedownmike commented 1 year ago

If anyone else got here after upgrading to Ubuntu 22.04 (3.10 Python default) and needs to get sam package working again, these steps got me running again:

  1. Install the deadsnakes PPA and python3.9 (note: I had to disable IPv6 to get the PPA to add)
  2. Install the python3.9-distutils sudo apt install python3.9-distutils, this will resolve the ModuleNotFoundError: No module named 'distutils.cmd' problem.

On a side note, I find this very embarrassing. I've been promoting these tools to my customers with the promise that AWS was going to take care of this "undifferentiated heavy lifting". If AWS can't take care of this basic maintenance, it makes you wonder how much we can trust that the real heavy lifting is being taken care of.

tzmara commented 1 year ago

If anyone else got here after upgrading to Ubuntu 22.04 (3.10 Python default) and needs to get sam package working again, these steps got me running again:

  1. Install the deadsnakes PPA and python3.9 (note: I had to disable IPv6 to get the PPA to add)
  2. Install the python3.9-distutils sudo apt install python3.9-distutils, this will resolve the ModuleNotFoundError: No module named 'distutils.cmd' problem.

On a side note, I find this very embarrassing. I've been promoting these tools to my customers with the promise that AWS was going to take care of this "undifferentiated heavy lifting". If AWS can't take care of this basic maintenance, it makes you wonder how much we can trust that the real heavy lifting is being taken care of.

Sad but true :/

orfisko commented 1 year ago

I gave up and created my own 3.11 images. Very good experience. Faster bootup, smaller image, better performance. The explanation how to do this was a little bit above here in the thread. Took me about 30mins to adapt the image to 3.11 based on the official python image. … ____ From: Jet @.> Sent: Monday, February 27, 2023 7:13:39 PM To: aws/aws-lambda-base-images @.> Cc: Torsten Zielke @.>; Comment @.> Subject: Re: [aws/aws-lambda-base-images] Add support for Python 3.10 (Issue #31) any ETA? — Reply to this email directly, view it on GitHub<#31 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AY5WJT7HBB2YXRA4MYL55P3WZTVFHANCNFSM5KQEVLWQ. You are receiving this because you commented.Message ID: @.***>

I got from earlier posts that using a custom runtime leads to longer cold start times? Are you saying now this is not the case in your experience? If so, I would also consider investing some time in setting up my own runtime. If we need to wait for AWS to finish writing all their marvelous blog posts and presentations on how to optimize usage of their services instead of cracking on with what they should actually be doing we're all dead.

tzmara commented 1 year ago

I gave up and created my own 3.11 images. Very good experience. Faster bootup, smaller image, better performance. The explanation how to do this was a little bit above here in the thread. Took me about 30mins to adapt the image to 3.11 based on the official python image. … ____ From: Jet @.**> Sent: Monday, February 27, 2023 7:13:39 PM To: aws/aws-lambda-base-images @.**> Cc: Torsten Zielke @.**>; Comment @.**> Subject: Re: [aws/aws-lambda-base-images] Add support for Python 3.10 (Issue #31) any ETA? — Reply to this email directly, view it on GitHub<#31 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AY5WJT7HBB2YXRA4MYL55P3WZTVFHANCNFSM5KQEVLWQ. You are receiving this because you commented.Message ID: @.***>

I got from earlier posts that using a custom runtime leads to longer cold start times? Are you saying now this is not the case in your experience? If so, I would also consider investing some time in setting up my own runtime. If we need to wait for AWS to finish writing all their marvelous blog posts and presentations on how to optimize usage of their services instead of cracking on with what they should actually be doing we're all dead.

For me personally the self-built images are now faster. But we used a custome build image with the aws base before. So this might not be comparable.

A very minimal approach:

# maybe alpine is even faster/tinier, but most of the time it takes more work to 
# do. So this way we have it a little bit easier ;)
FROM python:3.11-slim-bullseye  

ENV LAMBDA_TASK_ROOT=/var/task \   # that's the default by aws - feel free to change
    AWS_ACCESS_KEY_ID \
    AWS_SECRET_ACCESS_KEY

WORKDIR $LAMBDA_TASK_ROOT

RUN pip install --target $LAMBDA_TASK_ROOT awslambdaric

# This should contain lambda.py
COPY src/ $LAMBDA_TASK_ROOT/

# Obviously loading/installing requirements.txt/poetry settings should be done as well

# Precreate all the pyc files. This way the container needs few seconds more to build,
# but is starting a tiny bit faster
RUN python -m compileall .

ENTRYPOINT [ "python", "-m", "awslambdaric" ]

CMD ["lambda.handler"]

I stripped our own logic, so it might need tweaking, but feel free to ask, if there are problems :)

hsoiland commented 1 year ago

Piling on here. Can we get an eta?

jbddc commented 1 year ago

Just saw this on dockerhub:

https://hub.docker.com/layers/amazon/aws-lambda-python/3.10/images/sha256-5a3958ff08e49f20a36358bc5c4916d701471faa9b51f3db47a9fe157d653eda?context=explore