search

aws / aws-lambda-base-images

Apache License 2.0
648 stars 107 forks source link

Add support for Python 3.10 #31

Closed michael-k closed 1 year ago

michael-k commented 2 years ago

Latest official update for anyone who missed it in the wall of comments:

We have today [2023-02-28] published a Lambda container base image for Python 3.10, for both x86_64 and arm64 architectures. This image is suitable for production workloads.

Instructions for how to use Lambda container base images for Python are given in the ‘Usage’ tab at https://gallery.ecr.aws/lambda/python.

We expect to publish managed runtime support for Python 3.10 within 90 days.

Previous official update:

We recognize the extended delay in providing Python 3.10 support in AWS Lambda, and apologize for the inconvenience this has caused. Our plan is to publish a base container images for Python 3.10 this month [February 2023], with the managed runtime and tooling support (SAM CLI, CFN, CDK, AWS CLI, console, etc) support to follow. We will provide an ETA for the managed runtime when we publish the container image.

Python 3.10 was released on 2021-10-04

andreif commented 2 years ago

Please?

volphy commented 2 years ago

Any implementation timeline?

nikitajz commented 2 years ago

some features look interesting, looking forward!

leslie-alldridge commented 2 years ago

keen !

fadesibert commented 2 years ago

+1 as much as I love building python from source - maybe 3.7 is a little long in the tooth to be the standard...

alimcmaster1 commented 2 years ago

+1 as much as I love building python from source - maybe 3.7 is a little long in the tooth to be the standard...

Agree - looks like @jbesw did the work for 3.9... maybe he can help with 3.10 support?

https://aws.amazon.com/blogs/compute/python-3-9-runtime-now-available-in-aws-lambda/

jtuliani commented 2 years ago

Thank you everyone for your feedback regarding AWS Lambda support for Python 3.10. We are aware of the ask, however, we don't currently have a timeline to share.

acdha commented 2 years ago

Python 3.9 just hit the security-only stage of its maintenance lifecycle:

https://pythoninsider.blogspot.com/2022/05/python-3913-is-now-available.html

HWiese1980 commented 2 years ago

Two months since the last update. Any (hopefully good) news here? @jtuliani

jwgstr commented 2 years ago

I had a desire to update my lambda builds to 3.10 the other day and began looking into this more. Since this issue has been open for a little while I thought I might take a stab at building this from the amazonlinux base image (as the python 3.9 builds are). None of the guides I found quite made this copy / paste possible.

So I came up with this, which I tested and seems to work just fine at least for my use cases which includes things like numpy. But depending on your project complexity your mileage may vary. It doesn't follow all the same conventions with putting task logic in /var/task or app logic in /var/runtime but that didn't cause any conflicts for me. It's kind of a hodgepodge of various AWS docs and articles I found. Wouldn't surprise me if it can be cleaned up a bit.

I don't think I have access to contribute this back anywhere otherwise I would create PR. @jtuliani please feel free to point me in the right direction for that and I'm happy to help iterate on a better vetted and reviewed 3.10 image.

In the meantime I hope this helps others.


ARG FUNCTION_DIR="/app/"
ARG AWS_LINUX_VERSION="2022"
ARG PYTHON_VERSION="3.10.4"

FROM amazonlinux:${AWS_LINUX_VERSION} as python-layer

ARG PYTHON_VERSION

# install python
RUN yum update -y
RUN yum groupinstall "Development Tools" -y
RUN yum install openssl1.1 openssl1.1-devel libffi-devel bzip2-devel wget -y
RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz
RUN tar -xf Python-${PYTHON_VERSION}.tgz
RUN cd Python-${PYTHON_VERSION}/ && \
    ./configure --enable-optimizations && \
    make install
RUN ln -s /Python-${PYTHON_VERSION}/python /usr/bin/python
RUN python -m pip install --upgrade pip

FROM amazonlinux:${AWS_LINUX_VERSION} as base-layer

# copy over python
ARG PYTHON_VERSION
COPY --from=python-layer /Python-${PYTHON_VERSION} /Python-${PYTHON_VERSION}
COPY --from=python-layer /usr/local/bin /usr/local/bin
COPY --from=python-layer /usr/local/lib /usr/local/lib
RUN ln -s /Python-${PYTHON_VERSION}/python /usr/bin/python

ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}

FROM base-layer as build-layer

######## YOUR OWN SETUP PROCESS HERE ########################
# copy over requirements and install those
COPY setup.py .
RUN pip install . --target "${FUNCTION_DIR}"

# copy over configuration and service code then install it
COPY config/ ${FUNCTION_DIR}/config/
COPY service/ ${FUNCTION_DIR}/
RUN pip install . --target "${FUNCTION_DIR}"
######## ########################### ########################

# install lambda runtime interface client for python
RUN pip install awslambdaric --target "${FUNCTION_DIR}"

FROM base-layer as runtime-layer

# copy in the built dependencies
ARG FUNCTION_DIR
COPY --from=build-layer ${FUNCTION_DIR} ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}

# (optional) add lambda runtime interface emulator
#ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
#RUN chmod 755 /usr/bin/aws-lambda-rie
#ENTRYPOINT [ "/usr/bin/aws-lambda-rie", "python", "-m", "awslambdaric" ]

ENTRYPOINT [ "python", "-m", "awslambdaric" ]

######## REFERENCE YOUR OWN HANDLER HERE ########################
CMD [ "main.app" ]```
######## ############################### ########################
matthewdeanmartin commented 2 years ago

@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update fails on a self-hosted Gitlab.)

jwgstr commented 2 years ago

@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update fails on a self-hosted Gitlab.)

excellent idea: https://gist.github.com/jwgstr/4dcc9f16c52529486de0f6fa78c55aef

sjzabel commented 2 years ago

This is now an issue for users of Gitlab who use the Docker:latest image to build lambda images using DinD. The Docker image is built off of Alpine 3.16 which now only supports python3.10 natively.

Our workaround is to use pyenv to install a python 3.9 version... but that adds significant time to our builds.

BjoernPetersen commented 2 years ago

You could also just not use the latest tag and continue with the older image (e.g. the docker:20.10.16-dind-alpine3.15 tag). Using the latest tag is generally not a great idea, because it'll often lead to issues like these in the worst possible moment.

Not saying that it isn't ridiculous for AWS to still not support Python 3.10 in Lambdas, just offering my opinion on best practices when using container images.

sjzabel commented 2 years ago

Let me try that. (Thanks, I had gone through the list of available Docker tags and I didn't see that one.)

Ty, Stephen

obataku commented 1 year ago

now that 3.10 is the only available runtime for the latest codebuild images (https://github.com/aws/aws-codebuild-docker-images/pull/505), one cannot use these to build + package lambdas without manually downgrading or upgrading Python (e.g. via pyenv) in codebuild or lambda, respectively, or risk bundling incompatible C extension binaries between build and deployment. @jtuliani @jbesw any updates?

mikelane commented 1 year ago

Half way around the sun since this was opened and 10 months since 3.10 final was released. Hey AWS, any chance we can get a 3.10 image sometime soon?? @jtuliani?

HWiese1980 commented 1 year ago

Yeah, it can't be that hard...

jtuliani commented 1 year ago

We are working on support for Py3.10 in Lambda. However, I don’t currently have a date we can share. We appreciate your feedback and we are also investing in process improvements to enable us to ship runtimes more promptly in future.

leslie-alldridge commented 1 year ago

Thanks for the update

michael-k commented 1 year ago

I don’t currently have a date we can share.

Does this mean it's more than 3 months away? Because for Node.js 16 you made the following announcement:

Work is in progress, and we are planning to ship a Node 16 managed runtime and base container image within the next 90 days.

Source: https://github.com/aws/aws-lambda-base-images/issues/14#issuecomment-1029958936

Python 3.11 will be released in ~2 months, see https://peps.python.org/pep-0664/#schedule

HWiese1980 commented 1 year ago

Valid question. If your release cycle of Lambda containers with new Python versions takes longer than that of completely new Python versions, I'd say, there's something wrong with your release process.

HWiese1980 commented 1 year ago

Valid question. If your release cycle of Lambda containers with new Python versions takes longer than that of completely new Python versions, I'd say, there's something wrong with your release process.

I don't think you understand the definition of the word question. Refer to the comment made here a few days ago by jtuliani, a few comments above yours "we are also investing in process improvements to enable us to ship runtimes more promptly in future"

Excuse me? Is "Does this mean, it's more than 3 months away?" not a question?

leslie-alldridge commented 1 year ago

Can you guys argue on Facebook instead? Most of us are watching to see a PR be merged not two people trying to get one up on each other. Thanks.

HWiese1980 commented 1 year ago

@leslie-alldridge If you ask me the arguing can stop immediately. I wasn't actually keen on being jumped on for some arbitrary and worthless side-track meta-discussion.

liam-dig commented 1 year ago

With the 3.6 runtime officially hitting Phase 2 of depreciation on AWS Lambda today, is there an update on the 3.10 runtime? I'm assuming there are a lot of people like myself who will have to update their 3.6 Lambdas and would prefer to go straight to the latest version.

djstrong commented 1 year ago

3.11 might be interesting. It will be released in one month, declaring 10-60% speed ups.

ghost commented 1 year ago

Any status?

mervin-hemaraju-cko commented 1 year ago

Can someone provide an update on this? 3.6 is being deprecated on AWS Lambda and lots of people like myself will have to update their 3.6 Lambdas and would prefer to go straight to the latest version. This is a shame that AWS is still offering 3.9 as the latest version when 3.11 is just around the corner.

jtuliani commented 1 year ago

We appreciate there is strong interest in Python 3.10 (and 3.11), and we are working on adding support in AWS Lambda. However, we don't currently have a date to share, and we recommend customers migrating from Python 3.6 use Python 3.9.

HWiese1980 commented 1 year ago

What is it that makes the update to 3.10 so difficult? I don't get it...

JonZeolla commented 1 year ago

Can we please lock this thread until there's a status update? These complaints aren't productive.

HWiese1980 commented 1 year ago

@JonZeolla Locking would be productive? How?

I'm a developer myself. I just would like to understand what makes this so difficult.

monim67 commented 1 year ago

@JonZeolla no but it will stop you from irritating every subscriber of this thread. I subscribed to this page to get updates on it, not to get notified of you guys whining everyday.

HWiese1980 commented 1 year ago

@monim67 To me it seems we only get updates at all if we only keep whining. But alright, I'll keep my mouth shut for a while and we'll see how many updates we'll still get.

BjoernPetersen commented 1 year ago

@HWiese1980 I don't consider variations of "we are obviously still working on it and still won't give a timeline" an update, at least not a valuable one. Just accept the fact that – short of being hired into the AWS team – you can't influence the progress here. It's out of our hands. If they're taking too long for your tastes, file an official complaint with AWS, use a less highly managed service, or switch to a faster competitor.

Sorry to all the thread subscribers getting notified about this message, I just couldn't help myself.

MarcoPellicciottaOSF commented 1 year ago

Python 3.10 match functionality is a big, enormous improvement. "We don't have a timeline to share" feels like "we are not working on it at all". The first thing on any project is to have timelines and milestones. If you don't have one is because there is no planing for the work to be done in the near future. That's why we're worried, we want to know if we should consider shifting to other competitors.

leslie-alldridge commented 1 year ago

Feel free to shift to a competitor, door is wide open

orfisko commented 1 year ago

Why invest in releasing functionality that will actually cost your company money, right? Just makes one figure how long it will take before 3.11 will be released which will 25% to 30% faster across the board. That will cost AWS millions in lost revenue. Simple as that, and there you have major downside of going serverless.

BTripp1986 commented 1 year ago

Why invest in releasing functionality that will actually cost your company money, right? Just makes one figure how long it will take before 3.11 will be released which will 25% to 30% faster across the board. That will cost AWS millions in lost revenue. Simple as that, and there you have major downside of going serverless.

AWS provides countless resources in the form of tools, blog post, talks, etc. to help customers reduce their cost while using their products. It's one of The 6 Pillars of the AWS Well-Architected Framework. To suggest that this is an attempt to nickel and dime customers is ludicrous. AWS is most likely suffering from the same lack of skilled human resources as every other software company is today.

Complaining at the team in this thread, taking shots at their processes, and complaining isn't going to speed up this release. It's just going to make the dev teams job harder. They have already acknowledged that there are problems in their process and that they are working on them. They have given guidance that people upgrading their environment that they should go to 3.9 and not wait for 3.10

orfisko commented 1 year ago

AWS provides countless resources in the form of tools, blog post, talks, etc. to help customers reduce their cost while using their products. It's one of The 6 Pillars of the AWS Well-Architected Framework. To suggest that this is an attempt to nickel and dime customers is ludicrous. AWS is most likely suffering from the same lack of skilled human resources as every other software company is today.

Complaining at the team in this thread, taking shots at their processes, and complaining isn't going to speed up this release. It's just going to make the dev teams job harder. They have already acknowledged that there are problems in their process and that they are working on them. They have given guidance that people upgrading their environment that they should go to 3.9 and not wait for 3.10

Allowing to use a faster runtime is a change that will immediately turn down the cost for millions of users and would have a significant ecological impact. Admitted, suggesting that this is a conscious choice for AWS might be overly cynical on my part. Still, if saving on cost or trying to bring the carbon footprint of cloud computing down is at the heart of what AWS does, as you claim, this should be on top of the backlog. Mentioning here there is not even a timeline to support 3.10 proves that this clearly is not a priority to this date. Instead of coming up a with another blog post or tool, get your priorities straight and make sure to first get your basics covered AWS!

HWiese1980 commented 1 year ago

@orfisko "Immediately" is a bit of a stretch. The customers still have to upgrade their lambdas. And depending on the version they're on now this might include more than just replacing the engine version in some terraform script. At the very least it will require them to change a parameter.

jottenlips commented 1 year ago

@orfisko "Immediately" is a bit of a stretch. The customers still have to upgrade their lambdas. And depending on the version they're on now this might include more than just replacing the engine version in some terraform script. At the very least it will require them to change a parameter.

Tbf for a lot of people, it is as easy as updating a terraform script.

dexterinbk commented 1 year ago

over a year since 3.10 released now...

djstrong commented 1 year ago

3.11 is out! Please support it!

jtuliani commented 1 year ago

Work on 3.10 support is continuing. Let's use this issue to track 3.10 and #62 for 3.11.

vladmiller commented 1 year ago

@jtuliani hire me as a consultant to work on this ;-)

eric-musliner commented 1 year ago

@jtuliani hire me as a consultant to work on this ;-)

They should just opensource it. 3.11 is out now and at this rate we'll have to wait til 2025

hb2638 commented 1 year ago

@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update fails on a self-hosted Gitlab.)

excellent idea: https://gist.github.com/jwgstr/4dcc9f16c52529486de0f6fa78c55aef

Thank you. I took your solution and copied over some files that I saw in AWS's 3.9 image as well as made the environment variabels match.

My POC is @ https://github.com/hb2638/aws_lambda_python_3.11

I tested it out an a simple AWS lambda that returns the event that was passed in. I tried in on some other code but got blocked because I can't pip install pyarrow (There are no wheels and there are just too many things I need to do to get it to work)

HWiese1980 commented 1 year ago

I'd really love to know what the difficulties and obstacles are...