Closed michael-k closed 1 year ago
Please?
Any implementation timeline?
some features look interesting, looking forward!
keen !
+1 as much as I love building python from source - maybe 3.7 is a little long in the tooth to be the standard...
+1 as much as I love building python from source - maybe 3.7 is a little long in the tooth to be the standard...
Agree - looks like @jbesw did the work for 3.9... maybe he can help with 3.10 support?
https://aws.amazon.com/blogs/compute/python-3-9-runtime-now-available-in-aws-lambda/
Thank you everyone for your feedback regarding AWS Lambda support for Python 3.10. We are aware of the ask, however, we don't currently have a timeline to share.
Python 3.9 just hit the security-only stage of its maintenance lifecycle:
https://pythoninsider.blogspot.com/2022/05/python-3913-is-now-available.html
Two months since the last update. Any (hopefully good) news here? @jtuliani
I had a desire to update my lambda builds to 3.10 the other day and began looking into this more. Since this issue has been open for a little while I thought I might take a stab at building this from the amazonlinux base image (as the python 3.9 builds are). None of the guides I found quite made this copy / paste possible.
So I came up with this, which I tested and seems to work just fine at least for my use cases which includes things like numpy. But depending on your project complexity your mileage may vary. It doesn't follow all the same conventions with putting task logic in /var/task
or app logic in /var/runtime
but that didn't cause any conflicts for me. It's kind of a hodgepodge of various AWS docs and articles I found. Wouldn't surprise me if it can be cleaned up a bit.
I don't think I have access to contribute this back anywhere otherwise I would create PR. @jtuliani please feel free to point me in the right direction for that and I'm happy to help iterate on a better vetted and reviewed 3.10 image.
In the meantime I hope this helps others.
ARG FUNCTION_DIR="/app/"
ARG AWS_LINUX_VERSION="2022"
ARG PYTHON_VERSION="3.10.4"
FROM amazonlinux:${AWS_LINUX_VERSION} as python-layer
ARG PYTHON_VERSION
# install python
RUN yum update -y
RUN yum groupinstall "Development Tools" -y
RUN yum install openssl1.1 openssl1.1-devel libffi-devel bzip2-devel wget -y
RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz
RUN tar -xf Python-${PYTHON_VERSION}.tgz
RUN cd Python-${PYTHON_VERSION}/ && \
./configure --enable-optimizations && \
make install
RUN ln -s /Python-${PYTHON_VERSION}/python /usr/bin/python
RUN python -m pip install --upgrade pip
FROM amazonlinux:${AWS_LINUX_VERSION} as base-layer
# copy over python
ARG PYTHON_VERSION
COPY --from=python-layer /Python-${PYTHON_VERSION} /Python-${PYTHON_VERSION}
COPY --from=python-layer /usr/local/bin /usr/local/bin
COPY --from=python-layer /usr/local/lib /usr/local/lib
RUN ln -s /Python-${PYTHON_VERSION}/python /usr/bin/python
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}
FROM base-layer as build-layer
######## YOUR OWN SETUP PROCESS HERE ########################
# copy over requirements and install those
COPY setup.py .
RUN pip install . --target "${FUNCTION_DIR}"
# copy over configuration and service code then install it
COPY config/ ${FUNCTION_DIR}/config/
COPY service/ ${FUNCTION_DIR}/
RUN pip install . --target "${FUNCTION_DIR}"
######## ########################### ########################
# install lambda runtime interface client for python
RUN pip install awslambdaric --target "${FUNCTION_DIR}"
FROM base-layer as runtime-layer
# copy in the built dependencies
ARG FUNCTION_DIR
COPY --from=build-layer ${FUNCTION_DIR} ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}
# (optional) add lambda runtime interface emulator
#ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
#RUN chmod 755 /usr/bin/aws-lambda-rie
#ENTRYPOINT [ "/usr/bin/aws-lambda-rie", "python", "-m", "awslambdaric" ]
ENTRYPOINT [ "python", "-m", "awslambdaric" ]
######## REFERENCE YOUR OWN HANDLER HERE ########################
CMD [ "main.app" ]```
######## ############################### ########################
@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update
fails on a self-hosted Gitlab.)
@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular,
yum update
fails on a self-hosted Gitlab.)
excellent idea: https://gist.github.com/jwgstr/4dcc9f16c52529486de0f6fa78c55aef
This is now an issue for users of Gitlab who use the Docker:latest image to build lambda images using DinD. The Docker image is built off of Alpine 3.16 which now only supports python3.10 natively.
Our workaround is to use pyenv to install a python 3.9 version... but that adds significant time to our builds.
You could also just not use the latest
tag and continue with the older image (e.g. the docker:20.10.16-dind-alpine3.15
tag). Using the latest
tag is generally not a great idea, because it'll often lead to issues like these in the worst possible moment.
Not saying that it isn't ridiculous for AWS to still not support Python 3.10 in Lambdas, just offering my opinion on best practices when using container images.
Let me try that. (Thanks, I had gone through the list of available Docker tags and I didn't see that one.)
Ty, Stephen
now that 3.10 is the only available runtime for the latest codebuild images (https://github.com/aws/aws-codebuild-docker-images/pull/505), one cannot use these to build + package lambdas without manually downgrading or upgrading Python (e.g. via pyenv) in codebuild or lambda, respectively, or risk bundling incompatible C extension binaries between build and deployment. @jtuliani @jbesw any updates?
Half way around the sun since this was opened and 10 months since 3.10 final was released. Hey AWS, any chance we can get a 3.10 image sometime soon?? @jtuliani?
Yeah, it can't be that hard...
We are working on support for Py3.10 in Lambda. However, I don’t currently have a date we can share. We appreciate your feedback and we are also investing in process improvements to enable us to ship runtimes more promptly in future.
Thanks for the update
I don’t currently have a date we can share.
Does this mean it's more than 3 months away? Because for Node.js 16 you made the following announcement:
Work is in progress, and we are planning to ship a Node 16 managed runtime and base container image within the next 90 days.
Source: https://github.com/aws/aws-lambda-base-images/issues/14#issuecomment-1029958936
Python 3.11 will be released in ~2 months, see https://peps.python.org/pep-0664/#schedule
Valid question. If your release cycle of Lambda containers with new Python versions takes longer than that of completely new Python versions, I'd say, there's something wrong with your release process.
Valid question. If your release cycle of Lambda containers with new Python versions takes longer than that of completely new Python versions, I'd say, there's something wrong with your release process.
I don't think you understand the definition of the word question. Refer to the comment made here a few days ago by jtuliani, a few comments above yours "we are also investing in process improvements to enable us to ship runtimes more promptly in future"
Excuse me? Is "Does this mean, it's more than 3 months away?" not a question?
Can you guys argue on Facebook instead? Most of us are watching to see a PR be merged not two people trying to get one up on each other. Thanks.
@leslie-alldridge If you ask me the arguing can stop immediately. I wasn't actually keen on being jumped on for some arbitrary and worthless side-track meta-discussion.
With the 3.6 runtime officially hitting Phase 2 of depreciation on AWS Lambda today, is there an update on the 3.10 runtime? I'm assuming there are a lot of people like myself who will have to update their 3.6 Lambdas and would prefer to go straight to the latest version.
3.11 might be interesting. It will be released in one month, declaring 10-60% speed ups.
Any status?
Can someone provide an update on this? 3.6 is being deprecated on AWS Lambda and lots of people like myself will have to update their 3.6 Lambdas and would prefer to go straight to the latest version. This is a shame that AWS is still offering 3.9 as the latest version when 3.11 is just around the corner.
We appreciate there is strong interest in Python 3.10 (and 3.11), and we are working on adding support in AWS Lambda. However, we don't currently have a date to share, and we recommend customers migrating from Python 3.6 use Python 3.9.
What is it that makes the update to 3.10 so difficult? I don't get it...
Can we please lock this thread until there's a status update? These complaints aren't productive.
@JonZeolla Locking would be productive? How?
I'm a developer myself. I just would like to understand what makes this so difficult.
@JonZeolla no but it will stop you from irritating every subscriber of this thread. I subscribed to this page to get updates on it, not to get notified of you guys whining everyday.
@monim67 To me it seems we only get updates at all if we only keep whining. But alright, I'll keep my mouth shut for a while and we'll see how many updates we'll still get.
@HWiese1980 I don't consider variations of "we are obviously still working on it and still won't give a timeline" an update, at least not a valuable one. Just accept the fact that – short of being hired into the AWS team – you can't influence the progress here. It's out of our hands. If they're taking too long for your tastes, file an official complaint with AWS, use a less highly managed service, or switch to a faster competitor.
Sorry to all the thread subscribers getting notified about this message, I just couldn't help myself.
Python 3.10 match functionality is a big, enormous improvement. "We don't have a timeline to share" feels like "we are not working on it at all". The first thing on any project is to have timelines and milestones. If you don't have one is because there is no planing for the work to be done in the near future. That's why we're worried, we want to know if we should consider shifting to other competitors.
Feel free to shift to a competitor, door is wide open
Why invest in releasing functionality that will actually cost your company money, right? Just makes one figure how long it will take before 3.11 will be released which will 25% to 30% faster across the board. That will cost AWS millions in lost revenue. Simple as that, and there you have major downside of going serverless.
Why invest in releasing functionality that will actually cost your company money, right? Just makes one figure how long it will take before 3.11 will be released which will 25% to 30% faster across the board. That will cost AWS millions in lost revenue. Simple as that, and there you have major downside of going serverless.
AWS provides countless resources in the form of tools, blog post, talks, etc. to help customers reduce their cost while using their products. It's one of The 6 Pillars of the AWS Well-Architected Framework. To suggest that this is an attempt to nickel and dime customers is ludicrous. AWS is most likely suffering from the same lack of skilled human resources as every other software company is today.
Complaining at the team in this thread, taking shots at their processes, and complaining isn't going to speed up this release. It's just going to make the dev teams job harder. They have already acknowledged that there are problems in their process and that they are working on them. They have given guidance that people upgrading their environment that they should go to 3.9 and not wait for 3.10
AWS provides countless resources in the form of tools, blog post, talks, etc. to help customers reduce their cost while using their products. It's one of The 6 Pillars of the AWS Well-Architected Framework. To suggest that this is an attempt to nickel and dime customers is ludicrous. AWS is most likely suffering from the same lack of skilled human resources as every other software company is today.
Complaining at the team in this thread, taking shots at their processes, and complaining isn't going to speed up this release. It's just going to make the dev teams job harder. They have already acknowledged that there are problems in their process and that they are working on them. They have given guidance that people upgrading their environment that they should go to 3.9 and not wait for 3.10
Allowing to use a faster runtime is a change that will immediately turn down the cost for millions of users and would have a significant ecological impact. Admitted, suggesting that this is a conscious choice for AWS might be overly cynical on my part. Still, if saving on cost or trying to bring the carbon footprint of cloud computing down is at the heart of what AWS does, as you claim, this should be on top of the backlog. Mentioning here there is not even a timeline to support 3.10 proves that this clearly is not a priority to this date. Instead of coming up a with another blog post or tool, get your priorities straight and make sure to first get your basics covered AWS!
@orfisko "Immediately" is a bit of a stretch. The customers still have to upgrade their lambdas. And depending on the version they're on now this might include more than just replacing the engine version in some terraform script. At the very least it will require them to change a parameter.
@orfisko "Immediately" is a bit of a stretch. The customers still have to upgrade their lambdas. And depending on the version they're on now this might include more than just replacing the engine version in some terraform script. At the very least it will require them to change a parameter.
Tbf for a lot of people, it is as easy as updating a terraform script.
over a year since 3.10 released now...
3.11 is out! Please support it!
Work on 3.10 support is continuing. Let's use this issue to track 3.10 and #62 for 3.11.
@jtuliani hire me as a consultant to work on this ;-)
@jtuliani hire me as a consultant to work on this ;-)
They should just opensource it. 3.11 is out now and at this rate we'll have to wait til 2025
@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular,
yum update
fails on a self-hosted Gitlab.)excellent idea: https://gist.github.com/jwgstr/4dcc9f16c52529486de0f6fa78c55aef
Thank you. I took your solution and copied over some files that I saw in AWS's 3.9 image as well as made the environment variabels match.
My POC is @ https://github.com/hb2638/aws_lambda_python_3.11
I tested it out an a simple AWS lambda that returns the event that was passed in. I tried in on some other code but got blocked because I can't pip install pyarrow (There are no wheels and there are just too many things I need to do to get it to work)
I'd really love to know what the difficulties and obstacles are...
Latest official update for anyone who missed it in the wall of comments:
Previous official update:
Python 3.10 was released on 2021-10-04