search

amazonlinux / amazon-linux-2023

Amazon Linux 2023
https://aws.amazon.com/linux/amazon-linux-2023/
Other
521 stars 39 forks source link

[Package Request] - Python 3.12 #483

Open matejsp opened 11 months ago

matejsp commented 11 months ago

What package is missing from Amazon Linux 2023? Please describe and include package name. Python 3.12 that was released today. https://pythoninsider.blogspot.com/2023/10/python-3120-final-now-available.html

Is this an update to existing package or new package request? Update to existing package (would be nice to have python 3.11 and python 3.12 available as a transition)

Is this package available in Amazon Linux 2? If it is available via external sources such as EPEL, please specify. Old versions of python

Any additional information you'd like to include. (use-cases, etc) Using latest python 3.12 features.

stewartsmith commented 11 months ago

There's chatter over on https://github.com/aws/elastic-beanstalk-roadmap/issues/174 about Beanstalk and up to date Python support. By having Python 3.12 packaged in AL2023, adding this as a Beanstalk platform is less work for the Beanstalk team.

ccassidy-maxar commented 9 months ago

Adding a note from my accidental duplicate ticket: Lambda now supports Python 3.12 using the AL2023 base image. https://aws.amazon.com/about-aws/whats-new/2023/12/aws-lambda-support-python-3-12/

bryantbiggs commented 8 months ago

would this be switching the default python from 3.9 to 3.12, or is it simply adding support to install 3.12 (similar to 3.11)? I'd like to use just one python installation in the AL2023 minimal container image, ideally 3.12 due to the performance improvements

stewartsmith commented 8 months ago

would this be switching the default python from 3.9 to 3.12, or is it simply adding support to install 3.12 (similar to 3.11)? I'd like to use just one python installation in the AL2023 minimal container image, ideally 3.12 due to the performance improvements

It's for having it as an option, not replacing the system Python.

The good news is that the minimal container image doesn't have any version of Python in it due to it having microdnf rather than the full dnf, so if you added a version of Python to it, it's entirely your choice what version that is.

aperuru commented 7 months ago

Unable to install python 3.12.1 from source on AmazonLinux-2023 (2023.3.20231218.0)

here are the steps from my dockerfile

RUN dnf install -y gzip make tar wget which openssl \
         bzip2-devel gcc-c++ libffi-devel openssl-devel \
         openssh-clients sqlite-devel zlib-devel readline-devel \
         && dnf clean all
RUN wget -nv https://www.python.org/ftp/python/3.12.1/Python-3.12.1.tgz \
        && tar xzf Python-3.12.1.tgz \
        && cd Python-3.12.1 \ 
        && ./configure --enable-optimizations \
        && make -j $(nproc) \    #fails at this step
        && make altinstall \

Error on step make -j $(nproc)

gcc: internal compiler error: Segmentation fault signal terminated program cc1
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://github.com/amazonlinux/amazon-linux-2022> for instructions.
make[2]: *** [Makefile:2706: Parser/string_parser.o] Error 4
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/Python-3.12.1'
make[1]: *** [Makefile:777: profile-gen-stamp] Error 2
make[1]: Leaving directory '/tmp/Python-3.12.1'
make: *** [Makefile:789: profile-run-stamp] Error 2

I also tried using just make

make[2]: *** [Makefile:1366: Python/frozen_modules/frozen_only.h] Segmentation fault (core dumped)
make[2]: Leaving directory '/tmp/Python-3.12.1'
make[1]: *** [Makefile:777: profile-gen-stamp] Error 2
make[1]: Leaving directory '/tmp/Python-3.12.1'
make: *** [Makefile:789: profile-run-stamp] Error 2

Has anyone run into similar issues? Any help would be greatly appreciated

matejsp commented 7 months ago

If you get core dumped or killed process try assigning more RAM to docker builder and try again.

I didn't try building it on al2023, but we used Python 3.12 from official lambda repository. Its not perfect but does the job for now. I would really like to have it in al2023 instead.

FROM amazonlinux/amazonlinux:2023.3.20240108.0-minimal as base-arm64
# Taken from al2023 lambda package https://github.com/aws/aws-lambda-base-images/blob/python3.12/x86_64/Dockerfile.python3.12
ADD a6a32126a5d6ad6ee275fd1b290d6a3381de37020554fa2b4c1818a90f8c15a4.tar.xz /

FROM amazonlinux/amazonlinux:2023.3.20240108.0-minimal as base-amd64
# Taken from al2023 lambda package https://github.com/aws/aws-lambda-base-images/blob/python3.12/x86_64/Dockerfile.python3.12
ADD 08e61bf1511d156676b2e9ac9eea5a8727c2c5a67fa4d7a957cee02af4a51252.tar.xz /

FROM base-${TARGETARCH}

RUN dnf install -y gzip make tar wget which openssl \
         bzip2-devel gcc-c++ libffi-devel openssl-devel \
         openssh-clients sqlite-devel zlib-devel readline-devel \
         && dnf clean all

...
matejsp commented 7 months ago

I tried your command running on docker minimal and got ir built without any problems on 8gb docker machine.

 docker run -it amazonlinux/amazonlinux:2023.3.20240108.0-minimal bash
echo  $(nproc)
8
make -j $(nproc)
...
Checked 111 modules (31 built-in, 73 shared, 1 n/a on linux-x86_64, 0 disabled, 6 missing, 0 failed on import)
make[1]: Leaving directory '/Python-3.12.1'
bash-5.2#
maunzCache commented 7 months ago

Adding a note from my accidental duplicate ticket: Lambda now supports Python 3.12 using the AL2023 base image. https://aws.amazon.com/about-aws/whats-new/2023/12/aws-lambda-support-python-3-12/

Sidenote: I doesn't seem to be available as a package in the base image itself judging by https://docs.aws.amazon.com/linux/al2023/release-notes/all-packages-AL2023.3.html

Edit: If you do not rely on AL2023, you might directly reference the runtime via https://hub.docker.com/r/amazon/aws-lambda-python

dkavanagh commented 7 months ago

Having this package would help me bundling python libs like cryptography which include native code. We'd like to build for 3.12 on AL2023.

matejsp commented 7 months ago

@maunzCache No it is not availabe as a package yet. In docker you can just add docker layer but for EC2 you are out of luck currently (unless you build it yourself). Really hope to have 3.12 available soon in AL2023.

antm-pp commented 6 months ago

Given the lambda python3.12 runtime is the only one based on AL2023 it'd be useful to have a common EC2 base for testing outside of lambda. Presumably it must already be in codified in repos in some form to be available via dnf and maintained in the lambda docker images?

matejsp commented 5 months ago

Any update on the python 3.12 when avaialable? I was really looking forward to new 2024.4 release but then I saw there is still no new python available. Lambda supports python 3.12 since december. And we would like to have the same python as in lambda on our ec2 or docker image. Lambda 3.11 is using AL2 and only 3.12 is using AL2023.

matejsp commented 4 months ago

I stumbled upon release notes for RHEL 9.4 (https://developers.redhat.com/articles/2024/05/01/whats-new-red-hat-enterprise-linux-94). And they already support python 3.12.

I tried installed python 3.12 via AlmaLinux packages :

dnf install https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/python3.12-3.12.1-4.el9.x86_64.rpm https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/python3.12-libs-3.12.1-4.el9.x86_64.rpm        https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/python3.12-pip-wheel-23.2.1-4.el9.noarch.rpm https://repo.almalinux.org/almalinux/9/AppStream/x86_64/os/Packages/libnsl2-2.0.0-1.el9.x86_64.rpm

And got it working. Too bad still no official release in AL3.

zztopping commented 3 months ago

Chiming in here as well, anyone building packages (with OS lib deps) on Amazon Linux for python on Lambda are kinda stuck on 3.8 right now unless they build Python from source:

abhishek-parative commented 2 months ago

Any updates?

maunzCache commented 2 months ago

Any updates?

2023.4 did not bring a python update. Actually i would not expect a new python until AL2025. If you only need python 3.12 for e.g. local lambda tests or similar you may use https://hub.docker.com/r/amazon/aws-lambda-python instead.

matejsp commented 2 months ago

Any updates?

Just checked latest 2023.5 and unfortunately still lacking python 3.12 ... Is there any chance to get a least some timeline for python 3.12 since it was released 9 months ago.

direvus commented 2 months ago 
* Lambda's Python 3.11 is only available on AL2, but AL2 does not support 3.11 as a AWS-managed package (I assume it never will, given our place in the lifetime of AL2)

Yes, and to make matters worse, AL2 is stuck on GLIBC 2.26 (coming to you live from August 2017), which means AL2 is totally non-viable for some libraries.