Python 3.10 update. - Githubissues

dumol commented 1 year ago

Scope

Update Python.

Changes

Use latest Python 3.10. (Python 3.11 builds fail because of https://github.com/python/cpython/issues/101668.)

The libedit patch was removed, as it's not worth porting it to apply to 3.10 sources only for building on macOS…

Drive-by changes:

Removed 32bit Windows build.
Added an arm64 macOS build.
Build the Windows/macOS packages on whatever GitHub supports as -latest (currently Windows Server 2022 and macOS 12).
Build the generic Linux package on Amazon 2 with glibc 2.26.
Updated the other libs to build, where applicable: SQLite, libffi, xz.
Updated cryptography to newly-released version 40.0.1.
Build and check for PyNaCl.
Check for cffi too.
Collect and check for MarkupSafe, charset-normalizer, and pywin32 (the latter only on Windows).

Testing

Review changes and the automated test results.

Try the resulting test packages, e.g. https://bin.chevah.com:20443/testing/3.10.11.5cdf347/

dumol commented 1 year ago

Anything blocking this?

adiroiban commented 1 year ago

I am commenting here so that I don't forget

Here are the missing extra binary wheels.

For py3.8 on Linux, I have updated some pre-compiled versions , but we need them for Alpine.

We might just remove support for alpine for the initial py3, and later we can look into adding these packages

Saved ./bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl Saved ./cryptography-40.0.1-cp36-abi3-manylinux_2_28_x86_64.whl Saved ./cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Saved ./PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl

dumol commented 1 year ago

[…]

Saved ./bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl

bcrypt 4.0.1 is already present on both Linux builds.

Saved ./cryptography-40.0.1-cp36-abi3-manylinux_2_28_x86_64.whl

Updated cryptography to newly-released version 40.0.1 through https://github.com/chevah/pythia/pull/39/commits/9eb343cb2afb961a24d6646f4c4e14423b826815.

Saved ./cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

cffi 1.15.1 already present on all platforms, including Alpine. Now also checked through https://github.com/chevah/pythia/pull/39/commits/ef91078c5e286cca3af23fe7c8b986f49b1deb1a.

Saved ./PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl

PyNaCl is now built and checked for through https://github.com/chevah/pythia/pull/39/commits/7bc9f35f2204dd9c94561561fedf613f7cb974e0.

dumol commented 1 year ago

If you want a Python 3.8.x release with everything but the Python 3.10 update, let me know, I can do such a release independently of this update to Python 3.10.

adiroiban commented 1 year ago

I think that python 3.8 is ok.

But is important to make sure that for 3.10 we have the binary packages.

Another binary package is https://pypi.org/project/MarkupSafe/1.1.1/#files

I will continue to add the packages here

adiroiban commented 1 year ago

Another binary package is https://pypi.org/project/charset-normalizer/2.0.11/

Note that for now we use 2.0.11 since it works on both py2 and py3.

also on py3.8 pywin32 was missing

https://pypi.org/project/pywin32/#files

adiroiban commented 1 year ago

Basically, for py3 on both linux and Windows, we should look into our existing pypi.chevah.com files and review any file that ends with:

win_amd64.whl - windows binary
manylinux1_x86_64.whl (or similar) - linux binary
macosx_10_6_intel.whl (or similar)

Now, our pypi.chevah.com already has some binary package, since I used this as a quick fix, but for the long term, these packages should be provided by pythia.

adiroiban commented 1 year ago

Another thing. For now we need --use-deprecated=legacy-resolver from pip, so this most probably means that we can only use pip 21.0

adiroiban commented 1 year ago

on py3.8 there is another issue on Windows.

Somehow, the site packates are not autoamtically enabled.

I can see the lib/python38._pth which has the site.main() disabled.

I can patch as part of our deps system, but it should be enabled in pythia by defautl.

$ cat build-py3/lib/python38._pth
python38.zip
.

# Uncomment to run site.main() automatically
#import site
lib\site-packages

adiroiban commented 1 year ago

Another thing that should be done for py3.10

we no longer need the -3 check

diff --git a/pythia.sh b/pythia.sh
index baa9dd52e..a3ad6a61a 100755
--- a/pythia.sh
+++ b/pythia.sh
@@ -106,7 +106,7 @@ check_source_folder() {
 # Called to trigger the entry point in the virtual environment.
 # Can be overwritten in pythia.conf
 execute_venv() {
-    ${PYTHON_BIN} $PYTHON3_CHECK -c 'from paver.tasks import main; main()' "$@"
+    ${PYTHON_BIN} -c 'from paver.tasks import main; main()' "$@"
 }

@@ -905,15 +905,6 @@ if [ "$COMMAND" == "deps" ] ; then
     install_base_deps
 fi

-case $COMMAND in
-    test_ci|test_py3)
-        PYTHON3_CHECK='-3'
-        ;;
-    *)
-        PYTHON3_CHECK=''
-        ;;
-esac
-

dumol commented 1 year ago

Another thing. For now we need --use-deprecated=legacy-resolver from pip, so this most probably means that we can only use pip 21.0

Am I missing something? Documentation for pip 23.0.1 still shows this parameter as present: https://pip.pypa.io/en/stable/development/release-process/#use-deprecated.

adiroiban commented 1 year ago

Am I missing something? Documentation for pip 23.0.1 still shows this parameter as present: https://pip.pypa.io/en/stable/development/release-process/#use-deprecated.

True. it look like this feature was not removed .. yet It still works with 23.0.1 So all good for pip.

adiroiban commented 1 year ago

Regarding the python wheel binary packages.

It's ok to have them here.

The only hard requirement is for musl/alpine.

For windows and generic linux, I can find them already pre-compiled on the official PyPI site.

adiroiban commented 1 year ago

Also, it would help a lot to just run 64bit on Windows, since then we will no longer have to execute in pythia.sh the systeminfo.exe helper... it just slows down the whole thing

dumol commented 1 year ago

on py3.8 there is another issue on Windows.

Somehow, the site packates are not autoamtically enabled.

Is this solved now with your latest changes?

dumol commented 1 year ago

Thanks for all the fixes! Is there anything left to improve? I'm a bit dizzy after checking all the commits done through suggestions above… :-]

Also, as discussed on Friday on #chevah, what Python modules should I remove from Pythia? I see charset_normalizer is only available as a binary wheel. Other stuff?

You mentioned setproctitle and MarkupSafe, but the former is built from source on platforms other than Windows, while the latter is built on Alpine.

There is also cryptography which is not built anywhere lately, but I guess you want this included, so I'm not sure where to draw the line…

dumol commented 1 year ago

On a second thought, if we don't include cryptography, then it can be installed and packaged with SFTPPlus as needed. Which would save us from re-releasing Pythia whenever there is a cryptography security update… Maybe useful after all?

dumol commented 1 year ago

Another thing that should be done for py3.10

we no longer need the -3 check

Done through https://github.com/chevah/pythia/pull/39/commits/f2a18c5e583a25d41a75d167707bfa27bfd6bb01.

dumol commented 1 year ago

Went through all the comments once more, let me know if I missed something before approving this. Thanks!

dumol commented 1 year ago

For the record, I've also tried building the generic Linux package on Amazon Linux 1 (aka 2018.03), which has the same glibc version as RHEL 7. And which has gcc 7.2 available as an extra package.

The package builds with minimal changes, but it has the same issues as on CentOS 7, it links against /lib64/libfreebl3.so, which belongs to an irremovable system package.

adiroiban commented 1 year ago

I am leaving a comment here, but most probably needs to be done in a separate ticket.

For the Windows version, besides the lib/python310.zip we also need to distribute the source code.

Without the source code, the python works, but debugging is hard.

So for Windows, besides the current embeddable version, we need to find a way to also get the source code avaiable inside the build folder

dumol commented 1 year ago

So for Windows, besides the current embeddable version, we need to find a way to also get the source code avaiable inside the build folder

Sure, no problem, I'll fill in a new ticket… But what sources you want included? The win package of python-package only has some include files in /python2.7-win-x64/lib/include/… You want more? All the source code of Python? Only some parts of it?

What kind of debugging you have in mind? Debugging Python binaries?

dumol commented 1 year ago

For the record, merged into master as discussed on #chevah last week on Friday.

chevah / pythia

Python 3.10 update. #39

Scope

Changes

Testing