Python3 initial migration

dumol commented 3 years ago

Scope

Build Python 3.9.

Changes

Build Python 3.8.6 with latest pip, setuptools, wheels, etc. Have started with Python 3.9, but backtracked to latest 3.8 because:

latest current cryptography only supports 3.8
more wheels are missing for 3.9 (for 3.8 I only had to use 3rd party wheels for gmpy2 and setproctitle (and only on Windows).

As a general rule, no binary wheels are used for non-Windows platforms.

Build Python's sqlite3 module instead of pysqlite on platforms other than Windows (where the upstream package bundles it). Which also require actually building SQLite.

Adapted readline patch for Python 3.8.6. Tried newest Editline (with or without our patch, but it was problematic under Ubuntu). Still using 20170329-3.1. The patch was first adapted for 3.9, in case that's needed again.

Adapted the Windows "build" for Python 3.8.6, which means:

using the upstream embeddable Python distribution (instead of the installer)
the previous quirky structure is still kept, e.g. lib/Lib
not updating redistributable files any more (and no more manifest wiping)
the following applies: https://docs.python.org/3/using/windows.html#the-embeddable-package
nothing at all is built on Windows now, required wheels are present on https://pypi.chevah.com

Updated most libs and Python extensions to latest stable versions with the exception of:

libedit (as noted above)
bcrypt (missing Python 3.8 wheels for latest version on pypi.org).

Drive-by fixes:

Improved Alpine package check.
Run Python's own test after building it (but don't stop when failing, as this seems to be the norm).
Use https links everywhere.
Various other minor fixes and adaptations.

Testing

Automated? :-]

But please review all changes.

Also, assess the experimental embedded Windows package.

dumol commented 3 years ago

PYOPENSSL_VERSION can be removed from python as pyopenssl is not a pure python package.

For example, if you go to PyPI site and see that the package as no OS specific wheels, then is a sign that we don't need it here -

Ex https://pypi.org/project/pyOpenSSL/#files - no binary wheels https://pypi.org/project/bcrypt/#files - has binary wheels

Here we should only build packages that have C extensions.

OK, have removed pyOpenSSL, alongside pycrypto and scandir. Let me know if more should be removed.

Also, I think that pywin32 now has wheels... so we will no longer need it here.

Fine by me, removed. FYI, I had issues with using the upstream wheels though, for both 227 and 228. I've tried also third-party wheels from https://www.lfd.uci.edu/~gohlke/pythonlibs/.

dumol commented 3 years ago

Pinky promise, no more changes until your review… After asking for review, had some more ideas and couldn't help myself. :-]

adiroiban commented 3 years ago

I see that many tests are executed as part of the "Build Python" GitHub action step. I think that we should have "Build Python" only run the build process and have the tests in a separate step.

I see that we already have "Test Pyhon"

My suggestion is to have the following tests:

Test stdlib - current tests exeuted as part of build
Test pyhtia - current python tests
Test compat - to be added later once compat is ported to py3

adiroiban commented 3 years ago

I would also like to see the Build python process split into 2 steps.

Build python stdlib
Build python modules

The reason is that the current logs for each steps are huge.

adiroiban commented 3 years ago

For the Windows package we need to have everything inside a root "lib/" directory. In this way it will match the existing python-package structure.

Also, for windows we need to create x86 and x64 versions

adiroiban commented 3 years ago

The changes look good. I was able to download the artifacts and tests them on Ubuntu and my Windows 2019

We need to find a way to publish them to http://binary.chevah.com/testing

We need to support pythia on all the SFTPPlus v4 OSes.

My suggest is to keep the repo public but add private runners .... we will see if people will be tempted to create PR with malicious code... as a first steps, GitHub Actions can be configured to run actions for this repo only from GitHub and Chevah own org.

adiroiban commented 3 years ago

For the upload, my suggestion is to install SFTPPlus on binary.chevah.com on a non standard port and create an account that has restricted permissions (no delete, no overwrite)

I am starting to upgrade our tools to py3...for now using default OS python 3.8 ...but the final goal is to have compat running on pythia

adiroiban commented 3 years ago

For porting compat to py3 there is a chicken and egg issue... we can use brink without compat...but we need compat to test brink. .. and we need brink to build and test compat... and we can use compat without brink :)

dumol commented 3 years ago

I see that many tests are executed as part of the "Build Python" GitHub action step. I think that we should have "Build Python" only run the build process and have the tests in a separate step.

I see that we already have "Test Pyhon"

My suggestion is to have the following tests:
* Test stdlib - current tests exeuted as part of build

* Test pyhtia - current python tests

* Test compat - to be added later once compat is ported to py3

This a sensible suggestion, but things are not that simple… If OpenSSL tests fail after building it, I don't want to move further along in the build process. More so, this kind of tests only run from the build dir.

For the above reasons, some tests (GMP, OpenSSL, and now even Python) are run during the build.

dumol commented 3 years ago

I would also like to see the Build python process split into 2 steps.
* Build python stdlib

* Build python modules
The reason is that the current logs for each steps are huge.

Some modules are built while compiling Python. You suggest we split that too? Because that is the huge part and it's a single step in compiling Python.

dumol commented 3 years ago

For the Windows package we need to have everything inside a root "lib/" directory. In this way it will match the existing python-package structure.

That is trivial, I actually put a bit of effort in avoiding that. :-]

That's because I wanted the hierarchy to resemble the upstream embeddable Python more. And be more flat and logical (e.g. no more lib/Lib dir). Is it possible to keep this change?

Also, for windows we need to create x86 and x64 versions

We do create both versions, x86 on Server 2016 and x64 on Server 2019.

dumol commented 3 years ago

[…]

We need to support pythia on all the SFTPPlus v4 OSes.

My suggest is to keep the repo public but add private runners .... we will see if people will be tempted to create PR with malicious code... as a first steps, GitHub Actions can be configured to run actions for this repo only from GitHub and Chevah own org.

If possible, I would like to avoid that. But let's not enlarge the scope of this PR too much. Here I was focused on just getting an initial Python 3 build on the GitHub and Travis CI OS'es. I would like other stuff to happen in other PRs.

chevah / pythia