Only build generic Linux packages.

dumol commented 1 year ago

Scope

Fixes #36

Changes

Only build generic glibc and musl Linux packages. Also, renamed lnx and lnx_musl packages to linux and linux_musl.

The generic Linux package is now built on Ubuntu 18.04, to support RHEL 7/8. (For the issues building on CentOS 6/7, see the comments below.)

~~The macOS package is now built on macOS 11, as 10.15 runners are to be deprecated by GitHub soonish.~~ (Not for now, too many issues.)

Latest cryptography is now packaged, using wheels everywhere.

Drive-by changes:

Updated OpenSSL 1.1.1 version (it's only used for Python's ssl on non-Windows platforms now).
Updated pip and all the Python modules to latest versions (safety complained about a couple of them).
Removed subprocess32 module.
Forward-ported the clean-ups from https://github.com/chevah/python-package/pull/173 (no wheel in the distributable, include/ is moved to lib/, no ssl/).

Testing

Please review all changes.

Check the automated tests.

Test the resulting distributables?

dumol commented 1 year ago

This is maybe 90% done, but I hit a wall… Latest cryptography wheel is found on CentOS 7¹, but not on CentOS 6².

@adiroiban, what could be the reason?

(Before you get too enthusiastic about CentOS 7, the /lib64/libfreebl3.so lib is linked against dynamically when building on it.)

adiroiban commented 1 year ago

It's ok if generic Linux are rhel 7 only. We should merge this as it is

We can worry about rhel 6 or 5 later if any customer needs it

dumol commented 1 year ago

As said above, don't get too enthusiastic about CentOS 7, the /lib64/libfreebl3.so lib is linked against dynamically when building on it. As seen at https://github.com/chevah/pythia/actions/runs/4415314095/jobs/7739368782.

I've also checked how deep this dep goes, and it's quite bad… Everything and its mother is linked to it, including the python binary. And the /lib64/libfreebl3.so lib on CentOS 7 belongs to a package that is not removable, being a dependency of both yum and systemd. Some say it's because of a FIPS requirement.

Basically, this means that a Pythia package built on CentOS 7 would only work on RHEL 7+ and their derivatives.

dumol commented 1 year ago

On a second thought, don't get too enthusiastic about CentOS 6 either… Even without being able to integrate cryptography on CentOS 6, I've manually checked deps for the built binaries and libs. Unfortunately, the /lib64/libfreebl3.so is linked against there too, and the nss-softokn-freebl package is already a system one on CentOS 6.

Any other suggestions for a minimalistic distro with long-term supported versions? To support RHEL 7, it would have to be based on glibc 2.19 or older (e.g. Ubuntu 12.04). To support RHEL 6, it needs glibc 2.12 or older (e.g. Ubuntu 10.04).

adiroiban commented 1 year ago

OK. Then let's go with just RHEL 8 and 9. We should be fine for a start.

Thanks

dumol commented 1 year ago

Thanks for the changes. I am a bit worried about the usage of OpenSSL v3 via the latest cryptography

Why worry about this?

but maybe that is fine.

I you want, I think that you can go ahead and merge this and later we can look at how to handle the openssl version differences.

As the first step in SFTPPlus migration to py3 I would still prefer to use OpenSSL v1.1.1.

Not possible unless we build cryptography ourselves on all platforms again. (Which is not worth the trouble since Rust is required.)

but is not a big deal.

I am aproving it so that we can continue with the py3.10 PR

Thanks! Close the discussions satisfactorily answered, and I'll proceed with the merge into master when there are no outstanding problems to discuss…

dumol commented 1 year ago

As discussed in private on Slack, merging this…

chevah / pythia