Forward-port minor fixes from python-package.

dumol commented 2 years ago

Scope

Minor improvements copied over from python-package repo.

Changes

Build a generic musl Linux version on Alpine 3.12.

Handle musl-based Linux distros other than Alpine better.

Error out if /lib64 present after build.

Show full zlib module version.

Use PIP_INDEX_URL in pavement.py.

Drive-by fixes:

Updated zlib to version 1.2.13 fixing CVE-2018-25032.
Updated OpenSSL to verison 1.1.1s fixing CVE-2022-1292, CVE-2022-2068, CVE-2022-2097.

Testing

Automated. :D

dumol commented 2 years ago

Just take a quick look at these simple changes, thanks!

I've used this PR to test some stuff on macOS too.

adiroiban commented 2 years ago

Looks good. Note that I have asked python 3.8 as when this was started, there were some issues with python 3.9.

I think that those issues are fixed. I think that we can keep 3.8 as a start.

Most probably in production we will go with Python 3.10 or 3.11.

What I wanted to say, is that if using a non python 3.8 make it easier to build Python v3, that is ok

thanks

dumol commented 2 years ago

Looks good. Note that I have asked python 3.8 as when this was started, there were some issues with python 3.9.

I think that those issues are fixed. I think that we can keep 3.8 as a start.

Most probably in production we will go with Python 3.10 or 3.11.

What I wanted to say, is that if using a non python 3.8 make it easier to build Python v3, that is ok

The way we build Python, every major version upgrade is a massive pain in the rear, because I have to forward-port the libedit patches. And then fix any new issues with our builds.

Only minor version updates are doable, but these are limited by what upstream does. In the case of Python 3.8, upstream stopped providing embeddable Windows packages at version 3.8.10. Compare https://www.python.org/downloads/release/python-3810/ with https://www.python.org/downloads/release/python-3811/.

On a related note, even when not updating any version, our builds might fail because of external factors. Like it happens here with OpenSSL tests on both CentOS 5 and macOS 10.15, where they mysteriously started to fail with:

test/recipes/80-test_ssl_new.................... Dubious, test returned 1 (wstat 256, 0x100)

Given the Rust dependency of cryptography, I think the way forward would be to stop trying to build it ourselves. Just use the upstream wheels with embedded OpenSSL.

Ideally, also use upstream Python packages on all OS'es, just like on Windows. Effectively, this means not building anything at all, just like we already do on Windows for this repo. Given the way Python is packaged upstream, this would mean being ready to jump to the latest major Python version rather early. E.g. for the 3.8.x lifecycle, be on Python 3.9.x before Python 3.8.11 was released.

dumol commented 2 years ago

For the record, I've investigated the mysterious OpenSSL test failure on my machine. It looks like it's caused by an expired certificate. Output of make test V=1:

# Subtest: Test configuration 12-ct.conf
    1..3
../../util/shlib_wrap.sh /usr/bin/perl -I ../../util/perl ../generate_ssl_tests.pl ../ssl-tests/12-ct.conf.in > 12-ct.conf.32486.tmp => 0
    ok 1 - Getting output from generate_ssl_tests.pl.
    ok 2 - Comparing generated sources.
        # Subtest: ../ssl_test
        1..1
            # Subtest: test_handshake
            1..6
            ok 1 - iteration 1
            # ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
            # [2] compared to [0]
            # INFO:  @ test/ssl_test.c:34
            # ExpectedResult mismatch: expected Success, got ClientFail.
            # 140538595219264:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
            not ok 2 - iteration 2
            ok 3 - iteration 3
            # ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
            # [2] compared to [0]
            # INFO:  @ test/ssl_test.c:34
            # ExpectedResult mismatch: expected Success, got ClientFail.
            # 140538595219264:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
            not ok 4 - iteration 4
            # ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
            # [4] compared to [0]
            # INFO:  @ test/ssl_test.c:34
            # ExpectedResult mismatch: expected Success, got FirstHandshakeFailed.
            # 140538595219264:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
            not ok 5 - iteration 5
            # ERROR: (int) 'result->result == test_ctx->expected_result' failed @ test/ssl_test.c:33
            # [4] compared to [0]
            # INFO:  @ test/ssl_test.c:34
            # ExpectedResult mismatch: expected Success, got FirstHandshakeFailed.
            # 140538595219264:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:ssl/record/rec_layer_s3.c:1543:SSL alert number 45
            not ok 6 - iteration 6
        not ok 1 - test_handshake
../../util/shlib_wrap.sh ../ssl_test 12-ct.conf.32486.tmp => 1
    not ok 3 - running ssl_test 12-ct.conf

    #   Failed test 'running ssl_test 12-ct.conf'
    #   at ../test/recipes/80-test_ssl_new.t line 148.
    # Looks like you failed 1 test of 3.
not ok 12 - Test configuration 12-ct.conf

#   Failed test 'Test configuration 12-ct.conf'
#   at /home/dumol/src/pythia/build/openssl-1.1.1n/test/../util/perl/OpenSSL/Test.pm line 1212.

dumol commented 2 years ago

OpenSSL test failure reported upstream at https://github.com/openssl/openssl/issues/18544.

dumol commented 2 years ago

At least 1 approving review is required to merge this pull request. Thanks!

Very simple changes over all, I've used this PR mostly to test the installer under macOS.

danuker commented 2 years ago

I haven't seen any needs-review comment and this is why I haven't reviewed it yet.

@adiroiban It looks like @chevah-robot needs admin permissions for managing labels on this repo:

Traceback (most recent call last):
  File "/home/site/wwwroot/chevah/github_hooks_server/server.py", line 81, in hook
    response = handle_event(event)
  File "/home/site/wwwroot/chevah/github_hooks_server/server.py", line 137, in handle_event
    return handler.dispatch(event)
  File "/home/site/wwwroot/chevah/github_hooks_server/handler.py", line 38, in dispatch
    return handler(event)
  File "/home/site/wwwroot/chevah/github_hooks_server/handler.py", line 89, in pull_request_review
    self._setApproveChanges(
  File "/home/site/wwwroot/chevah/github_hooks_server/handler.py", line 186, in _setApproveChanges
    issue.add_labels('needs-merge')
  File "/home/site/wwwroot/.python_packages/lib/site-packages/github3/decorators.py", line 24, in auth_wrapper
    return func(self, *args, **kwargs)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/github3/issues/issue.py", line 99, in add_labels
    json = self._json(self._post(url, data=args), 200)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/github3/models.py", line 153, in _json
    raise exceptions.error_for(response)
github3.exceptions.ForbiddenError: 403 Must have admin rights to Repository.

dumol commented 1 year ago

An OpenSSL test fails on macOS, but there's a fix upstream already: https://github.com/openssl/openssl/commit/60f011f584d80447e86cae1d1bd3ae24bc13235b

This can wait for OpenSSL 1.1.1r, which would include the needed fix.

dumol commented 1 year ago

As I've waited with this PR for upstream to fix the OpenSSL build on macOS, I've sneaked in some more changes forward-ported from python-package repo. Mainly related to building a generic musl package on Alpine 3.12, so that I can use this on my musl-based Void system.

The diff is still quite minimal if you ignore the changes in pythia.sh which are copy-pasted from brink.sh in python-package. Actually, you might also take a look at the diff between those two:

25c25
< # then read from brink.conf as CHEVAH_BUILD_DIR,
---
> # then read from pythia.conf as CHEVAH_BUILD_DIR,
29c29
< # and then read from brink.conf as CHEVAH_CACHE_DIR,
---
> # and then read from pythia.conf as CHEVAH_CACHE_DIR,
32c32
< # You can define your own `execute_venv` function in brink.conf with the
---
> # You can define your own `execute_venv` function in pythia.conf with the
85c85
< # Initialize default values from brink.conf
---
> # Initialize default values from pythia.conf
89,90c89,90
< PYTHON_NAME='python2.7'
< BINARY_DIST_URI='https://github.com/chevah/python-package/releases/download'
---
> PYTHON_NAME='python3.8'
> BINARY_DIST_URI='https://github.com/chevah/pythia/releases/download'
96c96
< # If not, we are out of the source's root dir and brink.sh won't work.
---
> # If not, we are out of the source's root dir and pythia.sh won't work.
101c101
<         (>&2 echo 'Make sure you are running "brink.sh" from a source folder.')
---
>         (>&2 echo 'Make sure you are running "pythia.sh" from a source folder.')
107c107
< # Can be overwritten in brink.conf
---
> # Can be overwritten in pythia.conf
125c125
<         (>&2 echo 'Failed to run the initial "./brink.sh deps" command.')
---
>         (>&2 echo 'Failed to run the initial "./pythia.sh deps" command.')
131c131
< source brink.conf
---
> source pythia.conf
142,145d141
<     echo "Removing node_modules..."
<     delete_folder node_modules
<     echo "Removing web build"
<     delete_folder chevah/server/static/build/
161,162d156
<     # AIX's find complains if there are no matching files when using +.
<     [ $(uname) == AIX ] && touch ./dummy_file_for_AIX.pyc
461c455
<             cache_ver_file=${python_distributable}/lib/PYTHON_PACKAGE_VERSION
---
>             cache_ver_file=${python_distributable}/lib/PYTHIA_VERSION
490c484
<         local version_file=${BUILD_FOLDER}/lib/PYTHON_PACKAGE_VERSION
---
>         local version_file=${BUILD_FOLDER}/lib/PYTHIA_VERSION
492c486
<         # If we are upgrading the cache from an unversioned Python,
---
>         # If we are upgrading the cache from Python 2,
543,544c537
< # naming the package, for example: '8' for RHEL 8.2, '2004' for Ubuntu 20.04,
< # '71' for AIX 7.1, '114' for Solaris 11.4.
---
> # naming the package, for example: '8' for RHEL 8.2, '2004' for Ubuntu 20.04.
743a737
> 
807,810c801
<             # Tested on 10.13, but this works on 10.12 too. Older versions need
<             # "-Wl,-no_weak_imports" in LDFLAGS to avoid runtime issues. More
<             # details at https://github.com/Homebrew/homebrew-core/issues/3727.
<             check_os_version "macOS" 10.12 "$os_version_raw" os_version_chevah
---
>             check_os_version "macOS" 10.13 "$os_version_raw" os_version_chevah
847,852d837
<         AIX)
<             ARCH="ppc$(getconf HARDWARE_BITMODE)"
<             os_version_raw=$(oslevel)
<             check_os_version AIX 7.1 "$os_version_raw" os_version_chevah
<             OS="aix${os_version_chevah}"
<             ;;
881,889d865
<         "ppc64")
<             # Python has not been fully tested on AIX when compiled as a 64bit
<             # binary, and has math rounding error problems (at least with XL C).
<             ARCH="ppc"
<             ;;
<         "sparcv9")
<             # We build 32bit binaries on SPARC too. Use "sparc64" for 64bit.
<             ARCH="sparc"
<             ;;
927c903
< # Update brink.conf dependencies when running deps.
---
> # Update pythia.conf dependencies when running deps.

needs-review

dumol commented 1 year ago

Staging binaries available at https://bin.chevah.com:20443/testing/3.8.6.df02a9e/.

I'll also make a release after merging this into master.

chevah / pythia