Trouble compiling pyOpenSSL

[fosskers]

Hi there, I gave this a shot again recently in an attempt to build Aura statically. I used to be successful at this, but I think I cleaned out my Nix store recently, blowing away the custom static GHC, etc.

I can get pretty far rebuilding the whole toolchain, but it fails at pyOpenSSL-19.0.0:

builder for '/nix/store/aafcj2cnpqrs4vrlj5i66z1nhr0x20jd-python3.7-pyOpenSSL-19.0.0.drv' failed with exit code 1
cannot build derivation '/nix/store/lwi6hsj50bx87pbqcl0rxx28ywlf0sws-python3.7-urllib3-1.24.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/6v2795g0ixqryzaqkzz261rg5ikfr1jn-python3.7-requests-2.22.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ris2q6l28b7hscqvxp0q72whdyavhp6b-python3.7-sphinx-1.8.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/d817i4fqpv18h39dn0cr2jff7xy2igqr-ghc-8.6.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/fb5snsz8xlr46gbfhvgsxcl3yacsjzfh-aura-2.0.3.drv': 1 dependencies couldn't be built
error: build of '/nix/store/fb5snsz8xlr46gbfhvgsxcl3yacsjzfh-aura-2.0.3.drv' failed

I'm trying to build Aura's master branch, using this default.nix:

# Run using:
#
#     $(nix-build --no-link -A fullBuildScript)
{
  stack2nix-output-path ? "custom-stack2nix-output.nix",
}:
let
  cabalPackageName = "aura";
  compiler = "ghc865"; # matching stack.yaml

  # Pin static-haskell-nix version.
  static-haskell-nix =
    if builtins.pathExists ../.in-static-haskell-nix
      then toString ../. # for the case that we're in static-haskell-nix itself, so that CI always builds the latest version.
      # Update this hash to use a different `static-haskell-nix` version:
      else fetchTarball https://github.com/nh2/static-haskell-nix/archive/d24dea3d46a727e1cd93e67458ce2768109efe0a.tar.gz;

  # Pin nixpkgs version
  # By default to the one `static-haskell-nix` provides, but you may also give
  # your own as long as it has the necessary patches, using e.g.
  #     pkgs = import (fetchTarball https://github.com/nh2/nixpkgs/archive/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa123.tar.gz) {};
  pkgs = import "${static-haskell-nix}/nixpkgs.nix";

  stack2nix-script = import "${static-haskell-nix}/static-stack2nix-builder/stack2nix-script.nix" {
    inherit pkgs;
    stack-project-dir = toString ./.; # where stack.yaml is
    hackageSnapshot = "2020-01-27T00:00:00Z"; # pins e.g. extra-deps without hashes or revisions
  };

  static-stack2nix-builder = import "${static-haskell-nix}/static-stack2nix-builder/default.nix" {
    normalPkgs = pkgs;
    inherit cabalPackageName compiler stack2nix-output-path;
    # disableOptimization = true; # for compile speed
  };

  # Full invocation, including pinning `nix` version itself.
  fullBuildScript = pkgs.writeShellScript "stack2nix-and-build-script.sh" ''
    set -eu -o pipefail
    STACK2NIX_OUTPUT_PATH=$(${stack2nix-script})
    export NIX_PATH=nixpkgs=${pkgs.path}
    ${pkgs.nix}/bin/nix-build --no-link -A static_package --argstr stack2nix-output-path "$STACK2NIX_OUTPUT_PATH" "$@"
  '';

in
  {
    static_package = static-stack2nix-builder.static_package;
    inherit fullBuildScript;
    # For debugging:
    inherit stack2nix-script;
    inherit static-stack2nix-builder;
  }

Please and thanks for any aid you can offer.

[nh2]

/nix/store/aafcj2cnpqrs4vrlj5i66z1nhr0x20jd-python3.7-pyOpenSSL-19.0.0.drv was built fine on my machine/CI.

Can you post the build error that made it fail?

Perhaps it's a test failure? Are you running with the Nix sandbox enabled?

If you use my cachix cache, you should be able to build it without probelms because the pyOpenSSL would be fetched from the cache. If you want to build from source, we need to check why it failed.

[nh2]

(I've also just confirmed that your Aura master builds fine for me statically.)

[nh2]

I suspect your problem will be this: https://github.com/NixOS/nixpkgs/issues/76879 (the PyOpenSSL test suite having a year 2020 bug).

[nh2]

@fosskers PR #76 should fix it, please try to update your line

      else fetchTarball https://github.com/nh2/static-haskell-nix/archive/d24dea3d46a727e1cd93e67458ce2768109efe0a.tar.gz;

to put in the commit from that PR.

[fosskers]

Doesn't seem to fix it, unfortunately...

=================================== FAILURES ===================================
____________________ TestContext.test_add_extra_chain_cert _____________________

self = <tests.test_ssl.TestContext object at 0x7ffff6241f90>
tmpdir = local('/build/pytest-of-nixbld/pytest-0/test_add_extra_chain_cert0')

    def test_add_extra_chain_cert(self, tmpdir):
        """
        `Context.add_extra_chain_cert` accepts an `X509`
        instance to add to the certificate chain.

        See `_create_certificate_chain` for the details of the
        certificate chain tested.

        The chain is tested by starting a server with scert and connecting
        to it with a client which trusts cacert and requires verification to
        succeed.
        """
        chain = _create_certificate_chain()
        [(cakey, cacert), (ikey, icert), (skey, scert)] = chain

        # Dump the CA certificate to a file because that's the only way to load
        # it as a trusted CA in the client context.
        for cert, name in [(cacert, 'ca.pem'),
                           (icert, 'i.pem'),
                           (scert, 's.pem')]:
            with tmpdir.join(name).open('w') as f:
                f.write(dump_certificate(FILETYPE_PEM, cert).decode('ascii'))

        for key, name in [(cakey, 'ca.key'),
                          (ikey, 'i.key'),
                          (skey, 's.key')]:
            with tmpdir.join(name).open('w') as f:
                f.write(dump_privatekey(FILETYPE_PEM, key).decode('ascii'))

        # Create the server context
        serverContext = Context(TLSv1_METHOD)
        serverContext.use_privatekey(skey)
        serverContext.use_certificate(scert)
        # The client already has cacert, we only need to give them icert.
        serverContext.add_extra_chain_cert(icert)

        # Create the client
        clientContext = Context(TLSv1_METHOD)
        clientContext.set_verify(
            VERIFY_PEER | VERIFY_FAIL_IF_NO_PEER_CERT, verify_cb)
        clientContext.load_verify_locations(str(tmpdir.join("ca.pem")))

        # Try it out.
>       self._handshake_test(serverContext, clientContext)

tests/test_ssl.py:1370: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_ssl.py:1248: in _handshake_test
    s.do_handshake()
/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/SSL.py:1915: in do_handshake
    self._raise_ssl_error(self._ssl, result)
/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/SSL.py:1647: in _raise_ssl_error
    _raise_current_error()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

exception_type = <class 'OpenSSL.SSL.Error'>

    def exception_from_error_queue(exception_type):
        """
        Convert an OpenSSL library failure into a Python exception.

        When a call to the native OpenSSL library fails, this is usually signalled
        by the return value, and an error code is stored in an error queue
        associated with the current thread. The err library provides functions to
        obtain these error codes and textual error messages.
        """
        errors = []

        while True:
            error = lib.ERR_get_error()
            if error == 0:
                break
            errors.append((
                text(lib.ERR_lib_error_string(error)),
                text(lib.ERR_func_error_string(error)),
                text(lib.ERR_reason_error_string(error))))

>       raise exception_type(errors)
E       OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/_util.py:54: Error
______________ TestContext.test_use_certificate_chain_file_bytes _______________

self = <tests.test_ssl.TestContext object at 0x7ffff62655d0>
tmpfile = b'/build/pytest-of-nixbld/pytest-0/tmp_jg89a96'

    def test_use_certificate_chain_file_bytes(self, tmpfile):
        """
        ``Context.use_certificate_chain_file`` accepts the name of a file (as
        an instance of ``bytes``) to specify additional certificates to use to
        construct and verify a trust chain.
        """
        self._use_certificate_chain_file_test(
>           tmpfile + NON_ASCII.encode(getfilesystemencoding())
        )

tests/test_ssl.py:1417: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_ssl.py:1408: in _use_certificate_chain_file_test
    self._handshake_test(serverContext, clientContext)
tests/test_ssl.py:1248: in _handshake_test
    s.do_handshake()
/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/SSL.py:1915: in do_handshake
    self._raise_ssl_error(self._ssl, result)
/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/SSL.py:1647: in _raise_ssl_error
    _raise_current_error()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

exception_type = <class 'OpenSSL.SSL.Error'>

    def exception_from_error_queue(exception_type):
        """
        Convert an OpenSSL library failure into a Python exception.

        When a call to the native OpenSSL library fails, this is usually signalled
        by the return value, and an error code is stored in an error queue
        associated with the current thread. The err library provides functions to
        obtain these error codes and textual error messages.
        """
        errors = []

        while True:
            error = lib.ERR_get_error()
            if error == 0:
                break
            errors.append((
                text(lib.ERR_lib_error_string(error)),
                text(lib.ERR_func_error_string(error)),
                text(lib.ERR_reason_error_string(error))))

>       raise exception_type(errors)
E       OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/_util.py:54: Error
_____________ TestContext.test_use_certificate_chain_file_unicode ______________

self = <tests.test_ssl.TestContext object at 0x7ffff6280cd0>
tmpfile = b'/build/pytest-of-nixbld/pytest-0/tmp7xd3qnko'

    def test_use_certificate_chain_file_unicode(self, tmpfile):
        """
        ``Context.use_certificate_chain_file`` accepts the name of a file (as
        an instance of ``unicode``) to specify additional certificates to use
        to construct and verify a trust chain.
        """
        self._use_certificate_chain_file_test(
>           tmpfile.decode(getfilesystemencoding()) + NON_ASCII
        )

tests/test_ssl.py:1427: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_ssl.py:1408: in _use_certificate_chain_file_test
    self._handshake_test(serverContext, clientContext)
tests/test_ssl.py:1248: in _handshake_test
    s.do_handshake()
/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/SSL.py:1915: in do_handshake
    self._raise_ssl_error(self._ssl, result)
/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/SSL.py:1647: in _raise_ssl_error
    _raise_current_error()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

exception_type = <class 'OpenSSL.SSL.Error'>

    def exception_from_error_queue(exception_type):
        """
        Convert an OpenSSL library failure into a Python exception.

        When a call to the native OpenSSL library fails, this is usually signalled
        by the return value, and an error code is stored in an error queue
        associated with the current thread. The err library provides functions to
        obtain these error codes and textual error messages.
        """
        errors = []

        while True:
            error = lib.ERR_get_error()
            if error == 0:
                break
            errors.append((
                text(lib.ERR_lib_error_string(error)),
                text(lib.ERR_func_error_string(error)),
                text(lib.ERR_reason_error_string(error))))

>       raise exception_type(errors)
E       OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

/nix/store/0d9fkw7lk0vvm8rmzsg1y7xfz88g987m-python3.7-pyOpenSSL-19.0.0/lib/python3.7/site-packages/OpenSSL/_util.py:54: Error

[nh2]

@fosskers Ah sorry, I only bumped the nixpkgs submodule, not the tarball hash. Fixed now. Please try

      # Update this hash to use a different `static-haskell-nix` version:
      else fetchTarball https://github.com/nh2/static-haskell-nix/archive/d1f63679627d5322594736033a9cb1fc47358ee7.tar.gz;

I've also updated the PR.

[fosskers]

Unfortunately it still fails :)

[nh2]

@fosskers My bad, I had conflated the commit of nixpkgs with the one for static-haskell-nix.

I've fixed it and also put the right one here into my Aura fork:

https://github.com/nh2/aura/blob/static-haskell-nix-issue-75/default.nix

Please give that a try, sorry for the inconvenience :)

[fosskers]

We got past the openssl issue! However...

Compilation of GHC has died twice now, trying a third time.

make[1]: *** [libraries/Cabal/Cabal/ghc.mk:4: libraries/Cabal/Cabal/dist-install/build/Distribution/SPDX/LicenseId.o] Segmentation fault (core dumped)
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [compiler/ghc.mk:447: compiler/stage2/build/DynFlags.p_o] Segmentation fault (core dumped)
make: *** [Makefile:127: all] Error 2
builder for '/nix/store/30c22iymb1abrgnn40nr5qdvfw4x4hy0-ghc-8.6.5.drv' failed with exit code 2
cannot build derivation '/nix/store/hnl9zm6ikkhxvgxxzkkr6v85njfwnjfs-aura-2.0.3.drv': 1 dependencies couldn't be built
error: build of '/nix/store/hnl9zm6ikkhxvgxxzkkr6v85njfwnjfs-aura-2.0.3.drv' failed

[nh2]

Oh, segfault! That is unexpected.

[nh2]

I find it a bit suspicious that 2 ghcs segfault at the same time, that is very odd.

[fosskers]

I'd have to double-check that, they might not have been at the same location.

[nh2]

I'm not sure what to do about this, I cannot reproduce a segfault.

It says (core dumped), do you have a core file present whose stack trace we could look at in gdb? Otherwise, perhaps you can enable core file dumping explicitly?

[fosskers]

make[1]: *** [libraries/template-haskell/ghc.mk:4: libraries/template-haskell/dist-install/build/Language/Haskell/TH/Syntax.p_o] Bus error (core dumped)
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:127: all] Error 2
builder for '/nix/store/30c22iymb1abrgnn40nr5qdvfw4x4hy0-ghc-8.6.5.drv' failed with exit code 2
cannot build derivation '/nix/store/hnl9zm6ikkhxvgxxzkkr6v85njfwnjfs-aura-2.0.3.drv': 1 dependencies couldn't be built
error: build of '/nix/store/hnl9zm6ikkhxvgxxzkkr6v85njfwnjfs-aura-2.0.3.drv' failed

A different location than last time.

[nh2]

Sounds like either bad RAM or something being fundamentally messed up in some software.

Could you run memcheck to exclude the former (it's unlikely but easier to check than debugging), and otherwise check if you can get a core dump file?