search

ContinuumIO / anaconda-issues

Anaconda issue tracking
646 stars 220 forks source link

SSL: CERTIFICATE_VERIFY_FAILED with openssl=1.1.1d (latest available version of openssl) #11274

Open lesteve opened 4 years ago

lesteve commented 4 years ago

Actual Behavior

Error when doing urllib.request.urlopen('https://pypi.org')

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

Note: since openssl 1.1.1d is the latest version this likely affects all the new created conda environments (including the CIs of the project using conda).

This may be related to https://github.com/ContinuumIO/anaconda-issues/issues/11205 which was quite recent, not entirely sure.

Expected Behavior

No error when doing `urllib.request.urlopen('https://pypi.org').

Steps to Reproduce

openssl 1.1.1d

conda create -n testenv python -y
conda activate testenv
python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'

Output:

Traceback (most recent call last):
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/http/client.py", line 1414, in connect
    server_hostname=server_hostname)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/home/lesteve/miniconda3/envs/testenv/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)>

openssl 1.1.1c

conda create -n testenv-openssl-1.1.1c python openssl=1.1.1c -y
conda activate testenv-openssl-1.1.1c
python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'

This completes without error.

Anaconda or Miniconda version:
Operating System:
conda info
``` active environment : testenv active env location : /home/lesteve/miniconda3/envs/testenv shell level : 2 user config file : /home/lesteve/.condarc populated config files : /home/lesteve/miniconda3/.condarc conda version : 4.7.11 conda-build version : not installed python version : 3.7.3.final.0 virtual packages : base environment : /home/lesteve/miniconda3 (writable) channel URLs : https://conda.anaconda.org/conda-forge/label/pre-3.8/linux-64 https://conda.anaconda.org/conda-forge/label/pre-3.8/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /home/lesteve/miniconda3/pkgs /home/lesteve/.conda/pkgs envs directories : /home/lesteve/miniconda3/envs /home/lesteve/.conda/envs platform : linux-64 user-agent : conda/4.7.11 requests/2.22.0 CPython/3.7.3 Linux/4.15.0-60-generic ubuntu/16.04.6 glibc/2.23 UID:GID : 202587:26082 netrc file : None offline mode : False ```
conda list --show-channel-urls
``` # packages in environment at /home/lesteve/miniconda3/envs/testenv: # # Name Version Build Channel _libgcc_mutex 0.1 main defaults ca-certificates 2019.5.15 1 defaults certifi 2019.6.16 py37_1 defaults libedit 3.1.20181209 hc058e9b_0 defaults libffi 3.2.1 hd88cf55_4 defaults libgcc-ng 9.1.0 hdf63c60_0 defaults libstdcxx-ng 9.1.0 hdf63c60_0 defaults ncurses 6.1 he6710b0_1 defaults openssl 1.1.1d h7b6447c_0 defaults pip 19.2.2 py37_0 defaults python 3.7.4 h265db76_1 defaults readline 7.0 h7b6447c_5 defaults setuptools 41.0.1 py37_0 defaults sqlite 3.29.0 h7b6447c_0 defaults tk 8.6.8 hbc83047_0 defaults wheel 0.33.4 py37_0 defaults xz 5.2.4 h14c3975_4 defaults zlib 1.2.11 h7b6447c_3 defaults ```
ogrisel commented 4 years ago

Another slightly different case that also fails with the same error:

conda create -n testenv python=3.5 -y
conda activate testenv
python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'

=> ssl.SSLError

although in this case the openssl version is: 

$ conda list | grep openssl
openssl                   1.0.2t               h7b6447c_0

Downgrading to 1.0.2r fixes the issue:

conda install openssl=1.0.2r
python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'

=> everything is fine again
mingwandroid commented 4 years ago

Confirmed, it seems prefix replacement hasn't happened in lib/libssl.1.1.dylib. Looking into it now. After an emergency fix (Python 3 then 2 after, thanks @orgisel) I will also add a downstream test to the openssl package so that our python tests (which now cover this I believe!) get run on each rebuild. I think as this is just embarrassing at this stage. Thank you so much for helping us though!

mingwandroid commented 4 years ago

Test packages for openssl 1.1.1d build number 1 for macOS at https://anaconda.org/rdonnelly if anyone's got some cycles to test? About to do so myself!

mingwandroid commented 4 years ago

Works on my computer.

ccordoba12 commented 4 years ago

@mingwandroid, this problem is also present on Linux. We discovered that yesterday in our CI builds. See for example this one:

https://travis-ci.org/spyder-ide/spyder/jobs/583712016#L1994

I don't know why the link doesn't work correctly, but please look for line 1994. The error is just below that line.

mingwandroid commented 4 years ago

I know it is @ccrodoba12!

mingwandroid commented 4 years ago

Be right with you with something to test.

ccordoba12 commented 4 years ago

I know it is @ccrodoba12!

Sorry, since you said above

Confirmed, it seems prefix replacement hasn't happened in lib/libssl.1.1.dylib

I thought you were considering to fix it only on macOS.

Be right with you with something to test.

Ok, thanks!

mingwandroid commented 4 years ago

Please test. The linux ones are release candidates. I need to build release candidate macOS on the build infra next.

lesteve commented 4 years ago

It works fine for me on Linux using this command (hoping that this was what you had in mind):

❯ conda create -n testenv -c rdonnelly python -y
conda activate testenv
python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'

For completeness, here is the content of the environment:

❯ conda list
# packages in environment at /home/local/lesteve/miniconda3/envs/testenv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
ca-certificates           2019.5.15                     1  
certifi                   2019.6.16                py37_1  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
ncurses                   6.1                  he6710b0_1  
openssl                   1.1.1d               h7b6447c_1    rdonnelly
pip                       19.2.2                   py37_0  
python                    3.7.4                h265db76_1    rdonnelly
readline                  7.0                  h7b6447c_5  
setuptools                41.0.1                   py37_0  
sqlite                    3.29.0               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
wheel                     0.33.4                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               h7b6447c_3
mingwandroid commented 4 years ago

Probably want to remove my python build from the equation, but it probably doesn't matter. Thanks. I'll upload these soon.

ogrisel commented 4 years ago

It seems that I now get the build h7b6447c_1 for openssl-1.1.1d and that the original problem reported by @lesteve has been fixed.

The problem still occurs for Python 3.5 with openssl-1.0.2t / build h7b6447c_0.

mingwandroid commented 4 years ago

Great. 1.0.2t builds are in progress. Will reply when ready.

lesteve commented 4 years ago

It seems like both the openssl-1.1.1d-h7b6447c_1 and openssl-1.0.2t-h7b6447c_1 fixed the problems mentioned in this issue, so I am going to close this issue.

Thanks a lot @mingwandroid for the quick fix!

mingwandroid commented 4 years ago

Well, I'll keep it open if that's OK since we've not built 1.0.2t for macOS yet.

But thank you both very much for allowing me to jump on my mistake quickly!

mark-zajac commented 4 years ago

Great. 1.0.2t builds are in progress.

It has been a while. Are the builds still in progress? Under macOS 10.13.6 (17G7024), Anaconda Navigator is still refusing to update packages or create new environments. Out of desperation, I am hoping the new build might help, when it becomes available.

rjknight commented 4 years ago

I'm seeing the issue below when using opnessl=1.1.1d , I tried installing from the rdonnelly channel to see if it would work but I see the same failure.

[builder@c7858716ac27 ~]$ conda list openssl
# packages in environment at /opt/anaconda3/envs/blah-blah:
#
# Name                    Version                   Build  Channel
openssl                   1.1.1d               h7b6447c_1    rdonnelly
[builder@c7858716ac27 ~]$  python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/urllib/request.py", line 1286, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/http/client.py", line 1392, in __init__
    context = ssl._create_default_https_context()
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/ssl.py", line 502, in create_default_context
    context = SSLContext(PROTOCOL_TLS)
  File "/opt/anaconda3/envs/blah-blah/lib/python3.6/ssl.py", line 391, in __new__
    self = _SSLContext.__new__(cls, protocol)
ssl.SSLError: [SSL] malloc failure (_ssl.c:2805)
[builder@c7858716ac27 ~]$ conda info

     active environment : blah-blah
    active env location : /opt/anaconda3/envs/blah-blah
            shell level : 2
       user config file : /home/builder/.condarc
 populated config files : /home/builder/.condarc
          conda version : 4.6.14
    conda-build version : 3.18.9
         python version : 3.6.8.final.0
       base environment : /opt/anaconda3  (writable)
           channel URLs : <removed private channels>
                          https://repo.anaconda.com/pkgs/main/linux-ppc64le
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-ppc64le
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-ppc64le
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/anaconda3/pkgs
                          /home/builder/.conda/pkgs
       envs directories : /opt/anaconda3/envs
                          /home/builder/.conda/envs
               platform : linux-ppc64le
             user-agent : conda/4.6.14 requests/2.22.0 CPython/3.6.8 Linux/4.14.0-115.6.1.el7a.ppc64le rhel/7.6 glibc/2.17
                UID:GID : 1084:1500
             netrc file : None
           offline mode : False

if I downgrade opnessl using conda install openssl=1.1.1c then I no longer see the failures.

mingwandroid commented 4 years ago

platform : linux-ppc64le

Strange! Can you try with Python 3.7 on ppc64le and 1.1.1d build #1 from my channel?

rjknight commented 4 years ago

same results with py37 

(test-1.1.1d-py37) [builder@b98996cfec6a ~]$ conda list python
# packages in environment at /opt/anaconda3/envs/test-1.1.1d-py37:
#
# Name                    Version                   Build  Channel
python                    3.7.4                h2bede3c_1  
(test-1.1.1d-py37) [builder@b98996cfec6a ~]$ conda list openssl
# packages in environment at /opt/anaconda3/envs/test-1.1.1d-py37:
#
# Name                    Version                   Build  Channel
openssl                   1.1.1d               h7b6447c_1    rdonnelly
(test-1.1.1d-py37) [builder@b98996cfec6a ~]$ python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/urllib/request.py", line 1285, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/http/client.py", line 1383, in __init__
    context = ssl._create_default_https_context()
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/ssl.py", line 576, in create_default_context
    context = SSLContext(PROTOCOL_TLS)
  File "/opt/anaconda3/envs/test-1.1.1d-py37/lib/python3.7/ssl.py", line 399, in __new__
    self = _SSLContext.__new__(cls, protocol)
ssl.SSLError: [SSL] malloc failure (_ssl.c:3014)
(test-1.1.1d-py37) [builder@b98996cfec6a ~]$ conda info

     active environment : test-1.1.1d-py37
    active env location : /opt/anaconda3/envs/test-1.1.1d-py37
            shell level : 2
       user config file : /home/builder/.condarc
 populated config files : /home/builder/.condarc
          conda version : 4.6.14
    conda-build version : 3.18.9
         python version : 3.6.9.final.0
       base environment : /opt/anaconda3  (writable)
           channel URLs : 
                          https://repo.anaconda.com/pkgs/main/linux-ppc64le
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-ppc64le
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-ppc64le
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/anaconda3/pkgs
                          /home/builder/.conda/pkgs
       envs directories : /opt/anaconda3/envs
                          /home/builder/.conda/envs
               platform : linux-ppc64le
             user-agent : conda/4.6.14 requests/2.22.0 CPython/3.6.9 Linux/4.14.0-115.6.1.el7a.ppc64le rhel/7.6 glibc/2.17
                UID:GID : 1084:1500
             netrc file : None
           offline mode : False

full environment list

(test-1.1.1d-py37) [builder@b98996cfec6a ~]$ conda list
# packages in environment at /opt/anaconda3/envs/test-1.1.1d-py37:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
ca-certificates           2019.5.15                     1  
certifi                   2019.6.16                py37_1  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hf62a594_5  
libgcc-ng                 8.2.0                h822a55f_1  
libstdcxx-ng              8.2.0                h822a55f_1  
ncurses                   6.1                  he6710b0_1  
openssl                   1.1.1d               h7b6447c_1    rdonnelly
pip                       19.2.2                   py37_0  
python                    3.7.4                h2bede3c_1  
readline                  7.0                  h7b6447c_5  
setuptools                41.0.1                   py37_0  
sqlite                    3.29.0               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
wheel                     0.33.4                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               h7b6447c_3
mingwandroid commented 4 years ago

ssl.SSLError: [SSL] malloc failure (_ssl.c:3014)

How much memory do you have? I'm almost certain this ppc64le issue is unrelated to the issue being reported here, the call stacks are completely different (same module failing being the only similarity). I cannot spend a lot of time on ppc64le issues I am afraid. Can you try packages from conda-forge here? People there dedicate a lot of time to ppc64le packaging now.

mark-zajac commented 4 years ago

How much memory do you have?

16GB

Running under macOS 10.13.6 , Anaconda Navigator refuses to create new environments and refuses to update packages. The "progress bar" whirls and spins, for a while but, ultimately the operation fails. I would love to give more information but I don't know where to look for Anaconda Navigator error logs.

mingwandroid commented 4 years ago

@mark-zajac, this is to me clearly a different bug. The stack trace is unrelated and the operating system different. While macOS (and Linux) definitely did suffer this problem (specifically SSL: CERTIFICATE_VERIFY_FAILED) they no longer do, however you have been reporting something quite different. Please open another issue instead.

I don't know where to look for Anaconda Navigator error logs.

You should use the command-line for any debugging purposes for such low level issues (look for Navigator bugs alone in Navigator, it defers to conda for all of its real work).

It probably does keep logs but I don't use it.

mingwandroid commented 4 years ago

Thank you for your help @rjknight

Bringing this to @msarahan and @csoja's attention. I haven't tried to reproduce this personally and it's ppc64le, still if SSL in our latest Python isn't working then that's not good. I suspect it may be something in the sysconfigdata stuff.

@rjknight for a final bit of testing, could you try swapping out some of the likely implicated packages for equiv. ones from conda-forge to see if we can isolate things further?

I would start with:

conda install conda-forge::python

.. then move on to openssl, ca-certificates then certifi.

mark-zajac commented 4 years ago

Please open another issue instead.

I did. In the only reply somebody linked to this thread. Where should I post in hopes of a response to my issue?

(look for Navigator bugs alone in Navigator, it defers to conda for all of its real work).

Exactly! My problem is a bug with Navigator. I am able to create and switch environments just fine using the command line. It is Anaconda Navigator that is not working. Where do I report that bug for some help?

You should use the command-line

I, myself, am perfectly comfortable with the terminal and the command-line version. However, a working GUI lowers the bar for my students. The textbook that I am using provides step-by-step instructions for Anaconda Navigator but those steps are failing.

I had been using Anaconda Navigator for about a year, before adopting it for my class. Now, suddenly, it simply does not work (in macOS) and there seems nowhere to turn for help.

I know a little about computers and could try to debug things myself, if only Anaconda Navigator gave some error messages. It is hard to solve my own problem without information. There has been no response to my other posts in this "anaconda-issues" forum.

mingwandroid commented 4 years ago

@mark-zajac apologies for not seeing your issue earlier, but please let's keep discussion about that, there instead of here.

mark-zajac commented 4 years ago

apologies

You are gracious. The fault is mine for posting in the wrong place. I only posted here because the only previous reply to my issue cross-linked to this post. I assumed that the issues were related.

mingwandroid commented 4 years ago

No problem! Thanks for sticking with me.

rjknight commented 4 years ago

I replaced each of the packages mentioned one-at-a-time by the conda forge ones, I still see the isssue though.

I then created an env (py36) using the conda-forge channel, openssl from conda forge chooses the 1.1.1c version so I reinstalled 1.1.1d from the default channel, I still see the failure

(conda-forge-test) [builder@b98996cfec6a pytorch]$ python -c 'import urllib.request; urllib.request.urlopen("https://pypi.org")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/urllib/request.py", line 1286, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/http/client.py", line 1377, in __init__
    context = ssl._create_default_https_context()
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/ssl.py", line 502, in create_default_context
    context = SSLContext(PROTOCOL_TLS)
  File "/opt/anaconda3/envs/conda-forge-test/lib/python3.6/ssl.py", line 391, in __new__
    self = _SSLContext.__new__(cls, protocol)
ssl.SSLError: [SSL] malloc failure (_ssl.c:2788)
(conda-forge-test) [builder@b98996cfec6a pytorch]$ conda list
# packages in environment at /opt/anaconda3/envs/conda-forge-test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
ca-certificates           2019.9.11            hecc5488_0    conda-forge
certifi                   2019.9.11                py36_0    conda-forge
libffi                    3.2.1             hb209c28_1006    conda-forge
libgcc-ng                 8.2.0                h822a55f_1  
libstdcxx-ng              8.2.0                h822a55f_1  
ncurses                   6.1               hf484d3e_1002    conda-forge
openssl                   1.1.1d               h7b6447c_1  
pip                       19.2.3                   py36_0    conda-forge
python                    3.6.7             h88bc6d3_1005    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
setuptools                41.2.0                   py36_0    conda-forge
sqlite                    3.29.0               hd61ad8c_1    conda-forge
tk                        8.6.9             h151fe60_1003    conda-forge
wheel                     0.33.6                   py36_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h6eb9509_1006    conda-forge
stevehaertel commented 4 years ago

For the record, it looks like openssl 1.1.1d breaks Jupyter notebook / JEG on PPC64

Traceback (most recent call last):
  File "/tmp/minip/anaconda/envs/env/bin/jupyter-enterprisegateway", line 6, in <module>
    from enterprise_gateway import launch_instance
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/enterprise_gateway/__init__.py", line 4, in <module>
    from .enterprisegatewayapp import launch_instance
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/enterprise_gateway/enterprisegatewayapp.py", line 19, in <module>
    from notebook.services.kernels.kernelmanager import MappingKernelManager
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 15, in <module>
    from tornado import gen, web
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/tornado/web.py", line 87, in <module>
    from tornado.httpserver import HTTPServer
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/tornado/httpserver.py", line 32, in <module>
    from tornado.http1connection import HTTP1ServerConnection, HTTP1ConnectionParameters
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/tornado/http1connection.py", line 34, in <module>
    from tornado import iostream
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/tornado/iostream.py", line 40, in <module>
    from tornado.netutil import ssl_wrap_socket, _client_ssl_defaults, _server_ssl_defaults
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/site-packages/tornado/netutil.py", line 39, in <module>
    _client_ssl_defaults = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/ssl.py", line 565, in create_default_context
    context = SSLContext(PROTOCOL_TLS)
  File "/tmp/minip/anaconda/envs/env/lib/python3.7/ssl.py", line 388, in __new__
    self = _SSLContext.__new__(cls, protocol)
ssl.SSLError: [SSL] malloc failure (_ssl.c:2994)

While it's unclear to me if that malloc failure is related to any of the others, in this thread, I'd rather be safe than sorry, and bring it up here too, than let it go.

rjknight commented 4 years ago

I poked around a little in the openssl code today, using git bisect I narrowed it down to commit https://github.com/openssl/openssl/commit/3ff98f558157ada23a3527d82793a524eb8263ec as causing the problem, see https://github.com/openssl/openssl/pull/9595 for details on the commit.
I disabled the change by setting -DOPENSSL_RAND_SEED_DEVRANDOM_SHM_ID=-1 as suggested in the PR and was able to run the previously failing test successfully.

I made the following change in build.sh - 

 diff --git a/recipe/build.sh b/recipe/build.sh
index a6cb10d..2155c45 100644
--- a/recipe/build.sh
+++ b/recipe/build.sh
@@ -29,7 +29,7 @@ if [[ ${_BASE_CC} == *-* ]]; then
       ;;
     *powerpc64le-*linux*)
       _CONFIG_OPTS+=(linux-ppc64le)
-      CFLAGS="${CFLAGS} -Wa,--noexecstack"
+      CFLAGS="${CFLAGS} -Wa,--noexecstack -DOPENSSL_RAND_SEED_DEVRANDOM_SHM_ID=-1"
       ;;
     *darwin*)

I don't think I have enough context regarding the change to figure out the root cause, we will probably need to get some assistance from the OpenSSL team.

rjknight commented 4 years ago

I opened https://github.com/openssl/openssl/issues/10015 for assistance from the OpenSSL team

mingwandroid commented 4 years ago

Thank you for taking care of this so well @rjknight!

rjknight commented 4 years ago

https://github.com/openssl/openssl/issues/10015 has been fixed upstream - does the fix need to be included in an openssl release before it would be picked up in the recipe?

jjhelmus commented 4 years ago

does the fix need to be included in an openssl release before it would be picked up in the recipe?

I pulled this fix into the openssl recipe. Build 3 packages includes the change.

rjknight commented 4 years ago

@jjhelmus Thanks!