Open marc-wien opened 3 years ago
The thread which culminates in this fix seems relevant... Still beyond me what might need to be looked into here though. Also I confirmed mamba 0.15.3 didn't "magically" fix it
It says CERT_TRUST_REVOCATION_STATUS_UNKNOWN . I'll have to look into what that means
setting MAMBA_SSL_NO_REVOKE works for me
$env:MAMBA_SSL_NO_REVOKE=1
setting MAMBA_SSL_NO_REVOKE works for me
$env:MAMBA_SSL_NO_REVOKE=1
530
Good news/bad news -- This gets me past the "revocation" issue, thank you @shulinwz !!! Now getting a different error message on the same line:
Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
Let me write up what my latest test looks like now.
This post shows where I get the Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
error message relating to the SSL certificate file.
My .condarc file:
channels:
- conda-forge
channel_priority: strict
ssl_verify: C:\\path\\certificate.cer
In Anaconda Prompt:
(base) PS C:\Users\user> mamba --version
mamba 0.27.0
conda 22.9.0
(base) PS C:\Users\user> conda env config vars list
MAMBA_SSL_NO_REVOKE = 1
When MAMBA_SSL_NO_REVOKE
is not set, I get the different error from the earlier comments in this thread.
Note: I comment at the bottom about how the same error occurs even if I have CURL_CA_BUNDLE
, REQUESTS_CA_BUNDLE
, and SSL_CERT_FILE
variables set here.
(base) PS C:\Users\user> conda search python
Loading channels: done
# Name Version Build Channel
python 2.7.12 0 conda-forge
python 2.7.12 1 conda-forge
<snip>
conda search
will populate the index cache, which is shown here just to help narrow down the error conditions. mamba
will work in the next command because of the cache...
(base) PS C:\Users\user> mamba repoquery search python --no-banner
Executing the query python
conda-forge/win-64 Using cache
conda-forge/noarch Using cache
Name Version Build Channel
----------------------------------------------------------
python 3.10.6 h9a09f29_0_cpython conda-forge
python 3.10.6 hcf16a7b_0_cpython conda-forge/win-64
<snip>
mamba repoquery search
works here! Because it's using the index cache. However, it will fail when the cache is cleared.
(base) PS C:\Users\user> conda clean -i -y
Will remove 1 index cache(s).
Clear the index cache. Now mamba
will fail...
(base) PS C:\Users\user> mamba repoquery search python --no-banner
Executing the query python
Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "C:\Users\user\Miniconda3\lib\site-packages\conda\exceptions.py", line 1129, in __call__
return func(*args, **kwargs)
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\mamba.py", line 935, in exception_converter
raise e
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\mamba.py", line 928, in exception_converter
exit_code = _wrapped_main(*args, **kwargs)
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\mamba.py", line 886, in _wrapped_main
result = do_call(parsed_args, p)
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\mamba.py", line 760, in do_call
exit_code = repoquery(args, parser)
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\mamba.py", line 702, in repoquery
pool = repoquery_api.create_pool(channels, platform, use_installed)
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\repoquery.py", line 46, in create_pool
load_channels(
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\utils.py", line 129, in load_channels
index = get_index(
File "C:\Users\user\Miniconda3\lib\site-packages\mamba\utils.py", line 110, in get_index
is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST)
RuntimeError: Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
Now, the biggest lead that I had at this point was trying to set the CURL_CA_BUNDLE
environment variable, but the same error occurs even if all of the following environment variables are set:
(base) PS C:\Users\user> conda env config vars list
MAMBA_SSL_NO_REVOKE = 1
CURL_CA_BUNDLE = C:\\path\\certificate.cer
SSL_CERT_FILE = C:\\path\\certificate.cer
REQUESTS_CA_BUNDLE = C:\\path\\certificate.cer
Any thoughts? Other things I should try? Best, Marc
@wolfv Wanted to point out that this cert error happens similarly with Micromamba as well. I can post debug info from that tomorrow or later this week. Any pointers on what to test would be greatly appreciated!
Do you have any settings to successfully allow repodata.json
download with the curl
binary in your environment?
@jonashaag thanks for the response.
Strangely enough, the curl
command with no other modifications works standalone in this environment:
(base) PS C:\Users\user> curl https://conda.anaconda.org/conda-forge/noarch/repodata.json
StatusCode : 200
StatusDescription : OK
Content : {
<snip>
And this works whether I have CURL_CA_BUNDLE
set to "" (empty) or to "C:\path\certificate.cer"
I don't have a user .curlrc
file anywhere to my knowledge... mamba and micromamba are the only programs I haven't been able to figure out a fix for...
If it helps at all, the ssl_verify: C:\\path\\certificate.cer
line in my .condarc
explicitly solved this problem for Conda.
Without that line (or explicitly setting ssl_verify: true
), I get the following error:
(base) PS C:\Users\user > conda search python
Loading channels: failed
CondaSSLError: Encountered an SSL error. Most likely a certificate verification issue.
Exception: HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /conda-forge/win-64/repodata.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)')))
Do you think a look at how Conda handles the ssl_verify
variable might help illuminate what needs to be done analogously in my case in Curl via Mamba (or some other config solution)? Should Mamba pull this entry from .condarc
directly?
I think what you ran is in fact NOT curl but something built in to PowerShell that’s called curl
I think what you ran is in fact NOT curl but something built in to PowerShell that’s called curl
Wow, good catch. That's goofy, I had no idea. Here's using Anaconda Prompt (cmd.exe version):
I just need --ssl-no-revoke
for Curl to work. Note that the env vars are empty
(base) C:\Users\user>conda env config vars list
MAMBA_SSL_NO_REVOKE = 1
CURL_CA_BUNDLE =
SSL_CERT_FILE =
REQUESTS_CA_BUNDLE =
(base) C:\Users\user>conda clean -i -y
There are no index cache(s) to remove.
(base) C:\Users\user>curl --version
curl 7.83.1 (Windows) libcurl/7.83.1 Schannel
Release-Date: 2022-05-13
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets
(base) C:\Users\user>curl --help
Usage: curl [options...] <url>
-d, --data <data> HTTP POST data
-f, --fail Fail fast with no output on HTTP errors
-h, --help <category> Get help for commands
-i, --include Include protocol response headers in the output
-o, --output <file> Write to file instead of stdout
-O, --remote-name Write output to a file named as the remote file
-s, --silent Silent mode
-T, --upload-file <file> Transfer local FILE to destination
-u, --user <user:password> Server user and password
-A, --user-agent <name> Send User-Agent <name> to server
-v, --verbose Make the operation more talkative
-V, --version Show version number and quit
This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".
(base) C:\Users\user>curl https://conda.anaconda.org/conda-forge/noarch/repodata.json
curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - The revocation function was unable to check revocation for the certificate.
(base) C:\Users\user>(base) C:\Users\user>curl --ssl-no-revoke https://conda.anaconda.org/conda-forge/noarch/repodata.json > output.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 69.6M 100 69.6M 0 0 6762k 0 0:00:10 0:00:10 --:--:-- 772k
(base) C:\Users\user>
All I needed was --ssl-no-revoke
, and output.txt looks like a ~2.5 M line JSON, e.g.:
{
"info": {
"subdir": "noarch"
},
"packages": {
"_current_repodata_hack_gcc_linux_64_75-0.0.1-0.tar.bz2": {
"build": "0",
"build_number": 0,
"depends": [
"gcc_linux-64 7.5.*"
],
"license": "LicenseRef-OTHER",
"md5": "6f29ba77e8b03b191c9d667f331bf2a0",
"name": "_current_repodata_hack_gcc_linux_64_75",
"noarch": "generic",
"sha256": "ecde63af23e0d49c0ece19ec539d873ea408a6f966d3126994c6d33ae1b9d3f7",
"size": 3357,
"subdir": "noarch",
"timestamp": 1599854591823,
"version": "0.0.1"
},
<snip>
Hm that's curious, I wonder why you don't even need to provide the certificate file to curl? Are there any environment variables or system settings that may impact curl? From the code ssl_verify
seems to set CURLOPT_CAINFO
, I think this is the same as --cacert
(?), could you try multiple values with that curl flag?
Also could you run Mamba with -vv
, that should show some more curl logs, I'm expecting something like this
...
successfully set certificate verify locations:
...
Hm that's curious, I wonder why you don't even need to provide the certificate file to curl? Are there any environment variables or system settings that may impact curl? From the code
ssl_verify
seems to setCURLOPT_CAINFO
, I think this is the same as--cacert
(?), could you try multiple values with that curl flag?Also could you run Mamba with
-vv
, that should show some more curl logs, I'm expecting something like this... successfully set certificate verify locations: ...
Quick look at Mamba call that fails, with -vv
output:
There is a line (different phrase than you used):
info libcurl * schannel: added 1 certificate(s) from CA file 'C:\\path\\certificate.cer'
The final log outputs when failing are:
<snip>
info libcurl * schannel: failed to decrypt data, need more data
info libcurl * schannel: failed to decrypt data, need more data
info libcurl * schannel: failed to decrypt data, need more data
info libcurl * schannel: server closed abruptly (missing close_notify)
info libcurl * Closing connection 5
info libcurl * schannel: shutting down SSL/TLS connection with conda.anaconda.org port 443
info libmamba Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
DEBUG conda.gateways.logging:set_verbosity(236): verbosity set to 2
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
<snip>
I can post more detail from logs if needed but would need to take a bit to anonymize some things.
Couple thoughts:
I also have very little knowledge about configuring custom CAs correctly. Basically my approach is trying to reproduce the issue in curl and/or identifying the difference between curl invocation on the command line vs from Mamba.
I also have very little knowledge about configuring custom CAs correctly. Basically my approach is trying to reproduce the issue in curl and/or identifying the difference between curl invocation on the command line vs from Mamba.
Yeah, thanks regardless for helping me try to debug!
As it stands, this problem certainly adds a barrier to "corporate Windows" users with certain IT configurations from adopting Mamba and Micromamba considering the equivalent scenario "just works" with a single simple solution in Conda (which is equivalent as far as I can tell to how one would set up requests.get
). Unsure how common my IT configuration case is though.
That being said, I will ask around my workplace and see if others have worked this out, especially now that Mamba has gotten less and less niche since I originally posted the question (at least that's my perception :) )
I'll report back if I get something working
Hi, I have a very similar issue on macOS 12.6.2 and the latest micromamba 1.2.0.
We use a custom certificate in our corporate network, but it works fine in Python, pip and miniconda, unfortunately in micromamba I'm getting the following error message:
...
info libcurl * SSLRead() return error -9806
info libcurl * Closing connection 6
info libmamba Freeing pool.
critical libmamba Multiple errors occured:
Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
SSLRead() return error -9806
Subdir conda-forge/noarch not loaded!
I've tried multiple configurations using .condarc/.mambarc
files, various environment variables (SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE) and CLI args to make mamba curl work, but it just don't want to.
It's interesting, but my OS curl works like expected:
% curl --version
curl 7.79.1 (x86_64-apple-darwin21.0) libcurl/7.79.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets
% curl https://conda.anaconda.org/conda-forge/noarch/repodata.json -vvv -o tmp.json
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Trying 104.17.92.24:443...
* Connected to conda.anaconda.org (104.17.92.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations: <-- no such a line in micromamba logs!
* CAfile: $HOME/.../cacert.pem
...
I see that micromamba uses a newer version of curl:
% micromamba info
...
libmamba version : 1.2.0
micromamba version : 1.2.0
curl version : libcurl/7.86.0 SecureTransport (OpenSSL/1.1.1s) zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0
...
but it seems that the mamba curl is not picking my config at all (I reviewed verbose debug/trace logs from micromamba and couldn't find a line pointing to my cert file.
Is there a way to fallback to the system curl?
Any help will be much appreciated.
Can you please try a newer curl? Just install the latest one from conda-forge and try that.
@mopechowski-equinix can you add the certificate to your system? We try to load the certificates from the system.
We should check that we also respect variables like CURL_CA_BUNDLE
or the others you mentioned.
Thanks @jonashaag for your reply.
By install, do you mean brew install curl
? I haven't try this yet, but the curl available on my system works fine:
curl https://conda.anaconda.org/conda-forge/noarch/repodata.json
and I'm not able to install anything from conda-forge, because of the issue we have here ;-)
% micromamba install curl -c conda-forge -n base
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
critical libmamba Multiple errors occured:
Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
SSLRead() return error -9806
Subdir conda-forge/noarch not loaded!
I think it can work when you use --ssl-verify false
with micromamba. Yes, would be interesting if you can try the curl
from conda-forge
.
Thanks @wolfv, yes, I've got this custom cert in the macOS Keychain.
I've tried with --ssl-verify false
flag as well, but no luck:
% micromamba config list
channels:
- conda-forge
channel_priority: strict
show_banner: false
% micromamba create -f environment.yml --ssl-verify false
...
critical libmamba Multiple errors occured:
Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
SSLRead() return error -9806
Subdir conda-forge/noarch not loaded!
it seems that instead of the flag it still is trying to download data from https://...
.
I'll try to convert this cert according to this doc and install a newer version of curl.
You can also try the version from brew, which one is it?
I've tried everything and here are my findings:
When I use my macOS curl 7.79.1
(or the latest version from brew curl 7.87.0
) everything works. I can GET the https://conda.anaconda.org/conda-forge/osx-64/repodata.json file and I see in debug logs that cURL picks up the correct cert file using the CURL_CA_BUNDLE environment variable.
In micromamba, the --ssl-verify false
is not working in my case and my debug logs are as follows:
...
info libmamba No valid cache found
debug libmamba 'ssl_verify' not activated, skipping cURL SSL init
info libmamba Searching index cache file for repo 'https://conda.anaconda.org/conda-forge/noarch/repodata.json'
...
info libcurl * Connected to conda.anaconda.org (104.17.92.24) port 443 (#0)
info libcurl * WARNING: disabling hostname validation also disables SNI.
info libcurl * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
info libcurl * Server certificate: anaconda.org
info libcurl * Server certificate: here are our custom corp certs...
...
info libcurl * SSLRead() return error -9806
info libcurl * Closing connection 1
...
info libcurl * SSLRead() return error -9806
info libcurl * Closing connection 5
info libcurl * Closing connection 4
info libmamba Freeing pool.
critical libmamba Multiple errors occured:
Download error (56) Failure when receiving data from the peer [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
SSLRead() return error -9806
Subdir conda-forge/noarch not loaded!
Pointing to the same cert file using the ssl_verify
option in .mambarc
(as below) is not working as well and yields the same error as above.
channels:
- conda-forge
channel_priority: strict
ssl_verify: "/Users/UserName/certs/RootCA.pem"
Still, in debug logs I cannot see any message that libmamba/libcurl picks up my cert file.
For now, I need to just fallback to regular miniconda, but will be following this issue and waiting for a fix.
Thanks.
Can you try the curl version from conda-forge (install with Miniconda)? I wonder if they use different SSL backends
Hi @jonashaag, I installed curl from conda-forge in a fresh miniconda3 env:
% curl --version
curl 7.87.0 (x86_64-apple-darwin13.4.0) libcurl/7.87.0 (SecureTransport) OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.51.0
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe TLS-SRP UnixSockets
and curl https://conda.anaconda.org/conda-forge/osx-64/repodata.json
works without any issues as well as Miniconda3 just works out-of-the-box. When I commented out ENVS pointing to the custom cert in my .zshrc
it stoped, but it works fine with either SSL_CERT_FILE
or CURL_CA_BUNDLE
set.
In the same setting Micromamba fails because of the SSLRead() return error -9806
.
Thanks, and I hope this will help.
Thanks! It looks like we’re setting up something in curl in a wrong way here.
@jonashaag : Hi Jonas, I am not sure where you are getting with this one, but I had the same issue from a corporate system using Windows 11 and a Cisco firewall. For me a good workaround is to set the MAMBA_SSL_NO_REVOKE environment variable. I can either set it as a user environment variable or set it in the terminal whenever I want to use mamba. I need to have ssl_verify set to the path to the Certs (only needs to be done once, of course).
> conda config --set ssl_verify c:\Certificates\curl-ca-bundle.crt
> set MAMBA_SSL_NO_REVOKE=1
> mamba create -n testenv python=3.11 matplotlib
...
We're running into this error as well on different Windows machines behind corporate proxies. After trying the options mentioned here to no avail, I followed what @marc-wien said a while back:
The thread which culminates in https://github.com/jeroen/curl/issues/193#issuecomment-515662521 seems relevant...
And also saw the comment here:
Conda (and default curl in PowerShell) probably make use of this certificate and that is why connections work fine (presumably).
The discussions around this topic (https://github.com/curl/curl/discussions/11707) seem to be that when setting CURLOPT_CAINFO
it overwrites the default settings, but if you also set the bit for CURLSSLOPT_NATIVE_CA
in CURLOPT_SSL_OPTIONS
it adds the certificate provided to CURLOPT_CAINFO
to the list of certificates instead of overwriting (Docs for CURLSSLOPT_NATIVE_CA
).
As far as I can see in mamba's code, CURLSSLOPT_NATIVE_CA
is not set. Maybe that would help mitigating some of the issues?
We're running into this error as well on different Windows machines behind corporate proxies. After trying the options mentioned here to no avail, I followed what @marc-wien said a while back:
The thread which culminates in jeroen/curl#193 (comment) seems relevant...
And also saw the comment here:
Conda (and default curl in PowerShell) probably make use of this certificate and that is why connections work fine (presumably).
The discussions around this topic (curl/curl#11707) seem to be that when setting
CURLOPT_CAINFO
it overwrites the default settings, but if you also set the bit forCURLSSLOPT_NATIVE_CA
inCURLOPT_SSL_OPTIONS
it adds the certificate provided toCURLOPT_CAINFO
to the list of certificates instead of overwriting (Docs forCURLSSLOPT_NATIVE_CA
).As far as I can see in mamba's code,
CURLSSLOPT_NATIVE_CA
is not set. Maybe that would help mitigating some of the issues?
@jonashaag Hi Jonas, does @chaubold 's comment help?
I tried this solution locally but it didn't resolve the issue. For me setting CURLSSLOPT_NATIVE_CA
did not help 😞
I'm getting SSL errors in mamba when it tries to connect to remote repos. conda connects to them just fine with the same setup.
I have the following line in my .condarc file:
Here is the mamba error (showing "repoquery search" here, but "update" gets the same error message):
Here is the analogous conda call working:
As another data point, when I disconnect from my workplace network that the certificate is associated with and set
ssl_verify: false
, then the mamba command above works just fine. However this is not a complete solution for my needs.Here are my versions:
Thanks!