search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.51k stars 13.69k forks source link

SSL peer certificate or SSH remote key was not OK (60) #70939

Closed xbreak closed 4 years ago

xbreak commented 4 years ago

Describe the bug I get SSL peer certificate error from cache.nixos.org when running e.g nix-channel --update or nix-build. Update: The root user is unaffected. So e.g. this command works:

$ sudo HOME=/root NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt /nix/store/6chjfy4j6hjwj5f8zcbbdg02i21x1qsi-nix-2.3.1/bin/nix-channel --update nixpkgs

but not

$ NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt /nix/store/6chjfy4j6hjwj5f8zcbbdg02i21x1qsi-nix-2.3.1/bin/nix-channel --update nixpkgs

To Reproduce Steps to reproduce the behavior: Installed Nix 2.3.1 (multiuser) on a CentOS 7.4. (Linux localhost.localdomain 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux) and sandboxing enabled). I first ran into the issue when I was adding the nixos-19.09 channel as follows:

$ nix-channel --list
$ nix-channel --add https://nixos.org/channels/nixos-19.09-small nixos
$ nix-channel --update
unpacking channels...
warning: unable to download 'https://cache.nixos.org/rmqq4vmhf9kv4gc9g3x2rg488zyvgkqx.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 315 ms
...
error: unable to download 'https://cache.nixos.org/rmqq4vmhf9kv4gc9g3x2rg488zyvgkqx.narinfo': SSL peer certificate or SSH remote key was not OK (60)
error: program '/nix/store/6chjfy4j6hjwj5f8zcbbdg02i21x1qsi-nix-2.3.1/bin/nix-env' failed with exit code 1

Using nix-prefetch-url does not result in this error, which makes me think it might not the same as reported in #67540 or #67210 (or does nix-prefetch-url does something materially different?):

$ nix-prefetch-url https://cache.nixos.org/rmqq4vmhf9kv4gc9g3x2rg488zyvgkqx.narinfo
[0.0 MiB DL]
error: unable to download 'https://cache.nixos.org/rmqq4vmhf9kv4gc9g3x2rg488zyvgkqx.narinfo': HTTP error 404

Same error with nix-build -A curl using nixpkgs-19.09 release:

$ cd nixpkgs-19.09
$ nix-build -A curl
nix-build -A curl
warning: unable to download 'https://cache.nixos.org/4c4pi48ixfizi96arhh0dii91bg2ccb6.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 264 ms
warning: unable to download 'https://cache.nixos.org/4c4pi48ixfizi96arhh0dii91bg2ccb6.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 603 ms
warning: unable to download 'https://cache.nixos.org/4c4pi48ixfizi96arhh0dii91bg2ccb6.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 1274 ms
warning: unable to download 'https://cache.nixos.org/4c4pi48ixfizi96arhh0dii91bg2ccb6.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 2714 ms
error: unable to download 'https://cache.nixos.org/4c4pi48ixfizi96arhh0dii91bg2ccb6.narinfo': SSL peer certificate or SSH remote key was not OK (60)

Expected behavior No SSL error.

Additional context Since I can't build or download anything from Nix I can't compare e.g. output from openssl s_client -connect cache.nixos.org:443 vs the system provided openssl, which works and output included here for reference:

$ /usr/bin/openssl s_client -connect cache.nixos.org:443
CONNECTED(00000003)
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign Organization Validation CA - SHA256 - G2
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Fastly, Inc.", CN = e.sni.fastly.net
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=e.sni.fastly.net
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
 1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
<snip...>
SSL handshake has read 3133 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 8B567546BFCDBCF10AA7A535DF9D120CB8E6CF9E1C0D0A52780823034997FDCA
    Session-ID-ctx:
    Master-Key: 0CCA59BA683AC5083CFC41187CA241CF0D144A8A47F080DA59408E1BEFB6A01BC0B4BF8487AB1AF611C711535B484EF3
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 2a 16 b7 2c 4d 7a 22 5d-91 57 4d 22 8b 06 f9 13   *..,Mz"].WM"....
    0010 - 0f 85 ad d1 fa d9 5c 1d-2c c6 b8 2b 74 56 52 f6   ......\.,..+tVR.
    0020 - 52 55 f9 e5 08 eb 75 78-4c d8 03 be 72 76 d9 0d   RU....uxL...rv..
    0030 - 98 07 2c 63 80 51 0a 24-f1 10 41 4f 7e c3 70 05   ..,c.Q.$..AO~.p.
    0040 - 40 5c 3d d2 d5 7a c4 0e-87 0a f4 8e 9c b0 be 3f   @\=..z.........?
    0050 - ad 90 ce 63 6c 3e e2 bf-ed 36 bf ae b6 20 c2 74   ...cl>...6... .t
    0060 - c9 19 42 ec 5c d4 b4 2c-55 ec 81 50 f2 fd 94 1c   ..B.\..,U..P....
    0070 - 4e dc 35 1b c7 21 c0 49-40 f1 ba 30 e0 34 13 64   N.5..!.I@..0.4.d
    0080 - 1e 4d 96 d7 86 4f 9b 3a-4a f3 58 17 45 f0 b9 32   .M...O.:J.X.E..2
    0090 - 4e 52 69 26 c4 0b 7b 1e-fe 6e e5 94 82 6d ab 64   NRi&..{..n...m.d

    Start Time: 1570745404
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

I get the same result when disabling ipv6 with 

$ echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

I'm inside a corporate network at the moment which may have some side effects. Although since I can openssl s_client and curl that url using the CentOS or Anaconda curl without problems it doesn't seem likely to be the network playing tricks:

$ /usr/bin/curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets
$ /usr/bin/curl https://cache.nixos.org/rmqq4vmhf9kv4gc9g3x2rg488zyvgkqx.narinfo
404
$ /opt/anaconda3/bin/curl --version
curl 7.61.0 (x86_64-conda_cos6-linux-gnu) libcurl/7.61.0 OpenSSL/1.0.2s zlib/1.2.11
Release-Date: 2018-07-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

$ /opt/anaconda3/bin/curl https://cache.nixos.org/rmqq4vmhf9kv4gc9g3x2rg488zyvgkqx.narinfo
404

The result of the https://cache.nixos.org/ diagnostics script: http://ix.io/1YgA (executed with system tools as trying to run with Nix will result in the same SSL error).

Metadata Please run nix run nixpkgs.nix-info -c nix-info -m and paste the result.

nix run nixpkgs.nix-info -c nix-info -m
warning: unable to download 'https://cache.nixos.org/lkjpbd9k889yqp1zr0i8m05v9mpjgx89.narinfo': SSL peer certificate or SSH remote key was not OK (60);

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
veprbl commented 4 years ago

You can find some useful discussion in https://discourse.nixos.org/t/ssl-peer-certificate-or-ssh-remote-key-was-not-ok-error-on-fresh-nix-install-on-macos/3582/14

xbreak commented 4 years ago

Thanks for that link. The initial problem in that thread was solved with this:

My initial guess is that NIX_SSL_CERT_FILE isn’t set properly in your shell. Did you make sure to create a new shell that sources the Nix profile setup, instead of just manipulating your PATH to include nix?

Which didn't solve things for me.

I started a new login shell after installing nix and my environment seems ok:

$ echo $NIX_SSL_CERT_FILE
/etc/ssl/certs/ca-bundle.crt
$ SSL_CERT_DIR="" SSL_CERT_FILE="" /usr/bin/openssl s_client -CAfile $NIX_SSL_CERT_FILE -connect cache.nixos.org:443
...
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 42DC7AFC5AED14C286E7E3098E43758DB5C9D422D7EAF7A84258C174FADE5E03
    Session-ID-ctx:
    ...
    Start Time: 1570787548
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Without -CAfile the verification fails as expected.

xbreak commented 4 years ago

I've added the following additional detail to the main ticket: root user is unaffected for some reason, so the following works:

$ sudo HOME=/root NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt /nix/store/6chjfy4j6hjwj5f8zcbbdg02i21x1qsi-nix-2.3.1/bin/nix-channel --update nixpkgs

but e.g. the following executed as non root (but with Nix profile and otherwise OK):

$ NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt /nix/store/6chjfy4j6hjwj5f8zcbbdg02i21x1qsi-nix-2.3.1/bin/nix-env -i curl
warning: there are multiple derivations named 'curl-7.65.3'; using the first one
installing 'curl-7.65.3'
these paths will be fetched (49.90 MiB download, 192.95 MiB unpacked):
  /nix/store/03l6hvws0lfnacvmsvnrwq44xzr5f2gi-nghttp2-1.39.2-dev
  /nix/store/11ddf9h4gvxka1xsjcmzaiykr7lbbi89-libssh2-1.9.0-dev
  /nix/store/121pfq5xb2vv1k62hc9qasp9682hj13a-keyutils-1.6-lib
  /nix/store/1qf7hlv38gqzzspgcldxqjqyj29fvw2g-libkrb5-1.17-dev
  /nix/store/22h3f311fjymkvp683kb657jycs7i5pn-glibc-2.27-bin
  /nix/store/24l5rfm2xhqkx96jvq0rg8pplb89b9hv-gcc-8.3.0
  /nix/store/282634kzyw5irqaixs1zbrpbi0nwha14-libev-4.27
  /nix/store/3hdlcd12cc9c95ym9cic4237kgp5sh1n-libssh2-1.9.0
  /nix/store/4c4pi48ixfizi96arhh0dii91bg2ccb6-curl-7.65.3-bin
  /nix/store/53g2nqsz0p8gm6wvp8lvlpavrx44jmdg-curl-7.65.3
  /nix/store/5q5rc4jy6ahwlqy82237dmx5shf6l4q8-curl-7.65.3-dev
  /nix/store/95b8n40rwpw1wynfm1gl2p3s0gr4l4r9-openssl-1.1.1d-dev
  /nix/store/cvzxhlj94pfqi34gx9h0j9c91s0jxw50-nghttp2-1.39.2-lib
  /nix/store/d97kkhkxwj9lxjl4ba81jlzplyzlzjb8-gcc-8.3.0-lib
  /nix/store/f8zs7mknva4rdx7zxr6j54y0igh3pras-zlib-1.2.11
  /nix/store/fbgcs45cy46z8r8bf9vcv4czayxx01zr-nghttp2-1.39.2
  /nix/store/hnzhghlb1k8r1cibhircbba15q3c1j22-libkrb5-1.17
  /nix/store/ily14d68xl11cnbbkf9svwnzwsrrnzah-bash-4.4-p23
  /nix/store/jhnkyazfsqkmdpnbxa3him212bxg6v4x-nghttp2-1.39.2-bin
  /nix/store/kksyrix1bpklvgkmvngcv0q9nh8hn2fl-glibc-2.27
  /nix/store/lpdrw0mrfq8lmhqvinfrcryyh74x52r6-c-ares-1.15.0
  /nix/store/lrnpy7nv1rrigc4d62fiwr4wkryimzxr-linux-headers-4.19.16
  /nix/store/qwm593x9sxd48vz09s523qi4zywfhsks-glibc-2.27-dev
  /nix/store/rv0gdqsiyc9fx5byi77dfya1sh1wn961-openssl-1.1.1d
  /nix/store/y8azj9zp91v7mg3mldncxm90yjx3f81l-zlib-1.2.11-dev
  /nix/store/yfvzwavadc56lrjkb78gxln2k1ya4pj8-openssl-1.1.1d-bin
  /nix/store/zx8fiy6iqbjfy314ip0k49ja2i6fjqf0-curl-7.65.3-debug
copying path '/nix/store/kksyrix1bpklvgkmvngcv0q9nh8hn2fl-glibc-2.27' from 'https://cache.nixos.org'...
warning: unable to download 'https://cache.nixos.org/nar/0c22kgkv46kswl7ndqybcr22gb1ykbhidcj591qkf1740la8f0x1.nar.xz': SSL peer certificate or SSH remote key was not OK (60); retrying in 311 ms
...
veprbl commented 4 years ago

I think I had a similar problem once. I resoled it by pointing $NIX_SSL_CERT_FILE to the certificates from cacert instead of the system's file. But since it works for you when you are using root, you likely have a different problem.

Did you try to repeat the test you did in https://github.com/NixOS/nixpkgs/issues/70939#issuecomment-541015879 with openssl binary provided by nixpkgs?

xbreak commented 4 years ago

I ran openssl from nixpkgs now and it also works as expected:

$ echo $NIX_SSL_CERT_FILE
/etc/ssl/certs/ca-bundle.crt
$ SSL_CERT_DIR="" SSL_CERT_FILE="" /nix/store/k27q3yjvqn56axwycvyzivl49qhfj15v-openssl-1.1.1d-bin/bin/openssl s_client -CAfile $NIX_SSL_CERT_FILE -connect cache.nixos.org:443
CONNECTED(00000003)
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign CloudSSL CA - SHA256 - G3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Fastly, Inc.", CN = v2.shared.global.fastly.net
verify return:1
...
    Verify return code: 0 (ok)
    Extended master secret: yes
---

Using empty $NIX_SSL_CERT_FILE fails as expected:

$ NIX_SSL_CERT_FILE="" /nix/store/k27q3yjvqn56axwycvyzivl49qhfj15v-openssl-1.1.1d-bin/bin/openssl s_client -connect cache.nixos.org:443
...
    Verify return code: 20 (unable to get local issuer certificate)
xbreak commented 4 years ago

Another note:

xbreak commented 4 years ago

If I modify the nix-daemon.service systemd unit to include the following (systemctl edit nix-daemon.service), then it works!

[Service]
Environment="NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt"

But I don't understand why e,g, nix-env would work before for root and not my normal user? Apparently root doesn't connect to the daemon so the NIX_SSL_CERT_FILE is available to read.

There seems to be a mechanism to propagate the invoking user environment to the daemon and builders? Is that what is not working? Or is the nix-daemon.service unit missing the necessary environment variables?

My mistake. No environment propagation takes place, which points to the fact that the installed nix-daemon.service is incomplete/incorrect. I'll close this and open a ticket in Nix instead. Update: See https://github.com/NixOS/nix/issues/3155.

Edit: Looks like NixOS nix-daemon unit file adds CURL_CA_BUNDLE to the environment. I tried the same and it didn't work on my system: https://github.com/NixOS/nixpkgs/blob/e8bc181154e310d81fc5f1cf11356b50bcffd303/nixos/modules/services/misc/nix-daemon.nix#L413-L415

expipiplus1 commented 4 years ago

I bumped into this on nixos with curl 7.65.3, updating to 7.68 seems to have fixed things for me.

steve-chavez commented 2 years ago

I got this issue when doing nix-channel --update nixos-unstable. Upgrading nix 2.2.2 to nix 2.3.10 solved it for me.

catern commented 2 years ago

I also encountered this on Nix 2.2.2 and an upgrade to Nix 2.3.15 solved it.

zot commented 2 years ago

I'm getting this error with nix 2.7.0. The above fix https://github.com/NixOS/nixpkgs/issues/70939#issuecomment-541267840 worked for me (regression, maybe?):

If I modify the nix-daemon.service systemd unit to include the following (systemctl edit nix-daemon.service), then it works!

[Service]
Environment="NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt"
NoraCodes commented 2 years ago

FYI, I just ran into this on Fedora as well. The same fix (modifying the nix-daemon unit) worked for me.

pachiras commented 2 years ago

I just installed nix multiuser mode today.

$ nix --version
nix (Nix) 2.8.0

And encountered this problem.

@zot's solution solved the problem completely. Thanks!

NoraCodes commented 2 years ago

@xbreak given that people are encountering this again, perhaps we could reopen this bug, or open a new one as a regression ticket?

xbreak commented 2 years ago

@NoraCodes: I originally closed this after opening a corresponding ticket for nix when the problem was clarified: https://github.com/NixOS/nix/issues/3155. That one is still open though.

xbreak commented 1 year ago

I closed this one after opening a corresponding ticket in NixOS/nix: https://github.com/NixOS/nix/issues/3155, which is still open.

On Mon, May 2, 2022, 18:50 Leonora Tindall @.***> wrote:

@xbreak https://github.com/xbreak given that people are encountering this again, perhaps we could reopen this bug, or open a new one as a regression ticket?

— Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/70939#issuecomment-1115115007, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADG5H6ENQ5QUHQ6RYIJHWALVIABUTANCNFSM4I7SZ4CA . You are receiving this because you were mentioned.Message ID: @.***>

duncanam commented 9 months ago

I'm getting this error with nix 2.7.0. The above fix https://github.com/NixOS/nixpkgs/issues/70939#issuecomment-541267840 worked for me (regression, maybe?):

If I modify the nix-daemon.service systemd unit to include the following (systemctl edit nix-daemon.service), then it works!

[Service]
Environment="NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt"

Even after updating my NIX_SSL_CERT_FILE was still getting errors, the daemon fix above resolved it for me. One note, I also did a systemctl restart of the nix-daemon.service

nixos-discourse commented 1 week ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-in-corporate/50041/9