SSL certificate path does not propagate to builds during `nix-build`

[leon-thomm]

Describe the bug

Fetchers such as fetchzip rely on curl which by default will try to verify SSL certificates. When running behind an intercepting proxy, curl must either be run with --insecure, or it must be given a certificate bundle containing the self-signed certificate, e.g. through the CURL_CA_BUNDLE environment variable. It seems this information does not propagate to the build process through the nix daemon.

The documentation suggests the path to the SSL certificate bundle should propagate properly when setting NIX_SSL_CERT_FILE. I accordingly adjusted environment.variables. Indeed, systemctl show nix-daemon | grep Environment now shows correct entries for NIX_SSL_CERT_FILE and even CURL_CA_BUNDLE, but the build fails as described below.

Steps To Reproduce

1. Run behind an intercepting corporate proxy, and add self-signed certificates to security.pki.certificates
2. Run cntlm

     users.users.cntlm.group = "cntlm";
     users.users.cntlm.isSystemUser = true;
     users.groups.cntlm = {};
     services.cntlm = {
       enable = true;
       configText = <-- snip config and credentials -->;
     };
     networking.proxy.default = "http://127.0.0.1:3128/";
     networking.proxy.noProxy = "127.0.0.*,localhost,<-- snip internal sites -->";

3. Verify that your proxy works, and curl succeeds if and only if CURL_CA_BUNDLE is set properly

   > nix-shell --pure -p curl
   
   [nix-shell:~/projects/nix-pkg-test]$ curl -I https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz
   HTTP/1.1 200 Connection established
   Proxy-agent: netentsec
   Connection: close
   
   curl: (77) error setting certificate file: /no-cert-file.crt
   
   [nix-shell:~/projects/nix-pkg-test]$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt curl -I https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz
   HTTP/1.1 200 Connection established
   Proxy-agent: netentsec
   Connection: close
   
   HTTP/1.1 200 OK
   Date: Mon, 27 May 2024 11:59:24 GMT
   Server: Apache/2.4.29 (Trisquel_GNU/Linux)
   Strict-Transport-Security: max-age=63072000
   Last-Modified: Sun, 29 May 2022 23:05:08 GMT
   ETag: "fc451-5e02e9229e6eb"
   Accept-Ranges: bytes
   Content-Length: 1033297
   Content-Security-Policy: default-src 'self'; img-src 'self' https://static.fsf.org https://static.gnu.org https://gnu.org http://static.fsf.org http://static.gnu.org http://gnu.org http://gnu.org; object-src 'none'; frame-ancestors 'none'; child-src 'self' https://static.gnu.org http://static.fsf.org http://static.gnu.org http://gnu.org;
   X-Frame-Options: DENY
   X-Content-Type-Options: nosniff
   Content-Type: application/x-gzip

4. Build a basic package using fetchzip, fetchFromGitHub, or similar (no fetchTarball)

   let pkgs = import <nixpkgs> {}; in
   {
     myhello = pkgs.stdenv.mkDerivation {
       pname = "hello";
       version = "0.5";
       src = pkgs.fetchzip {
         url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz";
         sha256 = "";
       };
     };
   }

5. See error

   > nix-build -A myhello
   warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
   these 2 derivations will be built:
     /nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv
     /nix/store/mj82j0ljhn3ibs0lyqzy571igq881yia-hello-0.5.drv
   building '/nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv'...
   error checking the existence of https://tarballs.nixos.org/sha256/:
   curl: (60) SSL certificate problem: self-signed certificate in certificate chain
   More details here: https://curl.se/docs/sslcerts.html
   
   curl failed to verify the legitimacy of the server and therefore could not
   establish a secure connection to it. To learn more about this situation and
   how to fix it, please visit the web page mentioned above.
   
   trying https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
   curl: (60) SSL certificate problem: self-signed certificate in certificate chain
   More details here: https://curl.se/docs/sslcerts.html
   
   curl failed to verify the legitimacy of the server and therefore could not
   establish a secure connection to it. To learn more about this situation and
   how to fix it, please visit the web page mentioned above.
   error: cannot download source from any mirror
   error: builder for '/nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv' failed with exit code 1;
         last 10 log lines:
         >   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
         >                                  Dload  Upload   Total   Spent    Left  Speed
         >   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
         > curl: (60) SSL certificate problem: self-signed certificate in certificate chain
         > More details here: https://curl.se/docs/sslcerts.html
         >
         > curl failed to verify the legitimacy of the server and therefore could not
         > establish a secure connection to it. To learn more about this situation and
         > how to fix it, please visit the web page mentioned above.
         > error: cannot download source from any mirror
         For full logs, run 'nix log /nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv'.
   error: 1 dependencies of derivation '/nix/store/mj82j0ljhn3ibs0lyqzy571igq881yia-hello-0.5.drv' failed to build

Expected behavior

Succeeding build.

nix-env --version output

nix-env (Nix) 2.18.1

Additional context

The fact that the fetchers don't consider system certificates IMO is a bug in itself. They wrongly use ${cacert}/etc/ssl/certs/ca-bundle.crt which gives rise to this issue in the first place (explanation). Nevertheless, NIX_SSL_CERT_FILE should still override it.

Priorities

Add :+1: to issues you find important.

[cole-h]

I wonder if behavior changes with --option ssl-cert-file /path/to/bundle (it might not, since I think it takes effect in the same way NIX_SSL_CERT_FILE does, but worth a shot...)

[leon-thomm]

yeah same result, unfortunately

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-build-curl-issues-behind-proxy/45911/5

[raboof]

Fetchers such as fetchzip

since fetchzip is not a nix builtin but implemented in nixpkgs (based on fetchurl), this issue should perhaps be moved to the nixpkgs repo instead?

[leon-thomm]

this issue should perhaps be moved to the nixpkgs repo instead?

I thought it might also be a Nix (daemon?) issue that it kills NIX_SSL_CERT_FILE or something, because I experienced this with multiple fetchers, but I don't actually know. What do you think?

[raboof]

What do you think?

Hmm, indeed curl in nixpkgs does reference NIX_SSL_CERT_FILE in a comment - not sure how this is intended to fit together.

https://github.com/NixOS/nix/issues/3155 and https://discourse.nixos.org/t/nix-cannot-find-my-custom-ssl-certificate/27361/5 might also be relevant.

[leon-thomm]

yeah, I saw those issues and already applied the Environment fix, by adding NIX_SSL_CERT_FILE to systemd.services.nix-daemon.serviceConfig.Environment, and checking systemctl show nix-daemon | grep Environment, whose output correctly includes

CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt 
NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

[leon-thomm]

A workaround when using curl through fetchurl is to set NIX_CURL_FLAGS in the nix-daemon environment [1] [2]. On NixOS in configuration.nix you can do something like this:

  systemd.services.nix-daemon = {
    enable = true;
    serviceConfig = {
      Environment = [
        "NIX_CURL_FLAGS=\"--cacert /etc/ssl/certs/ca-certificates.crt\""
      ];
    };
  };

which seems to work in my setup. Still, I think NIX_SSL_CERT_FILE should be fixed and used instead.

edit: this workaround seems to work with nix-build but not with nixos-rebuild on NixOS which runs into the original error

[fricklerhandwerk]

Triaged in Nix maintainers meeting:

• This is a documentation issue, partly in the Nix manual, largely in the Nixpkgs manual.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-05-29-nix-team-meeting-minutes-148/46195/1

[leon-thomm]

Another hacky workaround is utilize the extra certificate arguments of the cacert package. This way, your certificates will also end up in ${cacert}/etc/ssl/certs/ca-bundle.crt. On NixOS, you can do

  nixpkgs.overlays = [
    (final: prev: {
      cacert = prev.cacert.overrideAttrs (old: {
        extraCertificateStrings=[your_certificates];
      });
    })
  ];

Since almost everything on your system transitively depends on cacert, nixos-rebuild will re-compile most of NixOS from scratch. I had to pass --max-jobs 1 in order to not run out of memory, and this took more than a day.