SSL certificates: figure out how libs/apps find them by default

[vcunat]

For some options already discussed see https://github.com/NixOS/nixpkgs/pull/8121#issuecomment-108033047.

[layus]

I found an excellent email exchange on the openssl mailing list: https://mta.openssl.org/pipermail/openssl-users/2015-December/002553.html. Also, a most enlightening blog post at https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/. And finally, a guix discussion about gnuTLS: https://lists.gnu.org/archive/html/guix-devel/2014-02/msg00245.html

Basically, we need

1. To define at NixOS level a "default trusted location" (currently /etc/ssl/certs/ca-certificates.crt) that can be hardcoded in every package requiring a trust store;
2. Ensure that all applications default to that location (Currently not the case for openssl, and possibly others. Openssl imparct Python for example.);
3. Allow users to specify a different trust store. (Currently using SSL_CERT_FILE, possibly other application-specific mechanisms.)

At the moment, openssl is broken without SSL_CERT_FILE because it has no default trust store. (Technically it is false, openssl trust store defaults to /nix/store/-openssl-version/etc/ssl/cert.pem, which does not exist :smiling_imp:) We therefore need to

• (option 1) Patch openssl to use /etc/ssl/certs/ca-certificates.crt as its default trust store;
• (option 2) Define SSL_CERT_FILE everywhere (including in systemd.globalEnvironment), to ensure that openssl finds its default store.
• (option 3) Make the ca-cert package a depedency of all the packages using it ?

Option 2 is straightforward, but we can do better than that! Option 3 does not seem wise as we would trigger a mass rebuild for nothing. I would advise to go for option 1 as a sane default. With option 1, we need to fix openssl trust store, and get rid of the SSL_CERT_FILE infection (already 30 occurrences, including the valid ones.) Regarding gnuTLS, we should add the option --with-default-trust-store-file (see #8121) as explained in the above mailing lists. GnuTLS provides no way to globally override this setting, but this is their design decision. Applications can always follow SSL_CERT_FILE if they want to.

So we need to ship openssl with sane defaults. The main issue is that we want --openssldir to be /etc/ssl, but this is not possible because openssl also tries to install files there at build/install time. However, we can use the following trick (instead of reverse-engineering and patching the build process).

--- a/pkgs/development/libraries/openssl/default.nix
+++ b/pkgs/development/libraries/openssl/default.nix
@@ ... @@ stdenv.mkDerivation rec {
   configureFlags = [
     "shared"
     "--libdir=lib"
-    "--openssldir=etc/ssl"
+    "--openssldir=etc/ssl/openssl"
   ] ++ stdenv.lib.optionals withCryptodev [
     "-DHAVE_CRYPTODEV"
     "-DUSE_CRYPTODEV_DIGESTS"
   ];
@@ ... @@ stdenv.mkDerivation rec {
   postInstall = ''
     # If we're building dynamic libraries, then don't install static
     # libraries.
     if [ -n "$(echo $out/lib/*.so $out/lib/*.dylib $out/lib/*.dll)" ]; then
         rm "$out/lib/"*.a
     fi

     # remove dependency on Perl at runtime
     rm -r $out/etc/ssl/misc $out/bin/c_rehash
+
+    # Setup default ssl files location
+    # The openssl subdir is a trick to avoid infinite loops when building the environment without cacert.
+    mv $out/etc/ssl/openssl/* $out/etc/ssl
+    rmdir $out/etc/ssl/openssl
+    ln -s /etc/ssl/ $out/etc/ssl/openssl
+    ln -s /etc/ssl/certs/ca-certificates.crt $out/etc/ssl/cert.pem
   '';

@vcunat, Anything against this patch ? If you agree, I will submit a pull-request.

[layus]

This is a small memo to understand how apps work with ssl files. Obviously, the memo got bigger than expected :-)

Nix and NixOS are missing strong guidelines on how to find and manage the default trust store. That store is the place where all the trusted root certificates are stored.

Due to this confusion, several issues were raised on this mailing list and on GitHub.

These issues concern

• NixOS: Programs working well in the shell, but failing within systemd units (services).
• Nix, without NixOS: Inconsistent certificate validity between applications in the nix store and "native" applications.

These issues arises because there is no consensus on where to find the default, system-wide trusted certificates. This confusion affects all the distros likewise, and is not specific to Nix.

There are four ways that applications can use to locate the default trust database:

1. Trust the default location provided by their security library. OpenSSL has --openssldir and $SSL_CERT_FILE, GnuTLS has --with-default-trust-store-file, etc.

   For example, ArchLinux compiles OpenSSL with --openssldir=/etc/ssl, and provides "/etc/ssl/cert.pem" in the package ca-certificates-utils required by its base environment. Most linux distros will pick a trust store location and configure their packages to default to it. This default location is however not the same for everyone.

2. By looking at a set of common locations, like /etc/ssl/certs/ca-certificates.crt, /etc/ssl/certs/ca-bundle.crt, /etc/pki/tls/certs/ca-bundle.crt, etc. These paths are also provisioned by NixOS in the hope to catch programs looking for them.
3. Ask for and/or receive an explicit path, either using the environment or command line arguments. (OpenSSL honors -CAfile, curl has --cacert, etc.)
4. Ship its own trust store. (This is the case with Python's requests module and with Firefox for example.)

Most distros try to configure and patch the programs that use certificates to use their main trust store. Nix however cannot provide packages with an hardcoded default store because nix packages are used on many different distros and even different systems. On these systems, nix-profile.sh (installed in "~/.nix-profile/etc/profile.d/nix.sh") tries to find the location of the system-wide trust store (using method 2. above), and configures SSL_CERT_FILE accordingly.

# Excerpt from nix-profile.sh.in :
# Set $SSL_CERT_FILE so that Nixpkgs applications like curl work.
if [ -e /etc/ssl/certs/ca-certificates.crt ]; then # NixOS, Ubuntu, Debian, Gentoo, Arch
    export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
elif [ -e /etc/ssl/certs/ca-bundle.crt ]; then # Old NixOS 
    export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt 
elif [ -e /etc/pki/tls/certs/ca-bundle.crt ]; then # Fedora, CentOS 
    export SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt 
elif [ -e "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" ]; then # fall back to cacert in Nix profile
    export SSL_CERT_FILE="$NIX_LINK/etc/ssl/certs/ca-bundle.crt"
elif [ -e "$NIX_LINK/etc/ca-bundle.crt" ]; then # old cacert in Nix profile 
    export SSL_CERT_FILE="$NIX_LINK/etc/ca-bundle.crt"
fi

This is extremely important for Nix itself to work properly as derivations and nix archives are downloaded using HTTPS. By setting this environment variable which overrides the default location used by OpenSSL, Nix ensures that programs like cURL will be able to download archives securely. We need standardization to avoid inconsistencies between applications (why does curl fail on https://XXX.YY's certificate while firefox works flawlessy?) and to provide a single location to update in case of issues with the root CA's.

As this variable only affects OpenSSL, the other security libraries and applications are left guessing for the right path to use and often fail. They are unaware of SSL_CERT_FILE, and nix cannot compile them with a default trust store because there is no global sane value. Therefore, nixpkgs sometimes merges patches that add support for SSL_CERT_FILE in crypto libraries.

On NixOS, the situation is completely reversed. We have full control over the environment of the packages, and we know where the certificates are located. We could even compile the packages to depend on cacert (the derivation providing the default trusted certificates).

However, making derivations depend on cacert is not a good idea because

1. NixOS is terrible at managing security updates. If one certificate needs to be revoked, then all the packages would require a rebuild.
2. Users would need to override the cacert package to provide custom trusted certificates, and hit the same mass rebuild issue.

For these reasons NixOS uses the path "/etc/ssl/certs/ca-certificates.crt" as the system's default trust store. This path (as well as other common paths) is provisioned by the security/ca.nix module using files from the cacert derivation. To ensure that packages find and use this path, NixOS exports SSL_CERT_FILE in the global environment.

However, this global environment is not available to systemd units, which is the reason why so many users hit the mailing list with the issue of a command working perfectly well in their shell and failing when started in a unit. This situation is clearly a failure of NixOS and the Nix model where a derivation should "always work the same". The easy fix on NixOS would be to add SSL_CERT_FILE to systemd.globalEnvironment, and have that variable added to all the systemd units. The biggest downside is that changing (and adding) this variable will require a restart of all the units, even the ones not requiring that environment variable.

Now, on NixOS we do not need the flexibility to change the trust store, because we "know" that it will be "/etc/ssl/certs/ca-certificates.crt".

My proposal is therefore to make all the crypto libraries look for a default trust database in $SSL_CERT_FILE when defined, and in "/etc/ssl/certs/ca-certificates.crt" when not . This way, NixOS would work flawlessly without defining SSL_CERT_FILE, because all the derivations would default on the right path. And all the other Nix users would have the best experience by just setting SSL_CERT_FILE to the right location, or relying on "~/.nix-profile/etc/profile.d/nix.sh" to do it for them.

This proposal requires subtle patches to nixpkgs. We need all the packages that provide a default store to default to "~/.nix-profile/etc/profile.d/nix.sh". We should not however patch the packages that get their default store from another lib.

As an example, git needs no patching, because it relies on the default trust store provided by libcurl. cURL however requires fixing, because it does not rely on the default trust store provided by openssl or its other backends.

The current lwp-protocol-https-cert-file.patch applied on the corresponding perl module fixes a non-issue:

Use $SSL_CERT_FILE to get the CA certificates.

diff -ru -x '*~' LWP-Protocol-https-6.02-orig/lib/LWP/Protocol/https.pm LWP-Protocol-https-6.02/lib/LWP/Protocol/https.pm
--- LWP-Protocol-https-6.02-orig/lib/LWP/Protocol/https.pm  2011-03-27 13:54:01.000000000 +0200
+++ LWP-Protocol-https-6.02/lib/LWP/Protocol/https.pm   2011-10-07 13:23:41.398628375 +0200
@@ -21,6 +21,11 @@
     }
     if ($ssl_opts{SSL_verify_mode}) {
    unless (exists $ssl_opts{SSL_ca_file} || exists $ssl_opts{SSL_ca_path}) {
+            if (defined $ENV{'SSL_CERT_FILE'}) {
+                $ssl_opts{SSL_ca_file} = $ENV{'SSL_CERT_FILE'};
+            }
+        }
+   unless (exists $ssl_opts{SSL_ca_file} || exists $ssl_opts{SSL_ca_path}) {
        eval {
        require Mozilla::CA;
        };

In this case, the Perl module does not rely on any default trust store, and requires the Mozilla::CA module when none is provided explicitly. This patch changes the behavior of the Perl module. Any user of that module should provide a trust store explicitly or also depend on Mozilla::CA.

Finally, we may want to patch derivations providing their own trust store to use the system one, or not. How can we ensure that the packages honor the default trust store if Firefox, Python's request module or Perl's Mozilla::CA bundle their own trusted certificates ? Is this a security issue ?

[layus]

@edolstra, the previous message explains why we should not "kill" SSL_CERT_FILE, and why we should therefore revert the pull request #12748.

[edolstra]

Contrary to what the name of the PR implies, it does not kill SSL_CERT_FILE, just provide a default value. The only problem is that are a few programs (e.g. git, but apparently not curl) that don't respect SSL_CERT_FILE out of the box, so we may want to revert those parts of the PR.

[layus]

My issue with the current merged patch is that we removed the support of SSL_CERT_FILE from cURL and provided a default location for the trust store. This means that the cURL on staging will not respect SSL_CERT_FILE and will only look into the provided default store ("/etc/ssl/certs/ca-certificates.crt"), as opposed to the old behaviour which respected SSL_CERT_FILE and looked in a set of known locations if SSL_CERT_FILE was not defined. I was too eager to remove SSL_CERT_FILE on NixOS, and did not think about Nix users on other systems/OS.

I made more moderate patches on my own repo at https://github.com/layus/nixpkgs/tree/fix-openssl-defaults. These patches include some of your own modifications, and respect the proposition made above to define the default location to NixOS's default and to respect SSL_CERT_FILE.

[zimbatm]

What if the default trust store was /nix/etc/ssl/cert.pem (or something under /nix) ? That's a path that we control on all systems. During install it could then be symlinked to the target OS's trust store. Before install it could also be pre-seeded with enough certs for nix to fetch stuff from the hydra cache.

On OSX the situation would be a bit more complicated because they store their certs in another format called the Keychain and it's all over the place. Homebrew's install of openssl solves the issue by generating their own cert.pem. I suppose that file needs to be kept in sync with the Keychain but again it could be put under /nix/etc/ssl/cert.pem.

The issue with relying on the SSL_CERT_FILE is that it might not be available when running sudo for example. On OSX, Homebrew is not able to fetch bottles (the pre-built binaries) when SSL_CERT_FILE is set. (but it works fine for source code for some reason). Homebrew uses /bin/curl to fetch. My guess is that's because Apple replaced curl's SSL with their own engine and forgot to remove the support of SSL_CERT_FILE.

[layus]

Looks like a great idea to me. You simply set the right symlink at install time. And it works equally well for Nix and NixOS. Perfect !

Does Windows support this ?

[zimbatm]

I haven't looked at Windows in details because it's not an official Nix target. Windows itself has it's own CA store like OSX and we would need an export script to generate the PEM file. It's possible that Cygwin and friends already deal with the issue. The day where we want to support Window I expect the target to be Windows+Cygwin or Windows+MinGW, ...

[vcunat]

What if the default trust store was /nix/etc/ssl/cert.pem (or something under /nix) ?

I'm not certain if someone actually wants to control the trust store globally for all nix stuff but separately from the OS. It seems enough to me to use the OS trust store as the default and allow overriding it by an env var.

BTW, rather than /nix/etc/ssl/ I would imagine /nix/var/nix/profiles/default/etc/ssl/.

[layus]

Using /nix/var/nix/profiles/default/etc/ssl/ might work if nix-env detects the location of the trust store. However, this is not as flexible as /nix/etc/ssl/ which could be changed independently of the current environment. Admitedly, there is no reason to change this once the /nix tree is installed.

[cleverca22]

also of note, setting SSL_CERT_FILE globally or per-service doesn't work in the case of php-fpm, it scrubs the env on startup by default, so it has to be entered into the php.ini file, or compiled in as a default

[nrolland]

@layus thank you it's really helpful to have a birdeye view to make sense of it..

[wmertens]

The real solution is to use the platform trust stores, but not all upstream projects support all platforms.

The second best solution is to periodically run something like https://github.com/raggi/openssl-osx-ca/blob/master/bin/openssl-osx-ca which dumps the trust store into a properly formatted file.

I am a big fan of storing global state in /nix/etc because, indeed, that is the one location under Nix control on all platforms. For backwards compatibility with existing installations, that can be a symlink to /etc.

So I propose that from now on all Nix-related global state is stored under /nix/etc, and that the Nix install script makes either a symlink to the trust store or runs a dump script for the platform and recommends running it periodically.

[magnetophon]

Any new on this? I just got bitten by it too.

There's a couple of related issues, with various workarounds. What is the best solution for now?

[layus]

Could you describe your specific issue, possibly in a new thread ? Link it to this thread, and tag me if needed. The current state should be OK for most use cases.

[4e6]

I want to bring up an issue when the user has custom root certificates installed. My setup is NixOS with extra certificates defined by security.pki.certificates config option.

Issue

There are two certificate bundles that are currently used in Nixpkgs:

• ${cacert}/etc/ssl/certs/ca-bundle.crt
• /etc/ssl/certs/ca-certificates.crt

Find usages:

[~/projects/nixos/nixpkgs]$ nix-shell -p ag --run 'ag /etc/ssl/certs/ca-bundle.crt pkgs'
[~/projects/nixos/nixpkgs]$ nix-shell -p ag --run 'ag /etc/ssl/certs/ca-certificates.crt pkgs'

The problem with the first, ${cacert}/etc/ssl/certs/ca-bundle.crt is that it doesn't contain extra certificates defined by NixOS config option security.pki.certificates. Currently they are assembled under the /etc/ssl/certs/ca-certificates.crt bundle.

The second approach, constant /etc/ssl/certs/ca-certificates.crt works perfectly with NixOS, but could have issues with Nix systems that use a different file for bundled certificates.

I experience problems with packages that use cacert bundle, ${cacert}/etc/ssl/certs/ca-bundle.crt, jdk and cargo in particular. I need to add postInstall step to openjdk8 package to reinstall correct certificates from /etc/ssl/certs/ca-certificates.crt. Wrapped cargo executable causes #8872 issue because of wrong certs file, but works fine unwrapped.

Proposed solution with managed bundle under the /nix directory should fix the issue.

[layus]

@4e6 You are right that ${cacert}/etc/ssl/certs/ca-bundle.crt should not be used. It forces a rebuild of the package everytime a certificate changes. That being said, /etc/ssl/certs/ca-certificates.crt should not be used directly, only as a fallback if NIX_SSL_CERT_FILE is not set. In all cases, NIX_SSL_CERT_FILE should be observed by nix packages.

This is the new standard as per https://github.com/NixOS/nix/commit/fb2dd3210072a03526e881cd2547cf4c2df4ba52. A migration has started with the new commit https://github.com/NixOS/nixpkgs/commit/942dbf89c6120cb5b52fb2ab456855d1fbf2994e Also, see https://github.com/NixOS/nix/issues/921.

@edolstra We should probably make an announcement on the mailing list about how to manage certificates. Using NIX_SSL_CERT_FILE is quite a big change and may break many configurations.

[edolstra]

Why would it break anything?

[layus]

@edolstra If someone relies on setting SSL_CERT_PATH, and this gets overridden by NIX_SSL_CERT_PATH. And its just one example. This is a sensitive area, it costs nothing to warn people.

Plus we need packagers to understand how certificates are managed. We want as many packages as possible to follow the rule NIX_SSL_CERT_PATH > SSL_CERT_PATH > /etc/ssl/certs/ca-certificates.crt and as little as possible using ${cacert}/etc/ssl/certs/ca-bundle.crt directly.

[edolstra]

I don't understand. Given that presumably nobody has NIX_SSL_CERT_PATH set today, what breaks by having packages use that environment variable if it's defined?

[vcunat]

Won't nix's new profile script set it for everyone? (And thus override any $SSL_CERT_PATH setting?)

[vcunat]

Self-answer: it did override the variable directly until that commit; now it will only override it for nix-built stuff. (EDIT: although interactions for some nixpkgs+nix version combinations probably won't be nice.)

[layus]

I may have missed something here, but I am pretty sure I got it right.

1. Every package should default to /etc/ssl/certs/ca-certificates.crt. This ensures correct behavior on NixOS.
2. Packages should look at NIX_SSL_CERT_FILE, because it is the only way to tell them where to find certificates on hosts where nix is installed as a user. We used to set SSL_CERT_FILE, but it confuses MacOS.

This is the reason why we will set NIX_SSL_CERT_FILE for every user-install of nix.

Now, if for some reason a user of nix on MacOS sets SSL_CERT_FILE to a location with custom certificates, encoded in the old way such that it does not confuse MacOS, then that user has a perfectly valid installation. By setting NIX_SSL_CERT_FILE and making packages use that instead of SSL_CERT_FILE when defined, then the above user will see nix-provided openssl look in the wrong location, because we have just overwritten his cache location[1].

My point is that certificate and trust management are crittical. Users should be warned and our decisions should be motivated and expained.

I will write such an email, and ensure it is reviewed before sending it on the mailing list.

---

[1] One solution might be to use ${SSL_CERT_FILE:-${NIX_SSL_CERT_FILE:-/etc/ssl/certs/ca-certificates.crt} instead of ${NIX_SSL_CERT_FILE:-${SSL_CERT_FILE:-/etc/ssl/certs/ca-certificates.crt} but this is not my point.

[edolstra]

I reverted 942dbf8 because it was intended to go staging instead of master. But, we can also use NIX_SSL_CERT_FILE for Nix itself only (via https://github.com/NixOS/nix/commit/fb2dd3210072a03526e881cd2547cf4c2df4ba52). That prevents the issue @layus describes (except for Nix itself), but also prevents packages in Nixpkgs from having a working certificate bundle on systems that lack /etc/ssl/certs/ca-certificates.crt.

[layus]

I think the NIX_SSL_CERT_FILE is the best way to go. The use case described above was just to outline the importance of communication, not to say that the idea is not good.

Using NIX_SSL_CERT_FILE globally on user isntalls fixes all the issues, but may conflict with former config, fixes and cruft. By explaining how it should work, people will adapt their config, and voilà. I am conviced that it is impossible to make such a change transparently to the users. There will always be someone with a strange config. Just make the best decision for the future and advertise it. Archlinux has a lot of breaking changes advertised on their main page for example.

[kirelagin]

Could someone, who knows what is going on with this stuff, write down some kind of documentation? I’m having really hard time figuring out how locating trusted roots works these days.

Am I right that the current plan (mostly implemented?) is to let all the packages locate system defaults the way they want during runtime (while forcing them to use $NIX_SSL_CERT_FILE if it is defined) and to use the bundle from pkgs.cacert to verify sources during build? I can see the last part applied in lots of rust and go stuff, as well as in fetchbower, fetchdarcs, fetchgit and (I’m not sure if it is done the right way there) fetchgx, but not in fetchurl or other fetch* functions.

As far as I can tell this has been implemented for openssl, but is there a plan to force other ssl libraries (e.g. gnutls) to respect $NIX_SSL_CERT_FILE? I am only asking because fetching gpg keys currently fails on darwin exactly because gnutls does not seem to be capable of locating any certificates.

[layus]

@kirelagin Yes, I am willing to write such documentation. But I have no idea where it belongs. Let's start here:

About certificates management, we want to configure globally the list of trusted certificates. We call this set of certificates the CA bundle or the CA store. As certificates should be allowed to change without recompiling the whole nixpkgs, they are not hardcoded as a dependency of the packages, but resolved dynamically during execution.

As we cannot recompile the nix package set for every distro around here, we want a fixed procedure working on NixOS and on other linux distros. The procedure can vary across architectures, as a different package set will be built anyway on a different arch. For example, we can have different defaults on darwin.

Also, the idea is to avoid altering the upstream intended behavior. In particular, we should not override command line options with our NIX_SSL_CERT_FILE variable. To keep things simple, we prefer to have only the lower ssl libraries managing default certificates, and let other program get their default CA store from these libraries. However, it is not easy because historically openssl provided no default CA store, and all the applications have taken the habit of trying of trying a list of usual store locations until they find one. For these, we can patch them to use the defaults from the underlying library, or tho patch them to use nix default bundle finding algorithm.

In nix, the correct way to look for the default CA store, is to look at $NIX_SSL_CERT_FILE, if defined, and then default to /etc/ssl/certs/ca-certificates.crt. The underlying idea is that packages outside of NixOS should always be executed within the environment defined by nix-profile.sh, which "Set[s] $NIX_SSL_CERT_FILE so that Nixpkgs applications like curl work". On NixOS, the default makes sense.

By derogation to that rule, on darwin it is preferred to default to /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt (see openssl package) because the other default makes no sense yet, and there is no penalty for specifying an alternate default. I personally consider the above trick as a pure hack, but it just works for now (see https://github.com/NixOS/nixpkgs/pull/23723 and https://github.com/NixOS/nixpkgs/pull/24203). If NixOS is ever ported to darwin, then this default will have to change back to /etc/ssl/certs/ca-certificates.crt.

---

TL;DR (So, what should we do for our packages ?)

1. Packages should be configured or patched to ask the default store location to the underlying ssl library, or just let the library pick it without interfering. Lists of hardcoded locations may be removed.
2. When it is not possible or sufficient, the package should duplicate the above procedure to find the correct default store location (first NIX_SSL_CERT_FILE, then the architecture dependent default).

Yes, gnutls should be patched, but the priority has been put on curl, git and the underlying openssl as these are needed by nix itself. Now that the crisis has calmed, and that we have a clear view of the situation, we could open a tracking issue with all the packages that need attention.

[kirelagin]

But I have no idea where it belongs.

I feel that a separate section in the Nixpkgs manual would be the right place. There should also be a link to that section somewhere in the sections that teach how to write expressions. And a separate section in the NixOS manual detailing the way the bundle gets linked into the default location or however it is supposed to be implemented.

By derogation to that rule, on darwin

I am not sure if there really has to be a derogation for darwin. $NIX_SSL_CERT_FILE is already being set to the bundle in the user profile, wouldn’t it be better to alter the profile script to also check the default profile? By the way, should the $NIX_SSL_CERT_FILE logic kick in only on non-NixOS? You are saying, this is an override of the default location, but right now this override is used unconditionally. Do we expect the user to set it manually (and are going to remove this logic from the profile script)?

[layus]

By the way, should the $NIX_SSL_CERT_FILE logic kick in only on non-NixOS? You are saying, this is an override of the default location, but right now this override is used unconditionally. Do we expect the user to set it manually (and are going to remove this logic from the profile script)?

No, the override is not used on NixOS. On NixOS, NIX_SSL_CERT_FILE is unset, and the default path /etc/ssl/certs/ca-certificates.crt is populated with the right symlink. See

$ readlinks /etc/ssl/certs/ca-bundle.crt 
/etc/ssl/certs/ca-bundle.crt
/etc/static/ssl/certs/ca-bundle.crt
/nix/store/5wv7p63ybssz0siyf51fk17q2fnj547m-etc/etc/ssl/certs/ca-bundle.crt
/nix/store/csv151bnq52hvir5yhv81rqg1yaj2qhs-ca-certificates.crt

cat $nixpkgs/nixos/modules/security/ca.nix
    [...]
    # NixOS canonical location + Debian/Ubuntu/Arch/Gentoo compatibility.
    environment.etc."ssl/certs/ca-certificates.crt".source = caCertificates;
    [...]

[layus]

I am only asking because fetching gpg keys currently fails on darwin exactly because gnutls does not seem to be capable of locating any certificates.

@kirelagin It seems to me that gnutls on darwin reads the real MacOS certificates. I see no reason to force it to use nix ca-cartificates.crt in place of the global system store. The code fetches them using SecTrustSettingsCopyCertificates. That should do it. Could you investigate more your particular issue ?

[kirelagin]

@layus Hm, I don’t know. dirmngr of gnupg invokes gnutls_certificate_set_x509_system_trust and it returns an error saying “An unimplemented or disabled feature has been requested.” Might be that our expression somehow blocks OS detection in gnutls. Will investigate.

[layus]

@kirelagin My bad, i looked at the master source code. A new commit (https://gitlab.com/gnutls/gnutls/commit/c0eb46d3463cd21b3f822ac377ff37f067f66b8d) added the feature a month ago. Building from master, it should work correctly, I think.

Now you tell me, what is the sensible thing to do with gnutls on macos, get default from Keychain (as per gnutls vanilla), or from NIX_SSL_CERT_FILE ? This topic is filled with not so easy questions like this.

[arianvp]

(triage) is this issue still actionable? Do we want to document the current behaviour in Nixpkgs manual?

[vcunat]

There's an action happening right now: https://github.com/NixOS/nixpkgs/pull/58611 (gnutls support).

---

Now, the general issue. Current state (mainly from openssl, now to happen for gnutls):

• if $NIX_SSL_CERT_FILE is set, we use only that file for the default "trust store" (utilizing secure_getenv() equivalent, i.e. careful around setuid stuff)
• otherwise there's some default that includes at least one of the /etc/ssl/certs/ we have on NixOS (we might hard-code that via a configure-time option, for example).

I see a patch for the first point for go as well; other implementations probably don't respect (some of) this ATM.

I can't remember any (other) discussion around this during the past year (or so), but I haven't been watching too much. So, if this behavior is what we want (rough consensus), I expect it will be best to document somewhere with a link to the most relevant discussion threads – perhaps in the NixPkgs manual as you suggest. Any more comments? (I'd wait with documentation until the gnutls issue gets merged/resolved.)

[kirelagin]

I have looked into all this just recently, and I have prepared a write up, that I was going to post as a separate issue/RFC to start the discussion, but let me post it here: https://www.notion.so/serokell/Trust-Stores-4e10e6f0ef1e4447afad745a688b3b72

tldr; switching everything to p11-kit is very likely the right thing to do (and it works on macOS), instead of patching openssl, gnutls, ... individually, we can patch p11-kit directly and make sure everyone else goes through it.

[vcunat]

The gnutls patch actually does result into code supporting using some p11 pseudo-URL instead of file-path IIRC.

[kirelagin]

The gnutls patch

Which patch? gnutls supports using a p11-kit URL natively.

[vcunat]

Yes... and the patch just makes the compiled-in default path overridable by an env variable.

[kirelagin]

Ok, I see, but there is a compile-time option in gnutls that specifies the default trust store url. My point is that if all libraries support using p11-kit one way or the other natively, then we should compile them that way, and then patch only p11-kit to take the certificates location path from the env variable – that’s one patch (which may even be upstreamable) rather than a patch per crypto library.

[vcunat]

Possibly. I know almost nothing around p11-kit, but I can see a risk... these patches we now have are fairly simple (to write and maintain). Adding support to p11-kit is likely to be upstreamable, but it would surely be significantly more complex.

Looking a bit more, I see this Fedora plan is very close and completed years ago – it doesn't seem ideal e.g. in the way OpenSSL is handled, but perhaps there's been some improvement since.

As noted multiple times, non-NixOS is quite a complication, as there we'd like to use a trust store that we do not control (while sharing the binaries with NixOS). I mean, on NixOS we only rarely need to use these overriding variables (and they still aren't defined by default); our gnutls didn't even recognize any so far.

[vcunat]

Let me merge that PR for now, regardless of whether we do some p11 project in the end.

[vcunat]

Closely related: https://github.com/NixOS/nixpkgs/pull/61179

[rnhmjoj]

Has anyone used the p11-kit supposed drop-in replacement for libnssckbi.so? I tried with this but it doesn't seem to work, meaning the system store is not used still.

diff --git a/pkgs/development/libraries/nss/default.nix b/pkgs/development/libraries/nss/default.nix
index 056f98472be..4f1a85d61b8 100644
--- a/pkgs/development/libraries/nss/default.nix
+++ b/pkgs/development/libraries/nss/default.nix
@@ -1,4 +1,4 @@
-{ stdenv, fetchurl, nspr, perl, zlib, sqlite, fixDarwinDylibNames }:
+{ stdenv, fetchurl, nspr, perl, zlib, sqlite, p11-kit, fixDarwinDylibNames }:

 let
   nssPEM = fetchurl {
@@ -113,6 +113,10 @@ in stdenv.mkDerivation rec {
     moveToOutput bin/nss-config "$dev"
     moveToOutput lib/libcrmf.a "$dev" # needed by firefox, for example
     rm -f "$out"/lib/*.a
+
+    # patch p11-kit in
+    rm $out/lib/libnssckbi.so
+    ln -s ${p11-kit}/lib/pkcs11/p11-kit-trust.so $out/lib/libnssckbi.so
   '';

   meta = with stdenv.lib; {

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[rnhmjoj]

Yep, still important to me. If anyone could figure out how to make browsers use the system store I would happily open a PR.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[mohe2015]

Still important to me

[rnhmjoj]

Finally #96763 made it into unstable. We now have all nss-based application using p11-kit and reading certificates from the system-wide trust store!

[sigprof]

Finally #96763 made it into unstable. We now have all nss-based application using p11-kit and reading certificates from the system-wide trust store!

…and this change needed to be turned off specifically for Firefox due to #126065 — the current form of p11-kit integration is not enough for Firefox, because the Mozilla code relies on the CKA_NSS_MOZILLA_CA_POLICY attribute, therefore the conversion of the builtin NSS certificate store to a format supported by p11-kit must be performed in a way that preserves that attribute (and some others too). If this is not fixed, at some future time (when the ESR branch will switch to 91) the firefox-esr and thunderbird packages would need the same treatment .

[rnhmjoj]

The solution should be to change pkgs.cacert and the security.pki module to generate the trust store in the p11-kit format and convert it to the openssl one. It shouldn't be too much work considering we can reuse the fedora script. I'll probably won't be able to work on this soon, though. Hopefully someone else will beat me to it.