Using `curl` to retrieve a HTTPS url that requires a self-signed certificate fails on macOS

DUOLabs333 commented 2 months ago

Describe the bug

Let's say you have a website at https://testing.local, and you need to set your DNS server to w.x.y.z in the Preferences Pane to access it. curl https://testing.local will fail with curl: (35) OpenSSL/3.0.14: error:16000069:STORE routines::unregistered scheme.

Steps To Reproduce

Steps to reproduce the behavior:

Run nix-shell with

# shell.nix
let
pkgs = import  <nixpkgs> {};
in
pkgs.mkShell {
packages = [ pkgs.git pkgs.curl ];
# ...
}

Run a server on a.b.c.d (it doesn't matter what it is)
Set up a DNS server such that foo.bar resolves to a.b.c.d
Add this DNS server to your DNS settings in the Preferences Pane
Run curl https://testing.local
See error

Expected behavior

There should be no error when running the command.

Screenshots

N/A

Additional context

Interestingly, curl https://a.b.c.d works as expected.

This seems to be a certificate issue: bypassing SSL certificate verification with -k fixes the issue. I am using a custom SSL certificate, but I'm not sure how to point NIX to it.

After some digging, I found that this has been brought up before: https://github.com/NixOS/nixpkgs/issues/283793

Notify maintainers

@lovek323

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 20.4.0, macOS 11.3.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/Users/system/.nix-defexpr/channels/nixpkgs`

Add a :+1: reaction to issues you find important.

DUOLabs333 commented 2 months ago

I got around this issue by exporting the keychains in Keychain to a single PEM-encoded PEM file, and exporting the path to NIX_SSL_CERT_FILE and GIT_SSL_CAINFO (since git uses libcurl, at least by default):

rm ${keychainPath}; export PATH="${builtins.getEnv "PATH" }:$PATH"; security list-keychains | xargs -I{} sh -c 'security find-certificate -a -p "{}" >> ${keychainPath}; cat ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt >> ${keychainPath}'

Adjust as needed.

vinylen commented 2 months ago

I also experienced this with a x86 system, adding the -k flag solves it but, this has never been an issue in other distributions with curl for me. Is there something that needs to be looked at here? Or are we missing something critical in our NixOS systems for Curl?

I think this should be reopened.

❯ curl -V
curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 OpenSSL/3.0.14 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.61.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zst

- system: `"x86_64-linux"`
 - host os: `Linux 6.10.8, NixOS, 24.05 (Uakari), 24.05.20240907.68e7dce`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - nixpkgs: `/nix/store/32y945fdcfh1j5b8zv677vw7np9r7bbw-source`

EDIT: I might have been too hasty, i found this issue upstream, so it's not specific to NixOS.

If i modify the curl command to specifically use /etc/ssl/certs/ca-bundle.crt the error is more human readable:

CURL_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt curl https://a.b.x.d
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

EDIT 2:

Interestingly i don't get this error when i use the package "curlWithgnuTls":

nix-shell -p curlWithGnuTls
these 8 paths will be fetched (0.84 MiB download, 3.69 MiB unpacked):
  /nix/store/3iiwbpn2ll9z9k5dzm4a20hh4gf60glh-curl-8.7.1
  /nix/store/npj6a06hjihxswcc8iaqfxw4mhb0q8kc-curl-8.7.1-bin
  /nix/store/vx4rvsk3g35qgk434c71krfvrlfrkzvh-curl-8.7.1-dev
  /nix/store/ai64cb372lg4gqd8lp5p3y4n31d7xf8l-curl-8.7.1-man
  /nix/store/aakpck7mmlzf9m9sf1hkp3bjkm90kxnq-gmp-with-cxx-6.3.0-dev
  /nix/store/k8bifycm95cvr8fd3zd52j69kp9kf8cs-gnutls-3.8.5-bin
  /nix/store/ch2f11z7bwi856zclmly123dmwxa2v96-gnutls-3.8.5-dev
  /nix/store/1k7cfhsdmfnbp560x3vr8qs4x2rxyj85-nettle-3.9.1-dev
copying path '/nix/store/ai64cb372lg4gqd8lp5p3y4n31d7xf8l-curl-8.7.1-man' from 'https://cache.nixos.org'...
copying path '/nix/store/3iiwbpn2ll9z9k5dzm4a20hh4gf60glh-curl-8.7.1' from 'https://cache.nixos.org'...
copying path '/nix/store/k8bifycm95cvr8fd3zd52j69kp9kf8cs-gnutls-3.8.5-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/aakpck7mmlzf9m9sf1hkp3bjkm90kxnq-gmp-with-cxx-6.3.0-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/1k7cfhsdmfnbp560x3vr8qs4x2rxyj85-nettle-3.9.1-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/npj6a06hjihxswcc8iaqfxw4mhb0q8kc-curl-8.7.1-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/ch2f11z7bwi856zclmly123dmwxa2v96-gnutls-3.8.5-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/vx4rvsk3g35qgk434c71krfvrlfrkzvh-curl-8.7.1-dev' from 'https://cache.nixos.org'...

[nix-shell:/]$ curl https://a.b.x.d
curl: (60) server certificate verification failed. CAfile: none CRLfile: none
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

NixOS / nixpkgs