On macOS, can't download distfiles for packages via HTTPS

[jlmuir]

On macOS Sierra 10.12.4, pkg_comp can't download the distfiles for packages via HTTPS. See sandboxctl issue #2 for why this is. This is problematic since a number of packages provide their distfiles via HTTPS only (e.g., devel/protobuf and devel/googletest). A workaround is to download the distfiles by hand outside of the sandbox and place them in /var/pkg_comp/distfiles, but that's obviously a pain and far from automated.

[jmmv]

I'm keeping the duplicate sandboxctl issue open for now as I don't know where the fix is going to happen.

For now, I think that adding the following to extra.mk.conf should workaround the problem:

USE_BUILTIN.curl=no

Let me know if it does.

[jlmuir]

Unfortunately, that doesn't work. I don't think www/curl has a built-in capability (i.e., there's no www/curl/builtin.mk).

I tried the following in extra.mk.conf:

TOOLS_PLATFORM.curl=

That caused bmake fetch to install pkgsrc's curl, but it said it couldn't resolve the host for each URL it tried to download the distfile from:

# bmake fetch
=> Bootstrap dependency curl-[0-9]*: NOT found
=> Verifying bin-install for ../../www/curl
===> Binary install for curl-[0-9]*
=> Installing curl-[0-9]* from /pkg_comp/packages/pkg/All
[...]
curl-7.53.1nb1 successfully installed.
=> Returning to build of googletest-1.8.0
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
=> Fetching googletest-1.8.0.tar.gz
=> Total size: 1281617 bytes
curl: (6) Couldn't resolve host 'github.com'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'cdn.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp6.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.fr.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.fr.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
curl: (6) Couldn't resolve host 'ftp.NetBSD.org'
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
*** Error code 1

Stop.
bmake: stopped in /pkg_comp/pkgsrc/devel/googletest

However, I'm still having issue #15 where DNS resolution with curl does not seem to work in the sandbox, so I'm thinking that is hiding this HTTPS issue. For example, native and pkgsrc curl are giving a different error message than that reported in sandboxctl issue #2; the error is now about being unable to resolve the host as opposed to an SSL certificate problem, so I think now it's not even getting to the SSL handshake:

# /usr/bin/curl -IsS https://github.com/ | head -n 1
curl: (6) Could not resolve host: github.com
# /opt/pkg/bin/curl -IsS https://github.com/ | head -n 1
curl: (6) Couldn't resolve host 'github.com'

I don't know when it started happening, but issue #15 seems like it might need to be the first thing to sort out.

[jlmuir]

OK, now that issue #15 is fixed (Thanks for that!), I tried again, and the native curl still doesn't work in the pkg_comp sandbox:

# /usr/bin/curl -IsS https://github.com/ | head -n 1
curl: (35) SSL certificate problem: Couldn't understand the server certificate format

For the case of downloading the devel/googletest distfile, it emits the same curl error message as above, but the distfile does get downloaded, so I assume pkgsrc is falling back to another fetch tool or a different download location that is not HTTPS:

# cd devel/googletest
# bmake fetch
=> Bootstrap dependency digest>=20010302: NOT found
=> Verifying bin-install for ../../pkgtools/digest
===> Binary install for digest>=20010302
=> Installing digest>=20010302 from /pkg_comp/packages/pkg/All
pkg_add: Warning: package `digest-20160304' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
digest-20160304 successfully installed.
=> Returning to build of googletest-1.8.0
===> Skipping vulnerability checks.
WARNING: No /opt/pkg/libdata/pkgdb/pkg-vulnerabilities file found.
WARNING: To fix run: `/opt/pkg/sbin/pkg_admin -K /opt/pkg/libdata/pkgdb fetch-pkg-vulnerabilities'.
=> Fetching googletest-1.8.0.tar.gz
=> Total size: 1281617 bytes
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) SSL certificate problem: Couldn't understand the server certificate format
fetch: Unable to fetch expected file googletest-1.8.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1251k  100 1251k    0     0  1181k      0  0:00:01  0:00:01 --:--:-- 1181k

Then I tried forcing the use of the pkgsrc curl instead of the native curl by adding the following to extra.mk.conf:

TOOLS_PLATFORM.curl=

This made the bmake fetch download the distfile without an error message:

# bmake fetch
=> Bootstrap dependency curl-[0-9]*: NOT found
=> Verifying bin-install for ../../www/curl
===> Binary install for curl-[0-9]*
=> Installing curl-[0-9]* from /pkg_comp/packages/pkg/All
pkg_add: Warning: package `libidn-1.33' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pkg_add: Warning: package `gettext-lib-0.19.8.1' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
gettext-lib-0.19.8.1: copying /opt/pkg/share/examples/gettext/locale.alias to /opt/pkg/share/locale/locale.alias
libidn-1.33: registering info file /opt/pkg/info/libidn-components.png
libidn-1.33: registering info file /opt/pkg/info/libidn.info
pkg_add: Warning: package `perl-5.24.1' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
openssl-1.0.2k: copying /opt/pkg/share/examples/openssl/openssl.cnf to /opt/pkg/etc/openssl/openssl.cnf
curl-7.53.1nb1 successfully installed.
=> Returning to build of googletest-1.8.0
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /opt/pkg/libdata/pkgdb/pkg-vulnerabilities file found.
WARNING: To fix run: `/opt/pkg/sbin/pkg_admin -K /opt/pkg/libdata/pkgdb fetch-pkg-vulnerabilities'.
=> Fetching googletest-1.8.0.tar.gz
=> Total size: 1281617 bytes
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   132    0   132    0     0    952      0 --:--:-- --:--:-- --:--:--   992
100 1251k  100 1251k    0     0  1711k      0 --:--:-- --:--:-- --:--:-- 1711k

However, the pkgsrc curl still fails the https://github.com/ test, but with a different error message:

# /opt/pkg/bin/curl -IsS https://github.com/ | head -n 1
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

So, I tried installing the security/mozilla-rootcerts package and installing the certs (all still in the sandbox):

# pkg_add mozilla-rootcerts
# cd /opt/pkg/etc/openssl/certs
# mozilla-rootcerts extract
# mozilla-rootcerts rehash

And now, the pkgsrc curl passes the https://github.com/ test:

# /opt/pkg/bin/curl -IsS https://github.com/ | head -n 1
HTTP/1.1 200 OK

I tried the native curl one more time with the --capath option to try to make it use the same certificate directory that the pkgsrc curl is using, but that didn't work:

# /usr/bin/curl --capath /opt/pkg/etc/openssl/certs -IsS https://github.com/ | head -n 1
curl: (35) SSL certificate problem: Couldn't understand the server certificate format

So, I still don't know why the native curl doesn't work. But if the decision is made to punt on that and just use the pkgsrc curl, then just setting TOOLS_PLATFORM.curl= in extra.mk.conf seems to be insufficient to solve the problem: the root certs from the pkgsrc mozilla-rootcerts package are needed as shown above (i.e., the mozilla-rootcerts package needs to be installed, and the certs need to be extracted and hashed).

[jmmv]

I dug into this a bit last week, seeing if maybe some obvious missing path in the sandbox was behind these failures, but couldn't find anything to explain why curl fails. I plan to research this further because, if curl is failing in this manner, other things will fail too: the sandbox is not working appropriately.

I'd rather not set up curl from pkg_comp to workaround this issue because it's complex as you discovered (need the certificates).

If we cannot fix the sandbox to make curl run... then I'd rather investigate making USE_BUILTIN.curl=no work properly in pkgsrc or similar so that pkg_comp can trivially tell pkgsrc to avoid curl without requiring special build logic.

[jlmuir]

In the sandbox, I can't seem to get the native curl's --cacert nor --capath options to work. So, I must be using them incorrectly, the file or directory they point at is not in the required format, or they're being ignored.

I don't know how far you got with researching why the native curl isn't working, but I'm finding things that make we wonder if the problem is that the native curl uses the System or login keychain for getting the certificates, and that won't work in a sandbox (?).

The things I found are as follows.

The curl(1) man page on macOS Sierra says the following about the --cacert option:

(iOS and macOS only) If curl is built against Secure Transport, then this option is supported for backward compatibility with other SSL engines, but it should not be set. If the option is not set, then curl will use the certificates in the system and user Keychain to verify the peer, which is the preferred method of verifying the peer's certificate chain.

A curl mailing list email from 2013 says (among other things):

In Mavericks, Apple changed from curl 7.24.0 to 7.30.0, and in the process, they switched the TLS/SSL engine used by their curl, from OpenSSL to their own Secure Transport engine.

and

Unless Apple changed curl's code behind my back, the --cacert option no longer works. This is because the new engine reads its certificates from a security database (the Keychain) instead of from certificate files.

A Phusion blog post says:

curl on macOS uses the macOS Security framework internally to load the client cert.

[jmmv]

Yeah, that (curl using the native keychains) was the theory I was chasing as well -- and attempted to mount various Keychain-related directories in the sandbox to no avail.

One thing I wanted to try was to play with security and see if that worked at all. Maybe you'd give it a try? I can't look until tomorrow at the earliest.

[jmmv]

Found the problem playing with security list-keychains. This is a sandbox-creation issue, so closing in favor of https://github.com/jmmv/sandboxctl/issues/2.