Security.framework fails to access keychain on Darwin

[sirn]

Issue description

Security.framework that were installed inside Nix could not access the keychain and cause packages that relies on SecureTransport on Mac OS X to fail (e.g. aria2 #15029, go #10875) with errors such as -9808 (errSSLBadCert) or -67674 (errSecMDSError).

The attached reduce test cases (see steps to reproduce), when compile under non-Nix-ified environment, DarwinSecurityHandshake will attempt to make a SSL handshake to example.com:443 and print a message, and DarwinSecurityCerts will read the certificate store and print the number of certs installed:

$ clang -framework Security -framework CoreFoundation -o DarwinSecurityHandshake DarwinSecurityHandshake.c
$ clang -framework Security -framework CoreFoundation -o DarwinSecurityCerts DarwinSecurityCerts.c
$ ./DarwinSecurityHandshake
SSL handshake OK
$ ./DarwinSecurityCerts
SSL certificates available: 195

Both files will fail when compile under nix-shell -p darwin.apple_sdk.frameworks.Security -p clang:

$ clang -framework Security -framework CoreFoundation -o DarwinSecurityHandshake DarwinSecurityHandshake.c
$ clang -framework Security -framework CoreFoundation -o DarwinSecurityCerts DarwinSecurityCerts.c
$ ./DarwinSecurityHandshake
SSL handshake error: -9808
$ ./DarwinSecurityCerts
SSL certificate retrieve error: -67674

However, it will pass when the Nix-compiled binaries are run under root:

$ sudo ./DarwinSecurityHandshake
SSL handshake OK
$ sudo ./DarwinSecurityCerts
SSL certificates available: 193

Steps to reproduce

1. Download reduced test cases from Gist
2. Run nix-shell -p darwin.apple_sdk.frameworks.Security -p clang and compile it.

   Technical details

   • System: Mac OS X 10.11.5 (Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64)
   • Nix version: nix-env (Nix) 1.11.2
   • Nixpkgs version: 16.09pre83147.df89584

[matthewbauer]

Apple has some of their own patches that they apply to OpenSSL and OpenSSH. I'm wondering if one of those being missing is causing the issue. Right now OpenSSL and OpenSSH are built from source but Apple provides some magic patches when they build them. iOpenSSL and iOpenSSH have some differences from their sources. Specifically this "Trust Evaluation Agent" that Apple has basically bypasses OpenSSL's builtin root certificates. And Apple uses this patch to get it to work.

Could you try running the first two commands that worked before again but this time set "OPENSSL_X509_TEA_DISABLE" to 1? If they don't work now, then we at least know that we need to build in "TEA" support with Libsecurity. It's possible this has nothing to do with that but it's just my hunch that Apple's got some special stuff baked into its version of OpenSSL.

[sirn]

@matthewbauer I tried env OPENSSL_X509_TEA_DISABLE=1 ./DarwinSecurityHandshake but no luck. Still the same error. I'll take a look at the patch later.

On the other hand, seeing Nix-built binaries works when running sudo reminds me of Xhyve where it couldn't access vmnet.framework due to the binary not being codesigned (but sudo can circumvent that), though the codesign -vvv output of the system's one and Nix's one looks the same. @zimbatm suggested me to run dtruss in #15029 (I think we should move the discussion here) and I'm in middle of understanding what it is doing.

The main difference between Nix-built, Nix-and-sudo and the native build I have found so far is that native build and Nix-and-sudo tries to access mds files in /private/var/db/mds/system/ while the Nix-built one did not. I'll continue to investigate.

Interesting enough, in the syslog there is confstr on _CS_DARWIN_USER_CACHE_DIR returned an error. message, which could be tracked down to MDSSession.cpp. Hmm.

[mmahut]

Are there any updates to this issue, please?

[sirn]

Not much from my side. #38624 could be a potential fix but it cannot be implemented due to private key management issue. With Apple getting more and more restrictive I'm not sure if there's any way around this except linking against system Security.framework.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.