Async scan seems to fail or result is not waited on

Hi Ken,

OSX version is Mojave 10.14.5, compiling master.

I am using your library to expose the keychain via PKCS#11 to the Viscosity VPN client ( https://www.sparklabs.com/viscosity/ ).

The behaviour that I am seeing is that the async call for the certificates either never completes, or, the code (or the caller of the shlib?) is not waiting for the list to be populated; as such, no certificates are being offered up.

If I move the background_cert_scan call into the foreground then it works as expected.

I didn't really understand your comment preceding the async dispatch (not delved much further into your code yet); as mentioned, I am not using this for Firefox but Viscosity. Having said that, I see the same behaviour when using p11tool to interact with the dylib.

Is this a bug that only manifests on Mojave, perhaps?

Ok, this is kind of complicated, so I will try to explain it in greater detail.

First ... do you have a lot of user certificates in your local Keychain? You would if, for example, you use Apple Mail and receive a lot of S/MIME signed/encrypted email.

I, personally, do not. And so when I first implemented the code to create the Keychain slot it would run it in the foreground and everything was happy. I noticed a slight slowdown (like 0.3 seconds) but I didn't really think it was an issue so I enabled it for everything.

Then I deployed this to my users, who DID use use Apple Mail and sent/received a lot of S/MIME email. WHOO boy. At every startup I was seeing it take 2-5 MINUTES (one user it was closer to 10 minutes; this happens on High Sierra and Mojave). So I dug into this further.

The core problem is IF you have a lot of user certificates (on the order of hundreds) then certain operations that involve iterating over them take a long time, like "minutes" long (I did benchmarking; it was all stuck in one Security framework call). This created unacceptable performance for every application that used this library. AND my original goal for the Keychain Certificate slot code was really just to make Firefox work, because importing certificates into Firefox is a huge pain.

So I ended up making two changes. First, I only enabled the Keychain Certificate slot for certain applications (by default only Firefox; I think you saw that). And secondly because I didn't want Firefox to hang at startup, what happens THERE is the slot is marked as "not ready" and when the background job completes then the slot is marked as "Ready". Firefox occasionally polls the slot status so when it sees the slot is ready it then will access it. But tools like p11tool don't do that; they see the slot as "not ready" and then don't do anything with it (which is totally reasonable) and because it's a command line tool it's not going to wait around for the slot to be ready.

I hope that all makes sense. Now. on to solutions ... I could easily implement a new preference that was like the existing keychainCertSlot preference, but instead ran in the foreground. From what I am hearing you say, that would solve your problem. The disadvantage here is that it would have bad performance if your keychain was large, but it doesn't sound like that is a problem for you. Would that work?

Hi Ken, thanks for the detailed explanation and for your offer of help with a solution. In my specific usecase we aren't dealing with users who have a large number of certificates. The typical user would have maybe five or six --- and, they would be filtered out from any others in the keychain by virtue of the CA filter.

(Incidentally, whilst my primary usecase for this is openvpn via Viscosity, it would also be good if I could hook it into Firefox, as the same certs are used for accessing browser resources as well as the VPNs; with that in mind, if we do make any changes to the background/foreground behaviour, perhaps it would be sensible to make that toggle-able per application, in the same vein as the other options? BTW I am happy to try and help by raising a PR with a proposed implementation).

Back to my primary use case. Unfortunately I'm still unable to get Viscosity to use a certificate provided via the dylib. By bringing the certificate scan into the foreground, it does make the query for available certs syncronous with the UI, which is great; however, when Viscosity tries to connect using such a certificate, it doesn't seem able to use the aforementioned cert.

Having done a bit of digging I can see that Viscosity interacts with openvpn by calling the binary with various options command-line options: "--pkcs11-id", "--pkcs11-providers", etc. (BTW one of those options it to launch the client with a management socket exposed; hopefully I won't need to debug that far though). Anyway, if I mimick what Viscosity is doing with openvpn (sans the management socket options), I am getting prompted in the terminal (by openvpn) to provide a password for "Keychain Certificate". e.g.

5 $ "/Library/Application Support/Viscosity/openvpn" --config config.conf 
Mon Jun  3 17:30:14 2019 OpenVPN 2.4.6 x86_64-apple-darwin [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] built on Nov 23 2018
Mon Jun  3 17:30:14 2019 library versions: OpenSSL 1.0.2q  20 Nov 2018, LZO 2.10
Mon Jun  3 17:30:14 2019 PKCS#11: Adding PKCS#11 provider '/Users/joewildish/Library/Developer/Xcode/DerivedData/keychain-pkcs11-ensfsnkzawtltdgmixiklyvrdabu/Build/Products/Debug/libkeychain-pkcs11.dylib'
Mon Jun  3 17:30:14 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]ww.xx.yy.zz:443
Mon Jun  3 17:30:14 2019 Attempting to establish TCP connection with [AF_INET]ww.xx.yy.zz:443 [nonblock]
Mon Jun  3 17:30:15 2019 TCP connection established with [AF_INET]ww.xx.yy.zz:443
Mon Jun  3 17:30:15 2019 TCP_CLIENT link local: (not bound)
Mon Jun  3 17:30:15 2019 TCP_CLIENT link remote: [AF_INET]ww.xx.yy.zz:443
Enter Keychain Certificates token Password:
Mon Jun  3 17:30:19 2019 PKCS#11: Cannot perform signature 512:'CKR_FUNCTION_REJECTED'
Mon Jun  3 17:30:19 2019 OpenSSL: error:14099006:SSL routines:ssl3_send_client_verify:EVP lib

However, the certificate it is requesting has no password, and the keychain itself is unlocked. Also, I specifically set an option on the cert key to allow all apps to be able to use it (for the purposes of testing at least).

Now, it feels like this may be an openvpn specific thing, rather than something to do with your library? Maybe openvpn is hardcoded to ask for a password? (I did wonder if this was the Keychain equivalent of a PIN, but I have not toggled the "askPIN" option, so, as I understand it, any authentication to the certificate itself should be undertaken by the OS rather than the caller of the dylib?). Anyway, if you have any ideas on this, they'd be gratefully received. I'll continue tomorrow with some more debugging to see if I can make it work.

Ooof. Alright, well ... this is a bit more complicated than I had originally hoped.

The idea behind the "Keychain Certificate" slot was to make it so certificates could just be imported by Firefox; I didn't really expect users (nor did I implement) the necessary bits to make it so you could do crypto operations on those certificates via the PKCS#11 interface. It's POSSIBLE, but it wasn't a target of this code. There are a number of places where it is assuming that the identity is on a smartcard.

First, some definitions. When you say "use a certificate with the VPN software", what EXACTLY do you mean? Because if you are just using a certificate and the associated public key you cannot typically perform a signature operation. You can do that with a PRIVATE key. In MacOS X parlance a private key plus a public key/certificate is called an identity.

If you want to use keychain-pkcs11 with a software identity, well ... that is possible, but will require some work. First, you might find it helpful/interesting to see the keychain-pkcs11 debug logs; see the home page under "Configuration & Debugging" for more information.

As for the whole business about asking for a password .. well, that's also a little complicated (you're probably sensing a pattern). Having the Security framework do the pop-up for entering a PIN is more of the "MacOS way", but the PKCS#11 API doesn't QUITE have that concept. What it does have is something called CKF_PROTECTED_AUTHENTICATION_PATH where it will indicate to the PKCS#11 application that the password/PIN will be entered "on the device" and as a result the application shouldn't ask for it. But not all applications work correctly with that flag. It may be that's the case here, or something else is going wrong.

Aha. Yes, I was assuming all crypto operations were going to be passed off to the keychain, including signing (the private key will be required for interacting with the VPN), which I guess means I need access to the "identity" in OSX-speak. I'm AFK at the moment but tomorrow I'll run up openvpn in lldb to better see how it interacts with your library. I've used the log stream stuff so far -- very useful BTW -- but nothing quite beats having a breakpoint :-)

Well, all crypto operations ARE passed off to the keychain, but it's important to understand what does what.

When I labeled that second virtual slot as "Keychain Certificates", that was literally correct. All it offers is certificates, full stop. Specifically it doesn't offer public keys or private keys, nor any crypto operations (because without any KEYS, there is no crypto operations to perform). It just offers certificates. This is why I want to make sure we're both on the same page when it comes to terminology. I am going to proceed with the assumption that you want to use an IDENTITY stored in your Keychain.

There MAY be a way to make it just work very simply. In the function scan_identities() the query is made into the Security framework to get a list of all known identities. It does this by building up a query dictionary for the SecItemCopyMatching() call (there are comments in there which explain every argument). Right now it is limited to ONLY items which appear on smartcards by the use of kSecAttrAccessGroup (which is set to kSecAttrAccessGroupToken). If you comment that item out of the query dictionary it MAY return your stored identity (you need to comment the item out of the keys and values array, because they need to match up). I say "may" because nothing is simple or easy when it comes to the Security framework. Using the logging facility will let you know if keychain-pkcs11 finds any identities.

I had a go at changing scan_identities by removing the filtering on kSecAttrAccessGroupToken. However, the add_identity function then started failing as the dict being passed to it did not contain the key type... so yes, it looks like there is more to it... If I get the chance I'll take a further look and hopefully submit a PR. Although, I can see from your comments here and in your code that the Apple Security framework is a pain to deal with :( In the meantime I am going with a pkcs#12 bundle file. Thanks for your help Ken!

kenh / keychain-pkcs11

Async scan seems to fail or result is not waited on #4