dlwyatt
commented
9 years ago In doing more research on this, I found that for RSA certificates that are for signatures only (such as code-signing certs), the public key object still looks like it could be used for encryption. However, the private key would produce the "Bad Key" error when it comes time to try to decrypt the data.
As a result, I decided to treat the key usage extension (specifying Key Encipherment for RSA or Key Agreement for ECDH) as mandatory. This is the only way I know of (at this time) to predict whether the private key is going to allow decryption of the AES key on the other end.
Have encountered some certificates that cannot be used for decryption, even though they should, according to their key usage / enhanced key usage extension values.
For RSA certificates, the more reliable way to verify that a cert is usable is to check the KeyExchangeAlgorithm property of the RsaCryptoServiceProvider objects for the public / private keys. If this is not $null, you can use that key for encryption or decryption. Need to research and see if there's a similar flag I can check for CNG keys.
Note: I've also come across some very weird certs where the public key claims to be usable for key exchange, but the private key claimed to be signature only. I have no idea how to reproduce this in the tests, and don't know if there's any way to predict this situation just by looking at the cert and public key.
In any case, when you try to call the Encrypt() or Decrypt() methods on an RSA cert without this KeyExchangeAlgorithm set, you get a generic error of "Bad Key." We can improve on this error output, and also exclude these certs from the Get-KeyEncryptionCertificate output.