Support for HSM stored certificates?

[JonathanLydall]

It seems that if one wants to have their certificate stored on an HSM (a requirement for EV code signing certs for DigiCert), it needs to be stored as a Key in the KeyVault.

https://docs.microsoft.com/en-us/azure/key-vault/general/about-keys-secrets-certificates#object-types

Are there any plans to support signing from certficates stored in HSMs?

[garrett-wood]

I've been using this for over a year with EV certs in Key Vault HSMs.

I'm pretty sure it was the purpose of the project.

[vcsjones]

This should already work, I believe @clairernovotny is using EV DigiCert certificates.

I'm not clear on what you think you need that AzureSignTool doesn't have - can you please elaborate?

[JonathanLydall]

Thanks @garrett-wood and @vcsjones for replying, I've found it hard to find clarification on this through internet searches so would appreciate any more information you're willing to offer.

As per the above link, it seems like just putting a certificate into the "Certificates" part of Azure KeyVault does NOT store it on an FIPS 140-2 Level 2 compliant HSM (as per DigiCert's requirements), instead it has to be in the "Keys" part. By all means please correct me if I'm wrong here by pointing me to something which clarifies this.

Otherwise, from basic experimentation with and a glance of AzureSignTool's documentation, I didn't see how to use a "Key" value rather than a "Certificate" from KeyVault, hence my asking here.

[judge2020]

https://docs.microsoft.com/en-us/azure/key-vault/keys/about-keys#about-keys:~:text=Vaults%20use%20FIPS%20140%2D2%20Level%202,HSM%2Dkeys%20in%20shared%20HSM%20backend%20infrastructure.

Vaults use FIPS 140-2 Level 2 validated HSMs to protect HSM-keys in shared HSM backend infrastructure.

I can't test this since I don't subscribe to the premium SKU, but I think (only new?) vaults are set as HSM-backed when you pay for the premium key vault, and this page doesn't make a distinction between keys and certificates (although the language is confusing). Might be useful to ask support for clarification.

[clairernovotny]

All Premium Key Vaults uses the HSM backed vaults. x.509 Certificates technically do not contain a key. The pfx/p12 file you often see is a key-pair, with the public x.509 certificate and the private key.

To get an x509 certificate that's rooted in the HSM, the certificate signing request (CSR) must be generated from Key Vault with the export option disabled. Then the CER file gets merged in and you have a key-pair that can be used for all the expected operations.

[garrett-wood]

DigiCert specifically will require you to document the HSM backed status but you can just generates a CSR with the correct EKU OIDs in it and merge the cert that DigiCert provides.

Just have your auditor observe the key generation process (making the CSR) and you'll be fine as long as you specify RSA- HSM or EC-HSM.

[clairernovotny]

Yup. I've filled out the HSM letter once a year or so as needed and they keep it on file.

[JonathanLydall]

DigiCert specifically will require you to document the HSM backed status but you can just generates a CSR with the correct EKU OIDs in it and merge the cert that DigiCert provides.

Just have your auditor observe the key generation process (making the CSR) and you'll be fine as long as you specify RSA- HSM or EC-HSM.

@garrett-wood, this is the part that's been challenging for us. Use you seem to be aware, DigiCert only has the option of selecting "generic customer provided compliant HSM" for this purpose and then requires that an accredited auditor signs off on our code signing practices. While DigiCert’s requirements make sense to me for organisations with their own HSM devices, we’re looking to use what seems to me to be a solution essentially no less standard than the hardware token they could alternatively send us which doesn’t have the onerous auditing requirement. I hope for the future that DigiCert more properly assesses the Azure KeyVault option to see if the auditing requirement is actually necessary.

Regardless of our frustations, while it’s a requirement of theirs we’ve now been trying to locate an auditor for a reasonable price. My expectation is that using an Azure KeyVault HSM would require a very minimal amount of auditing, essentially just a very basic process of how to securely get the certificate into the KeyVault and ensuring that access is appropriately restricted. But one crowd I found online quoted us thousands of USD, which is an order of magnitude higher than what we expected.

So I’m interested if people like you or @clairernovotny can give an idea what price we should expect to pay for such an audit and can offer recommendations of an accredited individual or organisation that offers such a service at a reasonable rate?

Thanks again for all your help so far.

[JonathanLydall]

https://docs.microsoft.com/en-us/azure/key-vault/keys/about-keys#about-keys.

Vaults use FIPS 140-2 Level 2 validated HSMs to protect HSM-keys in shared HSM backend infrastructure.

I can't test this since I don't subscribe to the premium SKU, but I think (only new?) vaults are set as HSM-backed when you pay for the premium key vault, and this page doesn't make a distinction between keys and certificates (although the language is confusing). Might be useful to ask support for clarification.

Thanks @judge2020, while that URL applies to "Keys" (which I don't think are used at by AzureSignTool), it did put me onto https://docs.microsoft.com/en-us/azure/key-vault/certificates/about-certificates which seems to clarify that an FIPS 140-2 Level 2 HSM is used.

I upgraded our existing KeyVault which we already had for a DV SSL cert from the normal to premium SKU, but with you mentioning there is a question of if it's actually using an HSM now, and along with other advice I've seen online that it's good to have a seperate KeyVault for the code signing certificate, it seems it may just be best to make a new seperate KeyVault which is the premium SKU from the start.

[garrett-wood]

So I’m interested if people like you or @clairernovotny can give an idea what price we should expect to pay for such an audit and can offer recommendations of an accredited individual or organisation that offers such a service at a reasonable rate?

We elected to use one I DigiCert's competitors who doesn't have this requirement, at least with our account type. (Entrust). That being said, we also have an auditor but that's outside my department.

I would add though, that my conversation with a DigiCert rep indicated that an internal, qualified individual could sign off. Basically anyone on staff aside from the person submitting the CSR with a relevant degree or a cert like CISSP.

[JonathanLydall]

We elected to use one I DigiCert's competitors who doesn't have this requirement, at least with our account type. (Entrust). That being said, we also have an auditor but that's outside my department.

We already had a non-EV code signing cert with about 9 months left which we wanted to upgrade and which they gave us credit for, so for at least this year, we've already paid for the certificate and can't use a competitor unless we essentially pay double the price.

I would add though, that my conversation with a DigiCert rep indicated that an internal, qualified individual could sign off. Basically anyone on staff aside from the person submitting the CSR with a relevant degree or a cert like CISSP.

Do you suppose a computer science or electrical engineering degree would be regarded as "relevant" enough for them?

Also, the process of getting the certificate from DigiCert into Azure KeyVault is not clear to me upfront, for example @clairernovotny said:

To get an x509 certificate that's rooted in the HSM, the certificate signing request (CSR) must be generated from Key Vault with the export option disabled. Then the CER file gets merged in and you have a key-pair that can be used for all the expected operations.

So that all sounds straightforward if you've done it before, but this will be the first time for us and I haven't seen anywhere that walks you through on how to do this. I am guessing it's done from clicking "Generate/Import" option under certificates, then selecting the right options (all guesses so far on my part):

[clairernovotny]

@garrett-wood is correct, any internal person with the relevant experience can attest to the proper usage and sign the form.

@JonathanLydall I'm not sure how to generate the correct CSR via the portal anymore as it's frequently getting updated.

I do recommend using a code signing service instead of the certificate directly, as it handles other file types (like nuget packages), orchestrates and logs everything.

This is what the .NET Foundation uses for its member projects and what I personally use. A number of other organizations have set it up as well: https://github.com/dotnet/signservice.

I understand that it might take a bit of setting up (unfortunately there's no "one click deploy" for something like this), but I'm happy to help and would always take doc update PR's that clarify things.

The signing service generates the CSR's and handles the merges back.

[garrett-wood]

@JonathanLydall While I can assist with this, your questions are indicating a lack of understanding in PKI as a whole. While this is totally understandable as it's a topic that is extremely complex and you should familiarize yourself with the core concepts to make sure you don't make a catastrophic mistake as that can and does happen especially in organizational PKIs. While doing this, you may want to consider whether an Elliptic Curve (EC) certificate better matches your requirements.

For your cert, all you need to do here is remove the cert type, set the EKU OID match only the one for Code Signing (1.3.6.1.5.5.7.3.3), remove the Key Encipherment Key Usage (ONLY have Digital Signature) and submit the generated CSR. The Certificate Type value is meaningless in this context and should be null. I would do something like the below, but you can use RSA-2048 too.

This is assuming the DigiCert integration with Key Vault still doesn't allow for Code Signing certs being requested through the API. We don't allow them to issue certificates to us anymore so I'm not really sure.

[JonathanLydall]

Thanks @clairernovotny and @garrett-wood. I'll do some more homework on the information you've both provided and come back here if I need more information.

My primary job is as a .NET and JS software developer, while I consider myself more knowledgeable than most in my position when it comes to PKI, I would not claim to have any kind of in depth knowledge on the details of elements of certificates. I only have to deal with PKI elements perhaps every few years.

Our use case is having our CI server sign our application and installer for download and meeting the requirements for Windows such that it doesn't show any scary security warnings, while at the same time ensuring we don't make any critical mistakes which would let a bad actor somehow get hold of our certificate. I would expect this use case is common enough that experts in the domain of PKI have easy enough to follow recommended best practices when it comes to the CSR.

[joseph-flinn]

@JonathanLydall I am just about to start migrating my company's current USB based EV cert into Azure's HSM backed Key Vault. Any gotchas that you have run across would be much appreciated.

[JonathanLydall]

@joseph-flinn, happy to offer what help I can. I think I have it sorted, just waiting for DigiCert to approve my submission from the 30th of December.

The process finally seems very straightforward to me, it is basically:

1. In Azure Key Vault, under Certificates, make a new CSR with the details as per @garrett-wood's comment (which I am very grateful for). In particular, ensuring that Exportable Private Key? is set to No.

2. Request an EV CS certificate from your certificate provider using the CSR downloaded from the Azure Portal.

3. Set up your CI environment to use Azure Sign Tool.

Step 3 had enough online documentation and was straightforward enough.

But for steps 1 and 2, annoyingly, DigiCert didn't offer any proper help nor do they seem to offer any guidance on doing the process with Azure Key Vault. If they had a support document outlining the process at least basically, that would have been very useful, instead, only through the help of people in this thread did I eventually come to understand it and which seems to have turned out to be as simple as I expected.

DigiCert seems to have a blanket policy that when you choose "other HSM", then they require you get an audit signed off by someone with an appropriate qualification, such as a CISSP. While this generally makes sense to me, I don't feel it should be needed for Azure Key Vault HSM which really should be well known to them and should just be an option on their list.

Google searching for individual's offering CISSP services was not generally fruitful, although I did find one company offering such services, but they quoted USD3000, which was outrageous for our company which is just a few employees and considering the amount work required which is nothing more than them confirming that we will do the CSR correctly in the Azure Portal.

Eventually it occurred to me to try on Fiverr. I found someone with a "Fiverr Pro" status who happened to be in the UK who I offered ~GBP250 for the job and they did it in a few hours, I looked up their CISSP number on the registry as a basic check of their credentials and it checked out. Maybe that's still overpaying, but it wasn't as outrageous as the other crowd.

Speaking as a software developer and with respect to qualified individual's in the security space, I feel that for our use case of wanting to digitally sign our software such that it no longer shows warnings in Windows, the process really should be very well known and simple for what should surely be an absurdly common use case.

The exact recommended parameters to use for the CSR should be well documented and then all that the certificate issuing authority should need to actually do is validate your identity. In regards to the HSM, I don't know for sure, but I would expect that one can see in the CSR whether or not the private key is exportable, and provided you state you will be using Azure Key Vault HSM, then that should also surely be enough for the certificate issuer.

[judge2020]

Speaking as a software developer and with respect to qualified individual's in the security space, I feel that for our use case of wanting to digitally sign our software such that it no longer shows warnings in Windows, the process really should be very well known and simple for what should surely be an absurdly common use case.

Yep, the whole code signing certificate market is one big racket. I'm sure MSFT is kicking themselves right now for distributing trust to different vendors when they could have done what Apple does and be the only place to get certs [if they didn't have existing partnerships and non-competes in place].

[joseph-flinn]

@JonathanLydall Thank you! Your overview provided a new perspective that has helped me better understand the research I have done so far.

[JonathanLydall]

So Digicert finally issued the certificate on Tuesday, but when trying to use AzureSignTool with it, an unhandled exception occurs:

trce: AzureSignTool.Program[0]
      Retrieving certificate digicert-ev-cs-intent-software.
trce: AzureSignTool.Program[0]
      Acquiring access token from client id
trce: AzureSignTool.Program[0]
      Acquired access token from client id
trce: AzureSignTool.Program[0]
      Retrieved certificate digicert-ev-cs-intent-software.
trce: AzureSignTool.Program[0]
      Creating context

Unhandled Exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.KeyVault.KeyVaultContext..ctor(KeyVaultClient client, KeyIdentifier keyIdentifier, X509Certificate2 publicCertificate) in C:\projects\rsakeyvaultprovider\RSAKeyVaultProvider\KeyVaultContext.cs:line 38
   at AzureSignTool.Program.<>c__DisplayClass0_1.<<Main>b__5>d.MoveNext() in D:\a\1\s\src\AzureSignTool\Program.cs:line 235
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at AzureSignTool.Program.Main(String[] args) in D:\a\1\s\src\AzureSignTool\Program.cs:line 306

I have been using AzureSignTool fine with our old non-EV certificate which I imported into KeyVault some time ago, but using the exact same command with just the -kvc being changed causes the above unhandled exception to occur.

The command I'm running is:

AzureSignTool
    "sign"
    "-v"
    "-du" "https://www.example.com"
    "-kvu" "https://<url>.vault.azure.net/"
    "-kvi" "***"
    "-kvs" "***"
    "-kvc" "digicert-ev-cs-intent-software"
    "<fullyQualifiedPathToExe>"

Possibly of significance is that strangely the details for the certificate show blank in the Azure Portal:

I opened a support case to Microsoft about it, but they have just replied saying its normal.

Can anyone offer any insight or suggestion on how to troubleshoot further?

[JonathanLydall]

Sorry, I should have searched before posting 🤦🏻‍♂️, I've found this comment: https://github.com/vcsjones/AzureSignTool/issues/36#issuecomment-744497228

@garrett-wood, I presume I could request the certificate be re-issued with a new CSR with a different supported key type, if so, what would you recommend?

[yowl]

Just adding my experience. I used SignService (https://github.com/dotnet/SignService) with a premium Azure KeyVault, EV code signing cert from Digicert, and a CISSP certified professional from fiverr.com and the process took a few weeks, but with help from @clairernovotny it all worked pretty well. Used to sign a .net core desktop app installed via AppInstaller. Thanks all for posting here.

[JonathanLydall]

To close the loop on this.

In the end I made a new CSR from the KeyVault with a key type of RSA-HSM and a key size of 2048 (which as far as I can tell is a pretty standard option for Authenticode code signing certificates).

DigiCert didn't require any additional checks to re-issue the certificate with the new CSR so it was quick to get the replacement. The newly issued RSA (2048 Bits) certificate then worked fine with AzureSignTool.

[clairernovotny]

That's surprising you were able to use a 2048 bit key as the baseline requirements are now 3072?

[JonathanLydall]

That's surprising you were able to use a 2048 bit key as the baseline requirements are now 3072?

I did it in January, so it's not really surprising to me that the requirements have increased since then.

After being mis-directed by a different user here to use the EC curve cipher despite it being unsupported by AzureSignTool, I was honestly quite frustrated at the huge delay it caused us and just chose the "default" recommendation at the time of DigiCert with the 2048 key with the hope that we would be up and running as soon as possible, and we were.

Azure KeyVault offers both 3072 and 4096 and I would imagine that AzureSignTool supports it too, so I don't see there being any complications getting it renewed next year.

Between DigiCert not having a simple guide on using Azure KeyVault, the other user here sending me in the wrong direction and AzureSignTool not more clearly explaining that the certificate's cipher was unsupported, this whole process seemed unnecessarily annoying. And like I said before, this is not an unusual use case, in fact I can't think of a more common one for code signing. I think it says something of the security industry that they can't make such a common use case more accessible to first time users.