search

kiwix / overview

:balloon: Start here for current projects, how to get involved with offline projects and joining community calls. A resource for new and veteran members
87 stars 14 forks source link

Code Signing solution for our CD #88

Open kelson42 opened 1 year ago

kelson42 commented 1 year ago

It won’t be allowed anymore to store private keys in files to sign code in the future. We need tomfind an HSM based solution.

kelson42 commented 1 year ago

https://www.ssl.com/guide/esigner-pricing-for-code-signing/

kelson42 commented 1 year ago

Considering the prices, I wonder if we should not consider to host this ourself:

rgaudin commented 1 year ago

It's clear to me that either a reasonable SaaS will develop (or get known better! might already exist) or others will follow this lower-cost approach.

My fear is that with our limited resources, we'd build a solution that's good-enough for us but would soon be shadowed by another, better solution that would receive more resources and spotlight and we'd have just wasted some resources.

Can we say that we wait until the end of 2023 for something to emerge and if by then there's still nothing, we can be build it ourselves? That would leave us 6+ months before the expiration of our current certificate.

kelson42 commented 1 year ago

@rgaudin yes

kelson42 commented 8 months ago

I might be able to find a free network HSM.

benoit74 commented 1 month ago

On-prem alternative

I suppose we cannot get a free network HSM but want to consider to build a "low-cost" solution on-premise

Solution is based on cheap Yubico modules, a certificate buyed at signmycode.com (https://signmycode.com/windows-code-signing) and a mini computer running at home.

Yubico modules seems to have proper integration with signtools from Microsoft: https://support.yubico.com/hc/en-us/articles/360016614840-Code-Signing-with-the-YubiKey-on-Windows

However it seems that this uses the SmartCard feature which requires you to enter a PIN at every signing request: https://signmycode.com/resources/how-to-use-yubikey-for-code-signing-purpose

⚠️ Last point seems to be confirmed, and would hence be a nogo for this alternative ; I keep information below for reference purposes should this point be invalidated, documentation is not that clear on this point ⚠️

CI changes

Simplest solution is probably to consider the local machine as a Github Self-hosted runner.

Then we can either

Budget

Nota: lifetime durations are pessimistic, based on the fact that the probability hardware might get obsolete in 5 years is not negligible

rgaudin commented 1 month ago

which requires you to enter a PIN at every signing request

Isn't this PIN just a secret passed on the command-line? That's what it looks like on that link.

Simplest solution is probably to consider the local machine as a Github Self-hosted runner.

/!\ a Self-hosted runner is very different from github-hosted: you don't have access to the github image so you must prepare and maintain one yourself: OS, software stack, etc. This means it's not a drop-in replacement (so no easy toggle between the two unless you've managed to replicate the GH hosted one). You also have to manage disk and all as the whole lifecycle is then on your hands.

I was kind of expecting a rough estimate of work necessary to go this way as this is important in this scenario.

I also doubt that a $300 machine would satisfy a scenario where we build everything on it

rgaudin commented 1 month ago

Ticket's context:

We sign Windows softwares to prevent annoying/scrary popups when launched on Windows. What exactly happens depends on several factors: the Windows version, whether software is signed, certificate type if signed and Microsoft's own reputation score for that very binary.

Roughly, if software is not signed, you get the scariest popups. If it's signed with an OV certificate, your only get one screen informing that software is from XXX (the cert name) and you confirm it's OK. If it's signed with an EV certificate, you're good to go from day 1.

Note: In March 2024, Microsoft changed the way MS SmartScreen interacts with EV Code Signing certificates. EV Code Signing certificates remain the highest trust certificates available, but they no longer instantly remove SmartScreen warnings.

Due to this new requirement, we cannot simply renew our certificate and must thus switch to a new one which private key will have to be stored on an HSM.

The main two options we have is to use a fully managed service that has the least impact on our current operations or to use our own hardware dongle (there are degrees here obviously).

Fully managed solution

There might be other providers but I've limited myself to those I could find (!), that seem trustworthy (CA must be in Microsoft Authenticode list) and for which I could find details on cost and how to sign with.

From what I understand, those Cloud Signing is always offered in two flavors: via a custom, provider-specific CLI tool and via a Microsoft CNG Plugin which allows tools such as SignTool (what we use) to use the Cloud Signing feature via the CSC-compliant API. In both cases, I believe only hashes are sent to the API for signing.

Digicert

Cost:

Item Cost
3 years certificate $1,650
KeyLocker (Cloud feature) $250
Setup & update all workflows 2d
Total 3y $1,900 + 2d
Limit 1,000 signs per certificate (1 credential)

Additional signatures can be purchased by increments of 1,000. Price could not be found.

Signing doc

SSL.com

Item Cost
3 years certificate $330
Loaded Yubikey (both cloud and key can sign) $280
Tier-1 Signing sub $180/y
Setup & update all workflows 2d
Total 3y $1,150 + 2d
Limit 240 signs per year (1 credential)

Additional signatures at $1 each for Tier-1

Signing doc Workflow doc

GoGetSSL

Item Cost
3 years certificate $1,120
Setup & update all workflows 2d
Total 3y $1,120 + 2d
Limit 1,000 signs (1 credential)

Certificate on device

Using an FIPS 140-2-L2 device to store the certificate and sign is an option but it seems to always require a human intervation: entering a PIN on a System Popup (Windows) or a phsical touch of the device.

I could not find devices that match FIPS requirements and don't enforce MFA.

Compatible YubiKeys Code Signing with a YubiKey Some doc that says

Nevertheless, you will be forced [to enter PIN] whenever you run singtool.exe – it’s unavoidable.

Seems incompatible with our workflow.

Still requires buying the certificate ; depending on providers you can install easily or not on your own hardware or have the provider ship you a device with the certificate on it.

Cheapest cert + device for 3y: $880 from Sectigo

HSM

Contrary to hardware key store, an HSM has crypto capabilities that means the private key for our cert will be generated and stored on the HSM.

signing using YubiHSM

From what I understand from this doc, we'd need the HSM to be connected to sign but signing would not require any interactions. ⚠️ Whether it requires interaction on restarts is unknown but that can probably be dealt with.

For this scenario to work, we'd need:

Item Cost
Computer $1,200
YubiHSM 2 FIPS $1,250
Initial Development 10d
Maintenance 1d/y
Certificate (3y - Comodo) $650
Always-on, always-connected -
Total 3y $3,100 + 10d
Provider 3y certificate
GoGetSSL $585
Comodo/Sectigo OV 634
Comodo/Sectigo EV $834
Digicert OV 1,124
[Digicert OV from Digicert]() $1,650

Note: CodeSigningStore is a Digicert sub that sells cert (from Digicert and others) at lower prices…

The deployment/development I imagined:

This scenario is not optimized in that binaries and uploaded from GH to that machine, signed locally and then downloaded in GH again. The advantage is that this is very simple to implement and probably very resilient. An hash based version could be implemented in a second phase.

Note: All examples are 3years but certificates can be acquired for one or two years as well (more than 3y is rare).

benoit74 commented 1 month ago

Just FYI, it is possible to sign Windows binaries on Linux machines with https://github.com/mtrojnar/osslsigncode

Looks it could even be used to generate the payload to sign, send it to whatever will sign it and then include the signature in the exe.

rgaudin commented 1 month ago

We've decided to go with SSL.com Cloud Signing for coming year and keep that ticket open.

Popolechien commented 6 days ago

Just to be clear, the Tier-2 Annual Cloud bundle is $63.75 per month so the total cost is more along the lines of $129 for the certificate + $701.25 for the cloud bundle (first month is free). @rgaudin is this still ok?

benoit74 commented 5 days ago

I think that this misunderstanding on price can change our decision on which provider to choose, let's check it before buying anything, we are not in a rush.