A lot of reading of Google Cloud KMS keyrings and keys during signing

[rvadim]

STR:

1. Generate asymmetric key and CA

   gcloud kms keyrings create test --location <region>```
   gcloud kms keys create my-key --keyring test --location <region> --purpose "asymmetric-signing" --default-algorithm "rsa-sign-pkcs1-2048-sha256"
   openssl genrsa -out ca.key 2048

2. Create CSR and Certificate

   git clone https://github.com/mattes/google-cloud-kms-csr
   cd google-cloud-kms-csr
   go build -o csr
   ./csr -key $(gcloud kms keys versions list  --key my-key --keyring test --location=<region>) -out my.csr --common-name MyOrg
   openssl -req -in my.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out my.crt

3. Run jsign

   for i in {1..1000}; do 
   jsign --storetype GOOGLECLOUD
   --storepass "$(gcloud auth print-access-token)"
   --keystore "projects/<project>/locations/<location>/keyRings/test"
   --alias "my-key"
   --certfile my.crt
   my.exe
   done

Expected result: Binary signed

Actual result:

java.io.IOException: 429 - RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'cloudkms.googleapis.com' for consumer 'project_number:<project-id>

So, each run of jsign makes 2 reading and 1 crypto operation. But default quotas are:

• Cryptographic requests per minute: 60,000
• Read requests per minute: 300

Also jsign require a lot of permissions for one signing For key:

cloudkms.cryptoKeyVersions.list
cloudkms.cryptoKeyVersions.useToSign
cloudkms.locations.get
cloudkms.locations.list
resourcemanager.projects.get

For keyring:

cloudkms.cryptoKeys.list

Actually only one permission required cloudkms.cryptoKeyVersions.useToSign and only 2 parameters to run key.id (example: projects/<project id>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key name>/cryptoKeyVersions/<version> and algorithm (example: RSA).

I have 2 suggestions:

• Cache keys list and keys versions list between runs of jsign
• Add ability to specify only key.id and algorithm

I will try to implement second.

[ebourg]

Interesting, I wasn't aware of the request limit. I can add a cache in GoogleCloudSigningService, but that won't work for repeated calls from the command line. Out of curiosity, how many files are you signing usually?

[ebourg]

I've made a couple of changes that should improve the situation:

• you can now specify multiple files from the command line (jsign [OPTIONS] file1.exe file2.exe ...)
• the private keys is now cached (but not persisted in an external file)

So if you sign n files this results in two requests to fetch the private key + n signing requests (instead of 3*n requests previously).

[rvadim]

Out of curiosity, how many files are you signing usually?

More then 100k per month ( Near 1000 files per pipeline, few piplines per day.

I've made a couple of changes that should improve the situation:

Wow! Thank you for such fast response. It's good feature actually. I will think how to use it in our pipelines)

[ebourg]

More then 100k per month ( Near 1000 files per pipeline, few piplines per day.

Impressive, you could consider buying your own HSM at this point ;)

I've added another tweak, you can now append the key algorithm to the key alias (only if the version is specified), this saves one more request: --alias mykey/cryptoKeyVersions/2:ECDSA

There is still one extra request to fetch all the keys available that could be removed.

[ebourg]

The request to get all the keys has been removed, there is now only one request per file signed.

I'd still recommend signing multiple files with a single invocation of jsign from the command line, it should be slightly faster.

[rvadim]

Impressive, you could consider buying your own HSM at this point ;)

Yes) We are thinking about it, but for now we will use Cloud HSM for a reason.

I've added another tweak, you can now append the key algorithm to the key alias (only if the version is specified), this saves one more request: --alias mykey/cryptoKeyVersions/2:ECDSA

The request to get all the keys has been removed, there is now only one request per file signed. I'd still recommend signing multiple files with a single invocation of jsign from the command line, it should be slightly faster.

You have greatly accelerated our work! Thank you very much. We will do performance testing and I will share the results here.

[rvadim]

2 peaks:

• 100 files (1Mb)
• 200 files (1Mb)

We don't have reads at all! Now we are limited only by the number of cryptographic operations per region. I will also try to use bulk signing.

Thank you again :+1:

[ebourg]

Nice! Thank you for torturing testing Jsign thoroughly.