Signing Windows exe with Asymmetric (signing) key - mechanism not permitted

[outrunthewolf]

Hi I'm trying to sign a windows exe file using the pkcs11 module and a key in googles KMS.

osslsigncode sign \
  -pkcs11engine /usr/lib/x86_64-linux-gnu/engines-3/pkcs11.so \
  -pkcs11module /tmp/libkmsp11-1.3-linux-amd64/libkmsp11.so \
  -certs my.crt \
  -key "pkcs11:object=hsm-generated-key;type=private" \
  -in my-app.exe \
  -out my-app-signed.exe

Everything seems to work nicely, and I get the correct key, but I end up with "Mechanism not permitted".

I20240220 14:58:41.247017  9025 logging.cc:177] returning 0x70 from C_EncryptInit due to status INVALID_ARGUMENT: at crypter_ops.cc:79: mechanism 0x1 is not valid for operation encrypt [type.googleapis.com/kmsp11.StatusDetails='CK_RV=0x70']
PKCS7_dataFinal failed
Failed to sign spcIndirectDataContent
Failed to set signed content
Unable to prepare new signature

My guess is it's trying to encrypt with an asymmetric signing key. My assumption was a signing wouldn't require encryption. Have I made a terrible mistake?

Here is a version dump of osslsigncode:

osslsigncode 2.8-dev, using:
    OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
    libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Default -CAfile location: /etc/ssl/certs/ca-certificates.crt

Many thanks

[mtrojnar]

Did you get this key working with signtool?

I guess https://github.com/OpenSC/libp11/ may be a better place to report this.

https://stackoverflow.com/questions/75625385/google-ksm-does-not-sign-rsa-pkcs-v1-5-through-openssl-pkcs11-with-mechanism seems to address a related issue.

[outrunthewolf]

Thank you for the response. I'm closing this as user error, I generated a different key and it worked fine.