Problems with GPG

[oajara]

Hi! First of all, thank you very much for this awesome work!

I'm having issues using GPG with this module. Working with a RSA_4096 key, I'm able to create the certificate and the GPG Agent is able to discover it:

OK Pleased to meet you
SCD LEARN
gpg-agent[68961]: no running SCdaemon - starting it
gpg-agent[68961]: DBG: first connection to SCdaemon established
S SERIALNO D2760001240111503131233D17681111
S APPTYPE PKCS11
S KEY-FRIEDNLY 97AF68BD18212B1E98A5FAA8B562EDA8A3DD6ECC /CN=oscar.test on my-signing-key
S CERTINFO 101 aws_kms/0/0/my\x2Dsigning\x2Dkey/39613163393832652D356365662D343238662D623766392D373862343763616462633266
S KEYPAIRINFO 97AF68BD18212B1E98A5FAA8B562EDA8A3DD6ECC aws_kms/0/0/my\x2Dsigning\x2Dkey/39613163393832652D356365662D343238662D623766392D373862343763616462633266
OK

However, when I try to gpg --expert --full-generate-key it fails in the last step:

...
GnuPG needs to construct a user ID to identify your key.

Real name: my-signing-key
Email address: 
Comment: 
You selected this USER-ID:
    "my-signing-key"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
gpg: signing failed: Card error
gpg: make_keysig_packet failed: Card error
Key generation failed: Card error

Here is the related part in the GPG Agent logs:

DBG: chan_11 <- OK
detected card with S/N D2760001240111503131233D17681111
DBG: encoded hash: 30 51 30 0D 06 09 60 86 48 01 65 03 04 02 03 05 00 04 40 37 8D 27 23 04 74 FE E8 F3 88 2C A0 75 75 D0 DE 0F D0 1C 39 7E FA 5A 68 B0 0A 19 60 C0 02 AA 05 B7 74 F5 52 93 AB BF 11 56 01 43 70 FC 21 9B 34 0B 03 04 24 B7 2E D3 62 9C 84 16 1C D4 37 68 0A
DBG: chan_11 -> SETDATA 3051300D060960864801650304020305000440378D27230474FEE8F3882CA07575D0DE0FD01C397EFA5A68B00A1960C002AA05B774F55293ABBF1156014370FC219B340B030424B72ED3629C84161CD437680A
DBG: chan_11 <- OK
DBG: chan_11 -> PKSIGN --hash=sha512 aws_kms/0/0/my\x2Dsigning\x2Dkey/39613163393832652D356365662D343238662D623766392D373862343763616462633266
DBG: agent_cache_housekeeping
DBG: chan_11 <- ERR 108 Card error <Unspecified source>
smartcard signing failed: Card error
command 'PKSIGN' failed: Card error
DBG: chan_10 -> ERR 67108972 Card error <GPG Agent>
DBG: chan_10 <- [eof]

And here is a log section of the gnupg-pkcs11-scd I think it's related:

PKCS#11: _pkcs11h_session_findObjects return rv=0-'CKR_OK', *p_objects_found=0
PKCS#11: _pkcs11h_session_getObjectById return rv=512-'CKR_FUNCTION_REJECTED', *p_handle=ffffffffffffffff
PKCS#11: _pkcs11h_certificate_resetSession return rv=512-'CKR_FUNCTION_REJECTED'
PKCS#11: __pkcs11h_certificate_getKeyAttributes return rv=512-'CKR_FUNCTION_REJECTED'
PKCS#11: pkcs11h_certificate_signAny return rv=512-'CKR_FUNCTION_REJECTED', *p_target_size=000000000000007f
PKCS#11: pkcs11h_certificate_freeCertificate entry certificate=0x55a827611660
PKCS#11: _pkcs11h_session_release entry session=0x55a827617190
PKCS#11: _pkcs11h_session_release return rv=0-'CKR_OK'

I attach the full logs for both, GPG Agent and gnupg-pkcs11-scd

gpg-agent.log gnupg-pkcs11-scd.log

I'm working on an Ubuntu 22.04 system with:

• gpg (GnuPG) 2.2.27
• gpg-agent (GnuPG) 2.2.27
• libgcrypt 1.9.4
• gnupg-pkcs11-scd 0.9.2

Thanks!

[bcressey]

The original guide I followed, Using PKCS#11 Token With GPG, mentions this:

As of Feb, 2022, the following does not work for me with gnupg-pkcs11-scd-0.9.2-6.fc35. So I compiled gnupg-pkcs11-scd-0.10.0-1.fedorap11.fc35 myself.

So you may want to try using gnupg-pkcs11-scd 0.10.0 instead, and see if that works. I've only tested with a Fedora 36 base image, which has the newer version.

[oajara]

It worked in Fedora 36 after a few tweaks. I had to install (downgrade to) openssl1.1 and also the libjson-c.so.4 library version.

Now, I have a question if I may. My understanding was that the gpg --expert --full-generate-key created a stub private key without private key material and that the signing operations were made using signing calls to the KMS service through the PKCS11 module. However, this does not seem to be the case since I can create signatures with the imported key without even having AWS credentials in place. What am I missing here?

[bcressey]

That's a great - and terrifying! - question.

My first guess is that perhaps there are ambient AWS creds available; if you have a default profile in ~/.aws/config or you're running on an EC2 instance with an instance role, the provider would automatically try to use those. That seems unlikely but good to rule out.

If that's not it, my second guess is that something about your gpg --expert --full-generate-key workflow might be generating a new private key, and not actually using the provider as expected. Can you share the specific steps so I can try to repro?

In my own local setup, running on an EC2 instance with role creds, if I replace the kms:Sign in the role with something else, like kms:UpdateAlias, then I get this error when I run these commands:

$ gpg --import ~/.gnupg/signing.gpg
$ gpg --card-status
$ gpg --detach-sign "${BOOT_MOUNT}/grub/grub.cfg"
gpg: signing failed: Card error
gpg: signing failed: Card error

If I omit the gpg --card-status step, then I get:

$ gpg --import ~/.gnupg/signing.gpg
$ gpg --detach-sign "${BOOT_MOUNT}/grub/grub.cfg"
gpg: no default secret key: No secret key
gpg: signing failed: No secret key

That leads to a third guess which is that you might have a "default secret key" that gpg is using instead.

[oajara]

So, while creating a recipe to reproduce the error it worked as expected! For the record, I will share the bogus process that led me to the confusion:

• Since I work with a multi-account AWS environment and also for security reasons I use aws-vault to manage AWS CLI access. This tool leverages the AWS STS service and generates environment variables with temporary session tokens.
• I was running these aws-kms-pkcs11 module tests from a docker container. I granted AWS CLI access in the container by pasting the needed AWS environment variables that I got using aws-vault in my workstation.
• At this point I could do the key import and signing using GPG and the module.
• Then, I "disabled" the AWS CLI access by unsetting all AWS related environment variables. At this point any AWS CLI operation would return Unable to locate credentials. You can configure credentials by running "aws configure". and that was the reasons I considered the signing shouldn't be allowed anymore. However, the signing still worked normally for some reason.
• Revoking the session from the AWS IAM service actually made the signing process fail returning gpg: signing failed: Card error as expected.

However, using plain AWS CLI profiles configuration (with credentials stored in ~/.aws/) worked as expected. Deleting the ~/.aws/ content would immediately result in signing errors. This behavior makes me think the agent process has it's own copy of the environment variables, therefore unsetting the environment variables after the agent processes are created does not affect the AWS access needed to sign.

Back to this observation:

It worked in Fedora 36 after a few tweaks. I had to install (downgrade to) openssl1.1 and also the libjson-c.so.4 library version.

Did you have to do these tweaks in your Fedora 36 testing environment as well?

[bcressey]

Did you have to do these tweaks in your Fedora 36 testing environment as well?

I didn't need to downgrade any packages.

For reference, my environment is now published as the newest versions of the Bottlerocket SDK:

public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.27.0
public.ecr.aws/bottlerocket/bottlerocket-sdk-aarch64:v0.27.0

The relevant packages are:

openssl-libs-3.0.5-1.fc36.x86_64
openssl-devel-3.0.5-1.fc36.x86_64
openssl-pkcs11-0.4.12-2.fc36.x86_64
json-c-0.15-3.fc36.x86_64
json-c-devel-0.15-3.fc36.x86_64

The full Dockerfile for this is pretty unwieldy - it's a very large container! - but the relevant build steps can be found in https://github.com/bottlerocket-os/bottlerocket-sdk/commit/d9fc285af739e042af23f7ece2411d0c4a0ed2c2, in case that's helpful.

[oajara]

Thank you very much for your help @bcressey!

[JackOfMostTrades]

Thanks for all the help @bcressey ! Is there anything from this thread we could/should capture in the docs to help folks in the future?

[bcressey]

Seems like the usage notes could clarify that gnupg-pkcs11-scd 0.10.0+ is required, though I'm not sure if that was ultimately what resolved the issue.

@oajara do you have any thoughts on what might've helped?

[oajara]

@bcressey gnupg-pkcs11-scd 0.10.0 did not fix the issue in Ubuntu 22.04. I still got the same error at the final step of gpg --expert --full-generate-key. I switched to a Fedora 36 environment and that problem did not happen.

One thing that might be worth putting here is that if one is using the aws_kms_pkcs11.x86_64.so v0.0.9 file published in the Github release the module will look for the files libcrypto.so.1.1 and libjson-c.so.4. In Fedora 36 the package openssl1.1 provides libcrypto.so.1.1. On the other side the json-c package available is json-c-0.15-3 which provides /usr/lib64/libjson-c.so.5, therefore my hotfix was creating a symlink /usr/lib64/libjson-c.so.4 -> /usr/lib64/libjson-c.so.5.

[bcressey]

For that category of issues, I am not sure there's a better solution than packaging aws-kms-pkcs11 for the various distros.

[JackOfMostTrades]

Thanks folks! I added a couple of notes to the gpg signing docs.