Obtain .pem file for private key using softhsm2-dump-object

[tarruda]

Hi

I would like to export a private key stored in a .object file. I don't understand the format used used by SoftHSM to store private keys, but it seems to be possible to obtain all the relevant information using softhsm2-dump-object. Here's what I did after initializing an empty token:

# generate a keypair
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --keypairgen --key-type rsa:512
# obtain the private key data, it is stored in "a2b0d15f..." object:
softhsm2-dump-file /var/lib/softhsm/tokens/810b3ea3-1b94-c997-d83c-e883739b455a/a2b0d15f-6a48-be50-e515-fe005be5a236.object

The above prints all relevant information about the private key to stdout. Do you know how can I convert that to a "PEM" file for using with openssl?

[rijswijk]

You should consider the internal format SoftHSM uses as opaque. The best way to export keys is to use the API, and then to use C_Wrap(..), which will export in industry-standard PKCS #8 format. You may want to search for tools that implement PKCS #11 and can do this for you. SoftHSM doesn't really come with tools to export keys directly, that is not how SoftHSM is meant to be used, if you want a key pair that you can use with OpenSSL, you could just generate it with OpenSSL directly?

[tarruda]

SoftHSM doesn't really come with tools to export keys directly, that is not how SoftHSM is meant to be used, if you want a key pair that you can use with OpenSSL, you could just generate it with OpenSSL directly?

I will try to give you a short summary: In Brazil citizens can buy digital certificates which can be used to sign documents that have legal validity (and do other kinds of tasks that would otherwise require us physically go to government offices). There are two types of certificates that can be bought:

• A1 certificates, which are stored as files in your computer. Valid for 1 year
• A3 certificates, which are stored in smart cards or cryptographic tokens. Valid for 3 years.

Both certificates are just SSL certificates, that only difference is the storage medium and expiration date.

Since I'm not a high value target, I'm perfectly happy with having certificates stored on files, where I can easily manage my secure encrypted backups. In other words, I'm more worried about losing the certificate (which will happen if my cryptographic token is stolen, lost or damaged) than being attacked by someone looking to steal my digital identity. I'm just a normal citizen looking for a convenient way to manage my citizenship (without going through the nightmare of dealing with government employees when I need to fix my tax information, for example).

Normally I would simply buy an A1 certificate and be done with it, however the process of renewing certificates is by itself a bit troublesome, not to mention it ends up being more expensive: I would spend more buying an A1 certificate and renewing 2 times than buying the A3 version. For that reason I'm going to buy an A3 certificate which only has to be renewed after 3 years, but I'm also not happy with storing a single copy of the certificate in a write-only cryptographic token.

This is where SoftHSM comes in. After I go through a physical validation process where they check documentation, the certificate is emitted via a web browser extension that writes to the token via pkcs11 API. The extension communicates with a native component that can specify a pkcs11 module that will handle the token communication. My plan is to use SoftHSM pkcs11 module as a fake token where I can extract the private key.

You should consider the internal format SoftHSM uses as opaque. The best way to export keys is to use the API, and then to use C_Wrap(..), which will export in industry-standard PKCS #8 format

Can you advise on how to do that? I assumed private keys were not exportable via pkcs11 API.

You should consider the internal format SoftHSM uses as opaque

As you can see from my use case, I'm fine with SoftHSM format being opaque since I will only extract the private key once.

[Aearsis]

I assumed private keys were not exportable via pkcs11 API.

It depends on how you create them. But the CA process you describe might verify the key attributes, e.g. that the key is not extractable (CKA_EXTRACTABLE = 0), or even that it was never extractable (CKA_NEVER_EXTRACTABLE = 1). If this is the case, it might be the easiest to extract the key first, then alter the database to pretend it was never extractable.

For calling C_Wrap, you might need to write a simple program / script. SoftHSM is just a library with standardized interface, you won't find any tools for using that interface here. And from the open source tools I know, none of them supports wrapping keys.

But you don't really need a file with your key, do you? For the purposes of backup, just store the whole token (directory 810b3ea3-1b94-c997-d83c-e883739b455a in your case). When you copy it to the tokens directory of a future installation, your token will be recovered (provided you remember the PIN from the time of backing up.) If it was for more than 3 years, I would make a backup of the SoftHSM installer and configuration as well - even if future SoftHSM changed the storage format without backwards compatibility (as a result of a mistake, probably), you will still have acces to your private key, allowing you to extract it later. The storage format is opaque, but the source code is open :)

[tarruda]

It depends on how you create them. But the CA process you describe might verify the key attributes, e.g. that the key is not extractable (CKA_EXTRACTABLE = 0), or even that it was never extractable (CKA_NEVER_EXTRACTABLE = 1). If this is the case, it might be the easiest to extract the key first, then alter the database to pretend it was never extractable.

I found a tool that allows extracting private keys that have CKA_EXTRACTABLE (https://github.com/EverTrust/pkcs11-keyextractor) and adapted into a python-pkcs11 script which does the same in a temporary softhsm2 directory:

#!/usr/bin/python3
# based on https://github.com/EverTrust/pkcs11-keyextractor/blob/master/src/main/scala/fr/itassets/p11/PKCS11KeyExtractor.scala
import pathlib
import subprocess
import os

import pkcs11
from pkcs11.mechanisms import Mechanism, KeyType
from pkcs11.constants import Attribute, ObjectClass

TOKEN_LABEL = 'test-extract'
KEYPAIR_LABEL = 'test-extract-keypair'

def setup_softhsm():
    softhsm_dir = pathlib.Path('/tmp/test-softhsm-key-export')
    softhsm_config = softhsm_dir / 'softhsm2.conf'
    token_dir = softhsm_dir / 'tokens'
    os.environ['SOFTHSM2_CONF'] = str(softhsm_config)
    if token_dir.exists():
        return
    token_dir.mkdir(parents=True)
    softhsm_config.write_text(f'directories.tokendir = {token_dir}\n')
    subprocess.check_call(['softhsm2-util', '--init-token', '--slot', '0',
        '--label', TOKEN_LABEL, '--so-pin', '1234', '--pin', '1234'])

def destroy_test_rsa_keypair(session):
    for key in session.get_objects({Attribute.LABEL: KEYPAIR_LABEL}):
        key.destroy()

def generate_test_rsa_keypair(session):
    session.generate_keypair(KeyType.RSA, 2048,
        label=KEYPAIR_LABEL, store=True, private_template={
            Attribute.EXTRACTABLE: True
        })

def generate_wrapping_key(session):
    template = {
        Attribute.ENCRYPT: True,
        Attribute.DECRYPT: True,
        Attribute.PRIVATE: True,
        Attribute.SENSITIVE: True,
        Attribute.EXTRACTABLE: True,
        Attribute.WRAP: True,
        Attribute.TOKEN: False
    }
    return session.generate_key(KeyType.AES, key_length=128,
        mechanism=Mechanism.AES_KEY_GEN, template=template)

def main():
    setup_softhsm()
    lib = pkcs11.lib('/usr/lib/softhsm/libsofthsm2.so')
    token = lib.get_token(token_label=TOKEN_LABEL)

    with token.open(user_pin='1234', rw=True) as session:
        destroy_test_rsa_keypair(session)
        generate_test_rsa_keypair(session)
        wrapping_key = generate_wrapping_key(session)
        for pkey in session.get_objects({
                Attribute.LABEL: KEYPAIR_LABEL,
                Attribute.EXTRACTABLE: True
            }):
            print('Extracting RSA private key private key:', pkey)
            wrapped_key = wrapping_key.wrap_key(pkey)
            print('wrapped key:', wrapped_key)

if __name__ == '__main__':
    main()

I'm facing a CKR_KEY_SIZE_RANGE in the line which tries to wrap the key. According to https://www.oasis-open.org/committees/download.php/50733/pkcs11-base-v2%2040-EC-error-pt3.pdf:

cannot be wrapped with the specified wrapping key and mechanism solely because of its length, then C_WrapKeyfails with error code CKR_KEY_SIZE_RANGE.

Any ideas?

But you don't really need a file with your key, do you? For the purposes of backup, just store the whole token (directory 810b3ea3-1b94-c997-d83c-e883739b455a in your case). When you copy it to the tokens directory of a future installation, your token will be recovered (provided you remember the PIN from the time of backing up.)

True, at first I wouldn't need to export and could just keep using SoftHSM. But I prefer to have the key exported in a more standard format, especially because I plan to import into a hardware token for daily use (after making a backup).

A side question: I noticed that softhsm2-dump-file shows the components of the private key, such as modulus/exponents/primes. Are those values in plain text or is it encrypted using the PIN? If it is in plain text, I guess it would be wise to use an encrypted directory as softhsm2 token storage.

[rijswijk]

I'm facing a CKR_KEY_SIZE_RANGE in the line which tries to wrap the key. According to https://www.oasis-open.org/committees/download.php/50733/pkcs11-base-v2%2040-EC-error-pt3.pdf:

cannot be wrapped with the specified wrapping key and mechanism solely because of its length, then C_WrapKeyfails with error code CKR_KEY_SIZE_RANGE.

Any ideas?

You're probably using the incorrect wrapping mechanism. Have you tried using GNUTLS's p11tool to do the job?

But you don't really need a file with your key, do you? For the purposes of backup, just store the whole token (directory 810b3ea3-1b94-c997-d83c-e883739b455a in your case). When you copy it to the tokens directory of a future installation, your token will be recovered (provided you remember the PIN from the time of backing up.)

True, at first I wouldn't need to export and could just keep using SoftHSM. But I prefer to have the key exported in a more standard format, especially because I plan to import into a hardware token for daily use (after making a backup).

Using the source code, you could always attempt to reconstruct the key yourself, but that would be a lot of work.

A side question: I noticed that softhsm2-dump-file shows the components of the private key, such as modulus/exponents/primes. Are those values in plain text or is it encrypted using the PIN? If it is in plain text, I guess it would be wise to use an encrypted directory as softhsm2 token storage.

If the object is private, it is stored in encrypted form, and the output from the dump utility (which is really only a debugging tool) will output the encrypted data.

[tarruda]

I've managed to extract the key by wrapping with with CKA_AES_KEY_WRAP_PAD mechanism, since SoftHSM does not allow this mechanism for decryption, I had to implement in my script. I will leave it here in case anyone needs in the future:

#!/usr/bin/python3
# based on https://github.com/EverTrust/pkcs11-keyextractor/blob/master/src/main/scala/fr/itassets/p11/PKCS11KeyExtractor.scala
import pathlib
import subprocess
import os
import struct
import subprocess

import pkcs11
from pkcs11.mechanisms import Mechanism, KeyType
from pkcs11.constants import Attribute, ObjectClass, MechanismFlag
from Crypto.Cipher import AES

MODULE_PATH = '/usr/lib/softhsm/libsofthsm2.so'
TOKEN_LABEL = 'test-extract'
KEYPAIR_LABEL = 'test-extract-keypair'
PIN = '1234'
KEK = os.urandom(16)
SAMPLE_TEXT = 'key extracted!\n'

def setup_softhsm():
    softhsm_dir = pathlib.Path('/tmp/test-softhsm-key-export')
    softhsm_config = softhsm_dir / 'softhsm2.conf'
    token_dir = softhsm_dir / 'tokens'
    os.environ['SOFTHSM2_CONF'] = str(softhsm_config)
    if token_dir.exists():
        return
    token_dir.mkdir(parents=True)
    softhsm_config.write_text(f'directories.tokendir = {token_dir}\n')
    subprocess.check_call(['softhsm2-util', '--init-token', '--slot', '0',
        '--label', TOKEN_LABEL, '--so-pin', PIN, '--pin', PIN])

def destroy_test_rsa_keypair(session):
    for key in session.get_objects({Attribute.LABEL: KEYPAIR_LABEL}):
        key.destroy()

def encrypt_sample_text(session, exported_key):
    pubkey = list(session.get_objects({
        Attribute.LABEL: KEYPAIR_LABEL
    }))[0]
    sample_ciphertext = pubkey.encrypt(SAMPLE_TEXT)
    inkey = '/tmp/test-softhsm-key-export/sample.der'
    with open(inkey, 'wb') as f:
        f.write(exported_key)
    openssl = subprocess.Popen(['openssl', 'rsautl', '-decrypt', '-inkey',
        inkey, '-keyform', 'DER', '-oaep'], stdin=subprocess.PIPE)
    openssl.communicate(sample_ciphertext)

def generate_test_rsa_keypair(session):
    # env = os.environ.copy()
    # env['PKCS11_PIN'] = PIN
    # subprocess.check_call(['pkcs11-tool', '--module', MODULE_PATH, '--login',
    #     '--pin', 'env:PKCS11_PIN', '--keypairgen', '--key-type', 'rsa:512'],
    #     env=env)
    session.generate_keypair(KeyType.RSA, 512,
        mechanism=Mechanism.RSA_PKCS_KEY_PAIR_GEN,
        label=KEYPAIR_LABEL, store=True, 
        public_template={
        },
        private_template={
            Attribute.SENSITIVE: False,
            Attribute.EXTRACTABLE: True,
        })

# Implementation of rfc5649. This doesn't account for the case of ciphertext
# with 2 blocks (since we are using it to export RSA private keys). While
def unwrap_key(kek, ciphertext):
    # use 6 steps of AES ECB 
    decrypt = AES.new(kek, AES.MODE_ECB).decrypt
    steps = 6
    # split the ciphertext in 8-byte blocks. 
    blocks = []
    block_size = 8
    block_count = len(ciphertext) // block_size - 1
    for i in range(block_count):
        block = (i + 1) * block_size
        blocks.append(ciphertext[block:block+block_size])
    # struct.pack format for serializing/deserializing 64-bit unsigned integers
    ulong_be = '>Q'
    # Integrity/size block. After decryption should contain the alternative
    # initial value (AIV) which is a constant (0xA65959A600000000) plus the
    # size of the original plaintext
    aiv = struct.unpack(ulong_be, ciphertext[:8])[0]
    for j in range(0, steps):
        for i in range(block_count, 0, -1):
            cipherblock = struct.pack(ulong_be,
                    aiv ^ ((5 - j) * block_count + i)) + blocks[i-1]
            block = decrypt(cipherblock)
            aiv = struct.unpack(ulong_be, block[:8])[0]
            blocks[i-1] = block[8:]
    assert aiv & 0xffffffff00000000 == 0xa65959a600000000
    # Integrity check passed, the original length is specified in the 32 LSB of
    # the aiv
    length = aiv & 0xffffffff
    plaintext_with_pad = b''.join(blocks)
    return plaintext_with_pad[:length]

def export_private_key(session, key):
    kek_handle = session.create_object({
        Attribute.CLASS: ObjectClass.SECRET_KEY,
        Attribute.KEY_TYPE: KeyType.AES,
        Attribute.VALUE: KEK,
        Attribute.WRAP: True,
        Attribute.UNWRAP: True,
        Attribute.TOKEN: False,
    })
    wrapped_key = kek_handle.wrap_key(key,
            mechanism=Mechanism.AES_KEY_WRAP_PAD)
    clear_key = unwrap_key(KEK, wrapped_key)
    return clear_key

def main():
    setup_softhsm()
    lib = pkcs11.lib(MODULE_PATH)
    token = lib.get_token(token_label=TOKEN_LABEL)

    with token.open(user_pin=PIN, rw=True) as session:
        destroy_test_rsa_keypair(session)
        generate_test_rsa_keypair(session)
        keys = list(session.get_objects({
            Attribute.LABEL: KEYPAIR_LABEL,
            Attribute.EXTRACTABLE: True,
        }))
        exported_key = export_private_key(session, keys[0])
        encrypt_sample_text(session, exported_key)

if __name__ == '__main__':
    main()

@Aearsis / @rijswijk thanks for pointing me in the right directions

[garlett]

@tarruda, I need to backup/export A3 (smartcard) private keys too...

I installed openSC and SoftHSMv2.5, then imported an A1(pfx file) certificate (for testing purposes) softhsm2-util.exe --init-token --slot 0 --label "My token 1" openssl pkcs12 -in cert.pfx -nocerts -out cert.key openssl pkcs12 -in cert.pfx -clcerts -nokeys -out cert.crt pkcs11-tool.exe -v --module softhsm2-x64.dll -l --pin 1234 --write-object cerj.key --type privkey --id 2222 pkcs11-tool.exe -v --module softhsm2-x64.dll -l --pin 1234 --write-object cert.crt --type cert --id 2222 then added SofHSM pkcs11 module on firefox and successfully logged on receita federal.

But the certificate authorities ( cert1s1gn ) may require chrome with windows store ( csp, mindriver, certutil.exe ...). Did you made SoftHSM work with windows store?

It works with IsoApplet3-> jcardsim -> OpenSC -> certiutil.exe -scinfo ... import certificate -> windows store, but the simulator loses all data on close, and IsoApplet3 does not export private key.

I was about to patch IsoApplet3 and OpenSC, then found this post.

thank you

[garlett]

Conseguiria me informar como teve sucesso no emitir? Precisou integrar no windows esse hsm ? (pelo que entendi aqui, precisaria um 'minidriver' para o hsm, pois o windows não usa o padrão pcks11) Ou a tua certificadora aceita linux ou firefox para emissão ?

obrigado

[tarruda]

Conseguiria me informar como teve sucesso no emitir? Precisou integrar no windows esse hsm ? (pelo que entendi aqui, precisaria um 'minidriver' para o hsm, pois o windows não usa o padrão pcks11) Ou a tua certificadora aceita linux ou firefox para emissão ?

obrigado

It has been 3 years since I made this post and my A3 certificates have expired. I will be emitting new ones in the next month, when I'm done I will post an update here