search

NLnetLabs / krill

RPKI Certificate Authority and Publication Server written in Rust
https://nlnetlabs.nl/projects/routing/krill/
Mozilla Public License 2.0
295 stars 42 forks source link

Manual PKCS#11 signer testing with YubiHSM2 #555

Closed ximon18 closed 2 years ago

ximon18 commented 2 years ago

Note: In the shell command logs below commands are prefixed by [N]> to indicate which of several shell sessions the command was/should be issued in.

Tested using Krill commit e479dea (the current head of the issue-698-record-keys-on-upgrade branch) and a YubiHSM2 on Ubuntu 21.10.

Krill was installed like so:

[1]> cargo install --git https://github.com/NLnetLabs/krill --rev e479dea --locked --features hsm

The Krill config file looked like this:

[1]> cat /tmp/krill.conf
admin_token = "abc"
data_dir = "/tmp/krill"
service_uri = "https://localhost:3000/"
log_level = "trace"
log_type = "stderr"

[[signers]]
type = "PKCS#11"
lib_path = "/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so"
slot = 0
user_pin = "0001password"

Prepare to use the YubiHSM2: _(based on the official YubiHSM 2: Practical Guide)_

[1]> wget https://developers.yubico.com/YubiHSM2/Releases/yubihsm2-sdk-2021-04-ubuntu1804-amd64.tar.gz
[1]> tar zxf ./yubihsm2-sdk-2021-04-ubuntu1804-amd64.tar.gz
[1]> cd yubihsm2-sdk

.. install the .deb files located in this folder ..

[1]> cat /etc/udev/rules.d/80-yubihsm-connector.rules 
# This udev file should be used with udev 188 and newer
ACTION!="add|change", GOTO="yubihsm2_connector_end"

# Yubico YubiHSM 2
# The OWNER attribute here has to match the uid of the process running the Connector
SUBSYSTEM=="usb", ATTRS{idVendor}=="1050", ATTRS{idProduct}=="0030", OWNER="yubihsm-connector"

LABEL="yubihsm2_connector_end"

.. connect the YubiHSM2 device to a USB port ..
.. (note: connect it directly, not via a USB hub) ..
.. verify that the YubiHSM2 is working ..

[1]> lsusb | grep -i yubi
Bus 003 Device 011: ID 1050:0030 Yubico.com YubiHSM
[1]> sudo yubihsm-connector -d
DEBU[0000] preflight complete                            cert= config= key= pid=98344 seccomp=false serial= syslog=false timeout=0s version=3.0.1
DEBU[0000] takeoff                                       TLS=false listen="localhost:12345" pid=98344

In another terminal factory reset the YubiHSM2 and verify that it is empty:

[2]> wget -qO- http://127.0.0.1:12345/connector/status
status=OK
serial=*
version=3.0.1
pid=98344
address=localhost
port=12345

[2]> yubihsm-shell
Using default connector URL: http://127.0.0.1:12345
yubihsm> connect
Session keepalive set up to run every 15 seconds
yubihsm> session open 1 password
Created session 0
yubihsm> reset 0
Device successfully reset
yubihsm> list objects 0
Invalid argument 2: 0 (e:session)
yubihsm> session open 1 password
Created session 0
yubihsm> list objects 0
Found 1 object(s)
id: 0x0001, type: authentication-key, sequence: 0

Ensure that any processes launched from this shell session that use the YubiHSM2 PKCS#11 library are able to access the device via the connector:

[2]> echo 'connector = http://127.0.0.1:12345' > /tmp/yubihsm_pkcs11.conf
[2]> export YUBIHSM_PKCS11_CONF=/tmp/yubihsm_pkcs11.conf

Inpsect the token using pkcs11-tool:

[2]> pkcs11-tool --module /usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so -I
Cryptoki version 2.40
Manufacturer     Yubico (www.yubico.com)
Library          YubiHSM PKCS#11 Library (ver 2.20)
Using slot 0 with a present token (0x0)
[2]> pkcs11-tool --module /usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so -T
Available slots:
Slot 0 (0x0): YubiHSM Connector localhost
  token label        : YubiHSM
  token manufacturer : Yubico (www.yubico.com)
  token model        : YubiHSM
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 2.200
  firmware version   : 2.200
  serial num         : 16499155
  pin min/max        : 12/68

Use the keyls helper tool to list the keys in the YubiHSM2 device before running Krill:

[2]> cargo install --git https://github.com/ximon18/keyls --branch main --locked
[2]> keyls "pkcs11:0:0001password@/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so"
Using PKCS#11 slot id 0 (0x0)
No keys found

Run Krill:

[2]> krill -c /tmp/krill.conf

Krill configures itself to use the PKCS#11 signer for and a fallback OpenSSL signer:

[INFO] ... Configuring signer 'PKCS#11' (type: PKCS#11, default: true, oneoff: false, random: false)
[INFO] ... Configuring signer 'Fallback OpenSSL signer' (type: OpenSSL, default: false, oneoff: true, random: true)

In another terminal use krillc to create a CA:

[3]> export KRILL_CLI_TOKEN=abc
[3]> krillc add --ca some_ca

On creation of a CA, Krill attempts to initialize the signers and contact the PKCS#11 token:

[TRACE] ... Loading PKCS#11 library '"/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so"'
[TRACE] ... Loaded PKCS#11 library '"/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so"'
[INFO] ... Using PKCS#11 token 'YubiHSM (model: YubiHSM, vendor: Yubico (www.yubico.com))' in slot 0 of server 'Yubico (www.yubico.com) (Cryptoki v2.20)' via library '/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so'
...
[INFO] ... Signer 'PKCS#11' is ready for use
[INFO] ... Signer 'Fallback OpenSSL signer' is ready for use

Prepare another terminal for issuing commands to a testbed instance of Krill, in this case the NLnet Labs public testbed:

[4]> export KRILL_CLI_SERVER=https://testbed.rpki.nlnetlabs.nl/ 
[4]> export KRILL_CLI_TOKEN=********

Using TWO DIFFERENT TERMINALS register Krill with the NLnet Labs public testbed as a publisher:

[3]> krillc repo request --ca some_ca > /tmp/req.xml
[4]> krillc pubserver publishers add --request /tmp/req.xml >/tmp/res.xml
[3]> krillc repo configure --ca some_ca --response /tmp/res.xml

Using TWO DIFFERENT TERMINALS register Krill as a child CA under the testbed:

[3]> krillc parents request --ca some_ca > /tmp/req2.xml
[4]> krillc children add --ca testbed --asn 18 --ipv4 10.0.0.0/24 --child some_ca --request /tmp/req2.xml >/tmp/res2.xml
[3]> krillc parents add --ca some_ca --response /tmp/res2.xml --parent testbed

Finally, create a ROA:

[3]> krillc roas update --ca some_ca --add "10.0.0.1/32 => 18"

There will now be three key pairs stored in SoftHSM:

[3]> keyls "pkcs11:0:0001password@/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so"
Using PKCS#11 slot id 0 (0x0)
+------+-------------+-------+-----------+--------+
| ID   | Type        | Name  | Algorithm | Length |
+------+-------------+-------+-----------+--------+
| 207D | Private Key | Krill | RSA       | 0      |
| 207D | Public Key  | Krill | RSA       | 2048   |
| 672B | Private Key | Krill | RSA       | 0      |
| 672B | Public Key  | Krill | RSA       | 2048   |
| A8CC | Private Key | Krill | RSA       | 0      |
| A8CC | Public Key  | Krill | RSA       | 2048   |
+------+-------------+-------+-----------+--------+

There will now be two key identifiers mapped for the PKCS#11 signer and none to the OpenSSL signer:

[3]> jq '.signer_name, .signer_identity.private_key_internal_id, .keys' /tmp/krill/signers/840f63d5-4b80-471f-a9f1-f8b4f05c27a1/snapshot.json 
"PKCS#11"
"207d72a89d68e7de2e63d356c23334a1720cc9e4"
{
  "5A1018EC67A85DD0AF7A5CF70B3479D9291F9735": "672b384ea483da0eeec9915b6dcf15c4bc7e58d1",
  "5B44AD82F1E6340E338CC91EAAEDD80A886B4757": "a8ccaa0803d83bc55d1115a8e3caf50eddeeae72"
}

[3]> jq '.signer_name, .signer_identity.private_key_internal_id, .keys' /tmp/krill/signers/a44bd738-65cd-418e-8c1a-4694e0b0b909/snapshot.json 
"Fallback OpenSSL signer"
"20B3BFF767415CA401026F82EA8717496DAC4DFC"
{}

And the OpenSSL keys directory contains only the identity key that Krill created for it:

[2]> ls -1 /tmp/krill/keys/
20B3BFF767415CA401026F82EA8717496DAC4DFC

Finally, cleanup the testbed:

[4]> krillc children remove --ca testbed --child some_ca
[4]> krillc pubserver publishers remove -p some_ca
ximon18 commented 2 years ago

One observation regarding the above is that the CKA_ID reported for each key is only the first 2 bytes, e.g. 207D instead of the whole key identifier. pkcs11-tool shows this too:

$ pkcs11-tool --module /usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so -O -p 0001password
Using slot 0 with a present token (0x0)
Private Key Object; RSA 
  label:      Krill
  ID:         207d
  Usage:      sign
  Access:     sensitive, always sensitive, never extractable, local
...

This doesn't seem to cause Krill any problems when using the keys to sign, but could it fail to select the right key or could there be collisions between keys due to the truncated identifier length, or is this only an artifact of the way the YubiHSM2 PKCS#11 library reports the PKCS#11 CKA_ID field value?

The yubihsm-shell command also shows the truncated identifiers:

$ yubihsm-shell
Using default connector URL: http://127.0.0.1:12345
yubihsm> connect
Session keepalive set up to run every 15 seconds
yubihsm> session open 1 password
Created session 0
yubihsm> list objects 0
Found 4 object(s)
id: 0x0001, type: authentication-key, sequence: 0
id: 0x207d, type: asymmetric-key, sequence: 0
id: 0x672b, type: asymmetric-key, sequence: 0
id: 0xa8cc, type: asymmetric-key, sequence: 0
yubihsm> get objectinfo 0 0x207d asymmetric-key
id: 0x207d, type: asymmetric-key, algorithm: rsa2048, label: "Krill", length: 896, domains: 1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16, sequence: 0, origin: generated, capabilities: sign-pkcs:sign-pss
ximon18 commented 2 years ago

Update: Still works after migrating from the pkcs11 crate dependency to the cryptoki crate dependency instead. See this tweet.

ximon18 commented 2 years ago

v0.10.0-rc2 testing

Environment:

$ cat /etc/fedora-release 
Fedora release 36 (Thirty Six)

$ uname -r
5.18.13-200.fc36.x86_64

$ cargo --version
cargo 1.62.1 (a748cf5a3 2022-06-08)

The YubiHSM 2 USB key is already attached to and setup on another host, in my test case it is available at 192.1.68.0.53 on port 12345.

Preparation:

Start the YubiHSM 2 connector on the machine to which the YubiHSM2 is physically attached and do a factory reset on the YubiHSM 2:

$ sudo yubihsm-connector -d -l 0.0.0.0:12345
DEBU[0000] preflight complete                            cert= config= key= pid=23579 seccomp=false serial= syslog=false timeout=0s version=3.0.2
DEBU[0000] takeoff

$ wget -qO- http://127.0.0.1:12345/connector/status
status=OK
serial=*
version=3.0.2
pid=23579
address=0.0.0.0
port=12345

$ yubihsm-shell 
Using default connector URL: http://127.0.0.1:12345
yubihsm> connect
Session keepalive set up to run every 15 seconds
yubihsm> session open 1 password
Created session 0
yubihsm> reset 0
Device successfully reset
yubihsm> session open 1 password
Created session 0
yubihsm> list objects 0
Found 1 object(s)
id: 0x0001, type: authentication-key, sequence: 0

On the Krill machine, ensure that Krill is able to access the YubiHSM 2 via the connector and install the YubiHSM 2 PKCS#11 library from the Fedora 36 version of the YubiHSM 2 SDK:

[1]> echo 'connector = http://192.168.0.53:12345' > /tmp/yubihsm_pkcs11.conf
[1]> export YUBIHSM_PKCS11_CONF=/tmp/yubihsm_pkcs11.conf
[1]> wget https://developers.yubico.com/YubiHSM2/Releases/yubihsm2-sdk-2022-06-fedora36-amd64.tar.gz
[1]> tar zxf yubihsm2-sdk-2022-06-fedora36-amd64.tar.gz
[1]> cd yubihsm2-sdk
[1]> sudo dnf install ./yubihsm-shell-2.3.2-1.fc36.x86_64.rpm
[1]> ll /usr/lib64/pkcs11/yubihsm_pkcs11.so 
-rwxr-xr-x 1 root root 388480 Jun 17 22:12 /usr/lib64/pkcs11/yubihsm_pkcs11.so

Install the keyls helper tool and prepare our krill.conf:

[1]> cargo install --git https://github.com/ximon18/keyls --branch main --locked

[1]> keyls "pkcs11:0:0001password@/usr/lib64/pkcs11/yubihsm_pkcs11.so"
Using PKCS#11 slot id 0 (0x0)
No keys found

[1]> cat /tmp/krill.conf
admin_token = "abc"
data_dir = "/tmp/krill"
service_uri = "https://localhost:3000/"
log_level = "trace"
log_type = "stderr"
default_signer = "YubiHSM2 via PKCS#11"

[[signers]]
name = "YubiHSM2 via PKCS#11"
type = "PKCS#11"
lib_path = "/usr/lib64/pkcs11/yubihsm_pkcs11.so"
slot = 0
user_pin = "0001password"

Also prepare two additional terminals for communicating with the local Krill and the external testbed:

In another terminal [2]:

[2]> $ export KRILL_CLI_TOKEN=abc

In yet another terminal [3] prepare to manage the testbed side of the setup:

[3]> export KRILL_CLI_SERVER=https://testbed.rpki.nlnetlabs.nl/ 
[3]> export KRILL_CLI_TOKEN=********

Testing

Install and run Krill:

[1]> cargo install --git https://github.com/NLnetLabs/krill --tag v0.10.0-rc2 --locked

[1]> krill --version
Krill 0.10.0-rc2

[1]> krill -c /tmp/krill.conf

In terminal [2] add a CA to Krill:

[2]> krillc add --ca some_ca

In yet another terminal [3] prepare to manage the testbed side of the setup:

[3]> export KRILL_CLI_SERVER=https://testbed.rpki.nlnetlabs.nl/ 
[3]> export KRILL_CLI_TOKEN=********

Using TWO DIFFERENT TERMINALS register Krill with the NLnet Labs public testbed as a publisher:

[2]> krillc repo request --ca some_ca > /tmp/req.xml
[3]> krillc pubserver publishers add --request /tmp/req.xml >/tmp/res.xml
[2]> krillc repo configure --ca some_ca --response /tmp/res.xml

Using TWO DIFFERENT TERMINALS register Krill as a child CA under the testbed:

(NOTE: Due to the testbed in this example not having been upgraded yet we have to talk to the testbed using older Krill, in this case just by running krillc on the testbed server itself, otherwise we get Error: Invalid JSON: missing fieldv4at line 7 column 3)

[2]> krillc parents request --ca some_ca > /tmp/req2.xml
[3]> scp /tmp/req2.xml ubuntu@testbed.rpki.nlnetlabs.nl:/tmp/ && ssh ubuntu@testbed.rpki.nlnetlabs.nl krillc children add --ca testbed --asn 18 --ipv4 10.0.0.0/24 --child some_ca --request /tmp/req2.xml >/tmp/res2.xml
req2.xml                          
[2]> krillc parents add --ca some_ca --response /tmp/res2.xml --parent testbed

Finally, create a ROA:

[2]> krillc roas update --ca some_ca --add "10.0.0.1/32 => 18"

And look at which keys we have now:

[2]> keyls "pkcs11:0:0001password@/usr/lib64/pkcs11/yubihsm_pkcs11.so"
Using PKCS#11 slot id 0 (0x0)
Found 6 keys
+------+-------------+-------+-----------+--------+
| ID   | Type        | Name  | Algorithm | Length |
+------+-------------+-------+-----------+--------+
| 1842 | Private Key | Krill | RSA       |        |
| 1842 | Public Key  | Krill | RSA       | 2048   |
| 7F05 | Private Key | Krill | RSA       |        |
| 7F05 | Public Key  | Krill | RSA       | 2048   |
| B5BE | Private Key | Krill | RSA       |        |
| B5BE | Public Key  | Krill | RSA       | 2048   |
+------+-------------+-------+-----------+--------+

:champagne: :+1:

Cleanup

Finally, cleanup the testbed:

[3]> krillc children remove --ca testbed --child some_ca
[3]> krillc pubserver publishers remove -p some_ca

Checking Krills internals

And for info, here is how Krill reported its connection to SoftHSM2 in its logs: (wrapped for readability)

2022-08-04 13:56:33 [INFO] [krill::commons::crypto::signing::signers::pkcs11::signer] 
Using PKCS#11 token 'YubiHSM (model: YubiHSM, vendor: Yubico (www.yubico.com))' 
in slot 0 of server 'Yubico (www.yubico.com) (Cryptoki v2.32)' via library '/usr/lib64/pkcs11/yubihsm_pkcs11.so'

And here are the signer store files on disk:

[2]> cat /tmp/krill/signers/074c61cc-39e2-4164-8e3a-eff473e722b2/snapshot.json

{
  "id": "074c61cc-39e2-4164-8e3a-eff473e722b2",
  "version": 3,
  "signer_name": "YubiHSM2 via PKCS#11",
  "signer_info": "PKCS#11 Signer [token: YubiHSM (model: YubiHSM, vendor: Yubico (www.yubico.com)), slot: 0, server: Yubico (www.yubico.com) (Cryptoki v2.32), library: /usr/lib64/pkcs11/yubihsm_pkcs11.so]",
  "signer_identity": {
    "public_key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvsTh5/4aMbKhYDEaT5Umo2G5pfqy6A0jqpkBuMomeiAIVZPN8rtuF/tXjL8kG6y1IXLON3jykPra/y1SNPABngin6rn49xIQ0CTWwfkGWPXlpNmdE06oG9Q7spHJXA24qWY8jxMNLGoWrMv7WjxLphGONNpthCApnx4HlGtLDdKgIKaqwcaUltTQgNorvvyO2xAdb+z2VOB7Dpj5yqWGWCt/8KHqcUp0BXRaKdSHrmtbXBGCIqB8XS67oS3KZyjyTj+RJ5XKSpY31aA9Y7xfB332Reiu96zXcGOLHszq3PkgSsWNizRC8skv34ph6ygPNKgcwDiXDKxslCrOcdpnowIDAQAB",
    "private_key_internal_id": "7f053880ab7ab3adb7811a14c50fe9c4116869c7"
  },
  "keys": {
    "5D3E34EAA5F96A8246F8270FCE9F917900B3C71A": "18423ed69a3cc824859c68a2c0dad98d3581af78",
    "6CC9B92A625D21051EFE1B0F30ECE1829E6B275B": "b5bea4e577545381d28e187714a8089ce1e16873"
  }
}

[2]> cat /tmp/krill/signers/512dabb5-085d-4ba7-a9f5-5e54a0c8842e/snapshot.json

{
  "id": "512dabb5-085d-4ba7-a9f5-5e54a0c8842e",
  "version": 1,
  "signer_name": "OpenSSL one-off signer",
  "signer_info": "OpenSSL Soft Signer [version: OpenSSL 3.0.5 5 Jul 2022, keys dir: /tmp/krill/keys]",
  "signer_identity": {
    "public_key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAscRSmjoevHQav8GUe87gSMgLTiZAKQP24WOxQ6c53wd3tWVAas+LwQc2I1CKwDi+hongoqxfwAKlx2RLLC9n1VQsy6gBu+eSKQrZZ00BrsVYLUpWGcjmFAza519jJhA781Y++zSPjzuxC5UXrQM2rUeZz99hFvXl2VDEsf5LuJG1bcZUKyPWcgGOZP3uc3o1keV3+CoOGvjPBLM3X1AEnlKin+aodNkHorlTIxjIeBe2bxaRoPMU8pfk6+G8MjvVQi1eEhQtCVUQJuSG060+s4j10YOgadyido/GAWerCSYbK4OYFQrEqvD9b7d7z+OwdReaxIRgwf2jcM9oru8HnwIDAQAB",
    "private_key_internal_id": "A2EF3DD65D1DDEF87DF27185AAD924018475B05D"
  },
  "keys": {}
}
ximon18 commented 2 years ago

HSM support was delivered with the Krill v0.10.0 release.