Using EST + Bootstrap + TPM2-PKCS11: Error "bootstrap identity cert private key: a parameter has an invalid value"

ksaye commented 3 years ago

Expected Behavior

Trying to setup Edge 1.2 using the TPM2-PKCS11 for the private key for the bootstrap cert.

Current Behavior

I am getting the erro:

Jun 02 22:21:56 slb6 systemd[1]: Started Azure IoT Certificates Service.
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - Starting service...
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - Version - 1.2.0
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - Starting server...
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [DBUG] - [hyper::proto::h1::io] parsed 1 headers
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [DBUG] - [hyper::proto::h1::conn] incoming body is empty
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - <-- GET /certificates/device-id?api-version=2020-09-01 {"host": "certd.sock"}
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - !!! parameter "id" has an invalid value
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - !!! caused by: not found
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - --> 400 {"content-type": "application/json"}
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [DBUG] - [hyper::proto::h1::io] flushed 190 bytes
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (959 bytes)
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - <-- POST /certificates?api-version=2020-09-01 {"content-type": "application/json", "host": "certd.sock", "content-length": "959"}
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [ERR!] - !!! internal error
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [ERR!] - !!! caused by: could not create cert
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [ERR!] - !!! caused by: cert "device-id" is configured to be issued by EST, but neither EST identity nor EST bootstrap identity could be obtained; could not get EST identity cert: entity not found could not get EST bootstrap identity cert private key: a parameter has an invalid value
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [INFO] - --> 500 {"content-type": "application/json"}
Jun 02 22:21:56 slb6 aziot-certd[12840]: 2021-06-02T22:21:56Z [DBUG] - [hyper::proto::h1::io] flushed 155 bytes

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

# install IoT Edge 1.2 on Ubuntu 18.04

# install the tpm2 tools and tpm2-pkcs11
wget https://raw.githubusercontent.com/ksaye/IoTDemonstrations/master/installTPM2Tools.sh && chmod +x installTPM2Tools.sh 
./installTPM2Tools.sh

# generate a key pair in the TPM, using the PIN 1234
export PKCS11_LIB_PATH='/usr/local/lib/libtpm2_pkcs11.so'
sudo -u aziotks pkcs11-tool --module "$PKCS11_LIB_PATH" --label="tpm-rsa-pair" --login --keypairgen --usage-sign --key-type rsa:2048
sudo -u aziotks pkcs11-tool --module "$PKCS11_LIB_PATH" --list-objects --login

# download the root CA public cert from the EST server
openssl s_client -showcerts -verify 5 -connect ***Removed***:443 < /dev/null \
   | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; out="cert"a".pem"; print >out}' && cp cert2.pem cacert.crt.pem && rm cert?.pem

# download our openssl.conf and install the libp11 for OpenSSL
sudo mkdir /var/secrets && cd /var/secrets && sudo wget https://raw.githubusercontent.com/ksaye/IoTDemonstrations/master/pkcs11_openssl.conf && cd
sudo apt install -y pkgconf libssl-dev && mkdir ~/libp11 && cd ~/libp11
wget https://github.com/OpenSC/libp11/releases/download/libp11-0.4.11/libp11-0.4.11.tar.gz && tar -xvf libp11-0.4.11.tar.gz && cd libp11-0.4.11
./configure && make && sudo make install && cd

# make a csr with the key still in the TPM
sudo -u aziotks OPENSSL_CONF=/var/secrets/pkcs11_openssl.conf openssl req -new -subj /CN=bootstrap -sha256 -engine pkcs11 -keyform engine \
    -key slot_1-label_tpm-rsa-pair -out /tmp/bootstrap.csr.pem

# submit our CSR to the EST Server
curl https://***Removed***:443/.well-known/est/simpleenroll --user estuser:estpwd --cacert cacert.crt.pem \
   --output cert.p7 --header "Content-Type: application/pkcs10" --header "Content-Transfer-Encoding: base64" --data-binary @/tmp/bootstrap.csr.pem

# convert our p7 to pem format
openssl base64 -d -in cert.p7 | openssl pkcs7 -inform DER -outform PEM -print_certs | openssl x509 -out bootstrap.cert.pem && rm cert.p7

# create /var/secrets, give aziotks ownership and copy the current certificates
sudo mkdir /var/secrets
sudo cp *.pem /var/secrets
sudo chown aziotks:aziotks /var/secrets/*.pem

# Configure /etc/aziot/config.toml:
[provisioning]
source = "dps"
global_endpoint = "https://global.azure-devices-provisioning.net"
id_scope = "***Removed***"

[provisioning.attestation]
method = "x509"
registration_id = "iotedgeksaye621"
identity_cert = { method = "est", common_name = "bootstrap" }    

[cert_issuance.est]
 trusted_certs = [
     "file:///var/secrets/cacert.crt.pem",
 ]

[cert_issuance.est.auth]
username = "estuser"
password = "estpwd"
bootstrap_identity_cert = "file:///var/secrets/bootstrap.cert.pem"
bootstrap_identity_pk = "pkcs11:token=IoTEdgeCert?pin-value=1234"

[cert_issuance.est.urls]
default = "https://***Removed***/.well-known/est"

Note my SSL Cert looks good, and if I use the same process but not store in TPM it works fine.

openssl x509 -in /var/secrets/bootstrap.cert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 33 (0x21)
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = estExampleCA
        Validity
            Not Before: Jun  2 22:19:24 2021 GMT
            Not After : Jun  2 22:19:24 2022 GMT
        Subject: CN = bootstrap

Context (Environment)

Output of `iotedge check`

Click here

``` sudo iotedge check Configuration checks (aziot-identity-service) --------------------------------------------- √ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK √ aziot-identity-service package is up-to-date - OK √ host time is close to reference time - OK √ production readiness: identity certificates expiry - OK × production readiness: EST identity and bootstrap certificates expiry - Error x509 identity certificate is neither preloaded nor configured to be dynamically issued, and thus cannot be used. √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ identityd is running - OK × read all preloaded certificates from the Certificates Service - Error could not load cert with ID "aziot-edged-trust-bundle" Caused by: parameter "id" has an invalid value caused by: not found √ read all preloaded key pairs from the Keys Service - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK Connectivity checks (aziot-identity-service) -------------------------------------------- ‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. ‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. ‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. √ host can connect to and perform TLS handshake with DPS endpoint - OK Configuration checks -------------------- √ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK ```

Device Information

Host OS [e.g. Ubuntu 18.04, Windows Server IoT 2019]: Ubuntu 18.04
Architecture [e.g. amd64, arm32, arm64]: amd64
Container OS [e.g. Linux containers, Windows containers]: Linux

Runtime Versions

aziot-edged [run iotedge version]: 1.2.0
Edge Agent [image tag (e.g. 1.0.0)]: 1.2.0
Edge Hub [image tag (e.g. 1.0.0)]: 1.2.0
Docker/Moby [run docker version]: 20.10.6+azure

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

aziot-edged logs

``` ```

edge-agent logs

``` ```

edge-hub logs

``` ```

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

arsing commented 3 years ago

could not get EST bootstrap identity cert private key: a parameter has an invalid value

The problem is the key, not the cert.

If that really is your entire /etc/aziot/config.toml, then it's missing the [pkcs11] section where you'd configure (at least) the path of the tpm2_pkcs11.so, so the bootstrap key specified via pkcs11: URI won't work.

If your /etc/aziot/config.toml does have a [pkcs11] section and you just redacted it when posting here, then post the logs of aziot-keyd from the same time period so we can see why it rejected the key handle request.

ksaye commented 3 years ago

@arsing, that was such a rookie mistake on my part (missing the [pkcs11] section). Thank you.

Azure / iotedge