Failed to get a certificate from EST server during x509 Provisioning when CSR subject contains additional fields (not only CN)

[bemol38]

Hi everybody, What I want to achieve is to provision my device to my DPS, using X509 certificate attestation, using as identity_cert a certificate issued by a Digicert EST end-point, which requires a CSR containing both a common_name (CN) and an organizational_unit (OU) fields. To authenticate to the EST server, a pre-existing "birth" certificate (according to our own terminology) is used, the key pair of which is stored in TPM.

Therefore my config.toml template looks like below, and the env variables are substituted with the envsubst command, before aziotctl config is applied.

## DPS provisioning with X.509 certificate
[provisioning]
source = "dps"
global_endpoint = "https://global.azure-devices-provisioning.net"
id_scope = "$scope_id"

[provisioning.attestation]
method = "x509"
registration_id = "$registration_id"

[provisioning.attestation.identity_cert]
method = "est"
url = "$cert_issuance_est_urls"
subject = { CN = "$common_name", OU = "$organization_unit" }

[aziot_keys]
pkcs11_lib_path = "$pkcs11_lib_path"
pkcs11_base_slot = "$pkcs11_base_slot"

[cert_issuance.est]
trusted_certs = [
    "$trusted_cert",
]

[cert_issuance.est.auth]
identity_cert = "$birth_cert_path"
identity_pk = "$pkcs11_birth_key_slot"

[cert_issuance.est.urls]
default = "$cert_issuance_est_urls"

By doing so, we get the following error (see end of the log):

sept. 05 11:01:21 myself aziot-identityd[46367]: 2022-09-05T09:01:21Z [INFO] - Starting service...
sept. 05 11:01:21 myself aziot-identityd[46367]: 2022-09-05T09:01:21Z [INFO] - Version - 1.3.0
sept. 05 11:01:21 myself aziot-identityd[46367]: 2022-09-05T09:01:21Z [INFO] - Provisioning starting. Reason: Startup
sept. 05 11:01:21 myself systemd[1]: Started Azure IoT Keys Service.
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Starting service...
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Version - 1.3.0
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Loaded libaziot-keys with version 0x02010000
sept. 05 11:01:21 myself aziot-keyd[46384]: WARNING:fapi:src/tss2-fapi/api/Fapi_List.c:226:Fapi_List_Finish() Profile of path not provisioned: /HS/SRK
sept. 05 11:01:21 myself aziot-keyd[46384]: ERROR:fapi:src/tss2-fapi/api/Fapi_List.c:81:Fapi_List() ErrorCode (0x00060034) Entities_List
sept. 05 11:01:21 myself aziot-keyd[46384]: ERROR: Listing FAPI token objects failed.
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Starting server...
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - <-- GET /keypair/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [ERR!] - invalid parameter "id": not found
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - !!! a parameter has an invalid value
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - --> 400 {"content-type": "application/json"}
sept. 05 11:01:21 myself systemd[1]: Started Azure IoT Certificates Service.
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Starting service...
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Version - 1.3.0
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Certificate est-id will be auto-renewed. Next renewal at 2023-05-31T06:41:52+00:00.
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Starting server...
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - <-- GET /certificates/device-id?api-version=2020-09-01 {"host": "certd.sock"}
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - !!! parameter "id" has an invalid value
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - !!! caused by: not found
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - --> 400 {"content-type": "application/json"}
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - <-- POST /keypair?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "56"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "355", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-certd[46406]: 2022-09-05T09:01:23Z [INFO] - <-- POST /certificates?api-version=2020-09-01 {"content-type": "application/json", "host": "certd.sock", "content-length": "951"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- GET /keypair/est-id?api-version=2021-05-01 {"host": "keyd.sock"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "244", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "244", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "244", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "632", "content-type": "application/json"}
sept. 05 11:01:24 myself aziot-keyd[46384]: 2022-09-05T09:01:24Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:24 myself aziot-keyd[46384]: 2022-09-05T09:01:24Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "632", "content-type": "application/json"}
sept. 05 11:01:25 myself aziot-keyd[46384]: 2022-09-05T09:01:25Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [ERR!] - !!! internal error
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [ERR!] - !!! caused by: could not create cert
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [ERR!] - !!! caused by: EST endpoint did not return successful response: 400 Bad Request b"{\"errors\":[{\"code\":\"invalid_input\",\"message\":\"Please provide value for subject.organization_unit\"}]}"
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [INFO] - --> 500 {"content-type": "application/json"}
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: internal error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - service encountered an error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - caused by: internal error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - caused by: could not create certificate
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - caused by: internal error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] -    0: <unknown>
sept. 05 11:01:25 myself aziot-identityd[46367]:    1: <unknown>
sept. 05 11:01:25 myself systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
sept. 05 11:01:25 myself systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.

As you can see, the EST server rejected the request because of a missing value for subject.organization_unit.

My first question would be: is there a way to look into what the CSR really contains ?

To check if the problem could come from the EST end-point, I first created a CSR manually:

sudo -Hu aziotks OPENSSL_CONF=/var/secrets/pkcs11_openssl.conf \
  openssl req -new -sha256 -subj "/CN=my-device/OU=my-device-ou" \
    -engine pkcs11 -keyform engine -key "pkcs11:token=device;object=device-rsa-pair?pin-value=1234"  \
    -outform PEM -out device_cert.csr.pem -nodes
sudo -Hu aziotks openssl req -in temp/device_cert.csr.pem -noout -text | head -n 4

which gave:

Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = my-device, OU = my-device-ou

and then sent the CSR to the EST end-point:

sudo -Hu aziotks OPENSSL_CONF=/var/secrets/pkcs11_openssl.conf \
  curl --output temp/device_cert.p7 \
  --location --request POST "$digicert_est_op_url" \
  --header "Content-Type: application/pkcs10" \
  --header "Content-Transfer-Encoding: base64" \
  --data-binary "@temp/device_cert.csr.pem" \
  --cacert "/var/secrets/cacert.crt.pem"  \
  --cert "/var/secrets/birth_cert.public.pem" \
  --key "$pkcs11_birth_key_slot"

# convert device cert from p7 to pem format
sudo openssl base64 -d -in temp/device_cert.p7 | \
 openssl pkcs7 -inform DER -outform PEM -print_certs | \
 openssl x509 -out temp/device_cert.public.pem

# mv device cert to the proper place
sudo mv temp/device_cert.public.pem /var/secrets/
sudo chown aziotks:aziotks /var/secrets/device_cert.*

sudo -Hu aziotks openssl x509 -in /var/secrets/device_cert.public.pem -text | head -n 6

one can see that the certificate could be issued:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            32:3a:5f:57:f3:48:cf:8b:ab:5f:ad:af:8e:44:4a:b2:d0:fe:8b:4e
        Signature Algorithm: sha256WithRSAEncryption

What I also tried is to use the Certificate Service to send the CSR, after having created the needed config files in certd and keyd. With that files configured, I could submit the CSR with this command:

sudo curl --unix-socket /run/aziot/certd.sock http://localhost/certificates?api-version=2020-09-01  \
  -H "content-type: application/json" \
  --data "$(jq -cn --arg 'certId' 'device-cert' --arg 'csr' "$(cat temp/device_cert.csr.pem)" '{"certId": $certId, "csr": $csr}')" | \
  jq -r '.pem' > temp/device_cert.public.pem

and got a valid certificate too. From this success I concluded that the problem is not sending the CSR but creating the CSR.

So my actual conclusion is: there is a problem with the CSR that the Identity Service is generating, and sending to the EST end-point

I would really appreciate any help to better diagnose the problem.

[arsing]

As you can see, the EST server rejected the request because of a missing value for subject.organization_unit.

Yes, it's a known bug. We've received the same bug report at https://github.com/Azure/iotedge/issues/6579 previously.

My first question would be: is there a way to look into what the CSR really contains ?

It's not logged anywhere by certd if that's what you're asking. You could compile your own debug build and step through with a debugger. Or you could spin up a tiny netcat server and set it as the EST endpoint.

[bemol38]

Thank you @arsing. You answered my questions. To my understanding, to build a correct CSR, the identityd service would first need to access the full subject, if any. Either the subject would need to be written to the identityd 00-super.toml too, or it should be retrievable from identityd through an api call to certd, with a new api function (get_cert_subject ?), since the subject is available in the certd 00-super.toml. And the create_csr function would need to be rewritten too... And maybe more ...

[jlian]

@bemol38 we've triaged the bug and are working on a fix. Might be slotted for a slightly later release as we line up some other fixes and security patches. How urgent is this for you?

[bemol38]

Hi @jlian First, thank you for working on a fix. I would be very happy to be able to test this fix before end of october.

[jlian]

Would you be able to have DigiCert configure the EST endpoint to not require organizational_unit (OU) field in the meantime?

[bemol38]

Hi @jlian, To answer your question, yes, we are using as a temporary solution an EST end-point which does not require the OU field.

[jlian]

Hey folks, the change is merged (thanks @onalante-msft).

Ideally, we could have you try it before we take it for the release:

1. Download the patched binary here https://github.com/Azure/iot-identity-service/actions/runs/3364093065
2. And then manually install following these steps https://azure.github.io/iot-identity-service/installation.html

@bemol38 do you think you could give it a try this week?

[bemol38]

Hi @jlian I gave a try this morning, and it worked. aziot service is able to build a CSR containing the OU field, as set in config.toml, and to send the CSR to our Digitcert EST Server, which now accepts it, and returns a certificate containing the OU field. The certificate lands in /var/lib/aziot/certd/certs, as expected. With that certificate, the automated provisioning is executed, the OU field is checked with a custom allocation policy and the device is registered to the IoTHub. So far so good :-) (What I did not test already is the certificate renewal.)

Thanks a lot for your valued assistance !

[jlian]

Identity service 1.4.2 is available and includes this fix

[pebneter]

Thanks for fixing this!