search

aws / aws-iot-device-sdk-embedded-C

SDK for connecting to AWS IoT from a device using embedded C.
MIT License
978 stars 625 forks source link

Unable to run fleet provisioning demo #1783

Closed itwondersteam closed 2 years ago

itwondersteam commented 2 years ago

We are able to run the fleet provisioning using the python script fleetprovisioning.py but unable to do so with the embedded-C demo

Below are the command used (xxx to replace some info)

AWS_IOT_ENDPOINT="xxx-ats.iot.us-west-2.amazonaws.com"
ROOT_CA_CERT_PATH="/certs/Amazon-root-CA-1.pem"
CLAIM_CERT_PATH="/certs/xxxx99138f-certificate.pem.crt"
CLAIM_PRIVATE_KEY_PATH="/certs/xxxx99138f-private.pem.key"
PROVISION_NAME="xxx"

python3 python-fleet-provisioning.py \
    --endpoint "$AWS_IOT_ENDPOINT" \
    --templateName "$PROVISION_NAME" \
    --templateParameters '{"SerialNumber": "xxxx"}' \
    --client-id 'xxxx' \
    --key "$CLAIM_PRIVATE_KEY_PATH" \
    --cert "$CLAIM_CERT_PATH" \
    --root-ca "$ROOT_CA_CERT_PATH"

Output

Connecting to xxx-ats.iot.us-west-2.amazonaws.com with client ID 'xxx'...
Connected!
Subscribing to CreateKeysAndCertificate Accepted topic...
Subscribing to CreateKeysAndCertificate Rejected topic...
Subscribing to RegisterThing Accepted topic...
Subscribing to RegisterThing Rejected topic...
Publishing to CreateKeysAndCertificate...
Waiting... CreateKeysAndCertificateResponse: null
Published CreateKeysAndCertificate request..
Received a new message awsiot.iotidentity.CreateKeysAndCertificateResponse(
// ....

For Embedded C SDK Fleet Provisioning Demo

cmake -S . -Bbuild \
        -DAWS_IOT_ENDPOINT="$AWS_IOT_ENDPOINT" \
        -DROOT_CA_CERT_PATH="$ROOT_CA_CERT_PATH" \
        -DCLAIM_CERT_PATH="$CLAIM_CERT_PATH" \
        -DCLAIM_PRIVATE_KEY_PATH="$CLAIM_PRIVATE_KEY_PATH" \
        -DPROVISIONING_TEMPLATE_NAME="$PROVISION_NAME" \
        -DDEVICE_SERIAL_NUMBER="serial-num-testfleet$1"

cd build
make fleet_provisioning_with_csr_demo 
./bin/fleet_provisioning_with_csr_demo

Output

[INFO] [PKCS11] [core_pkcs11_mbedtls.c:1403] PKCS #11 successfully initialized.
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x3 type object.
[INFO] [PKCS11] [core_pkcs11_pal.c:63] Could not open corePKCS11_Claim_Certificate.dat for reading.
[INFO] [FLEET_PROVISIONING_DEMO] [pkcs11_operations.c:770] Writing certificate into label "Claim Cert".
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x1 type object.
[INFO] [FLEET_PROVISIONING_DEMO] [fleet_provisioning_with_csr_demo.c:517] Establishing MQTT session with claim certificate...
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 7 ms backoff.
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 802 ms backoff.
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 351 ms backoff.
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 363 ms backoff.
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 1322 ms backoff.
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[ERROR] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:422] Connection to the broker failed, all attempts exhausted.
[ERROR] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:667] Failed to connect to MQTT broker xxx-ats.iot.us-west-2.amazonaws.com.
[ERROR] [FLEET_PROVISIONING_DEMO] [fleet_provisioning_with_csr_demo.c:525] Failed to establish MQTT session.
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:1932] Successfully closed PKCS #11 session.
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:1446] PKCS #11 was successfully uninitialized.
[WARN] [FLEET_PROVISIONING_DEMO] [fleet_provisioning_with_csr_demo.c:750] Demo iteration 1 failed. Retrying...
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:1403] PKCS #11 successfully initialized.
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x3 type object.
[INFO] [PKCS11] [core_pkcs11_pal.c:63] Could not open corePKCS11_Claim_Certificate.dat for reading.
[INFO] [FLEET_PROVISIONING_DEMO] [pkcs11_operations.c:770] Writing certificate into label "Claim Cert".
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x1 type object.
[INFO] [FLEET_PROVISIONING_DEMO] [fleet_provisioning_with_csr_demo.c:517] Establishing MQTT session with claim certificate...
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 490 ms backoff.
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 380 ms backoff.
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.

Not sure if we have missed out any step?

johnrhen commented 2 years ago

We're looking into this issue. Could you modify the "LIBRARY_LOG_LEVEL" in demo_config.h to be "LOG_DEBUG", do the same to the log level in mbedtls_pkcs11_posix.h, then post the output?

Additionally, what is the policy JSON of the policy which is attached to your claim certificate?

johnrhen commented 2 years ago

Could you also set MBEDTLS_DEBUG_LOG_LEVEL in mbedtls_pkcs11_posix.h to 3?

itwondersteam commented 2 years ago

@johnrhen Thanks for your reply! Here's the policy attached to the provisioning template

Claim Cert's Policy (xxx to replace sensitive info)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Connect"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive",
        "iot:RetainPublish"
      ],
      "Resource": [
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topic/$aws/certificates/create/*",
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topic/$aws/provisioning-templates/xxx-provisioning-template-name/provision/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Subscribe"
      ],
      "Resource": [
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topicfilter/$aws/certificates/create/*",
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topicfilter/$aws/provisioning-templates/xxx-provisioning-template-name/provision/*"
      ]
    }
  ]
}

Below is the output after modifying the log level,

[INFO] [PKCS11] [core_pkcs11_mbedtls.c:1403] PKCS #11 successfully initialized.
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x3 type object.
[INFO] [PKCS11] [core_pkcs11_pal.c:63] Could not open corePKCS11_Claim_Certificate.dat for reading.
[INFO] [FLEET_PROVISIONING_DEMO] [pkcs11_operations.c:770] Writing certificate into label "Claim Cert".
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x1 type object.
[INFO] [FLEET_PROVISIONING_DEMO] [fleet_provisioning_with_csr_demo.c:517] Establishing MQTT session with claim certificate...
[DEBUG] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:399] Establishing a TLS session to xxx-ats.iot.us-west-2.amazonaws.com:8883.
[DEBUG] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:347] Configured MbedTLS context.
mbedTLS: |2| 0x5592cfdfc6e8: => handshake
mbedTLS: |2| 0x5592cfdfc6e8: client state: 0
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: client state: 1
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: => write client hello
mbedTLS: |3| 0x5592cfdfc6e8: client hello, max version: [3:3]
mbedTLS: |3| 0x5592cfdfc6e8: client hello, current time: 1643073489
mbedTLS: |3| 0x5592cfdfc6e8: dumping 'client hello, random bytes' (32 bytes)
mbedTLS: |3| 0x5592cfdfc6e8: 0000:  61 ef 4f d1 05 1d cd 7c e8 dd 16 55 00 e8 a5 07  a.O....|...U....
mbedTLS: |3| 0x5592cfdfc6e8: 0010:  10 37 bf cd fc f9 f8 c3 20 a1 df 80 33 11 2a b8  .7...... ...3.*.
mbedTLS: |3| 0x5592cfdfc6e8: client hello, session id len.: 0
mbedTLS: |3| 0x5592cfdfc6e8: dumping 'client hello, session id' (0 bytes)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc00a (TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc014 (TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc02b (TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc02f (TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc023 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc027 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc009 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc013 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, got 8 ciphersuites (excluding SCSVs)
mbedTLS: |3| 0x5592cfdfc6e8: adding EMPTY_RENEGOTIATION_INFO_SCSV
mbedTLS: |3| 0x5592cfdfc6e8: client hello, compress len.: 1
mbedTLS: |3| 0x5592cfdfc6e8: client hello, compress alg.: 0
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding server name extension: xxx-ats.iot.us-west-2.amazonaws.com
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding signature_algorithms extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding supported_elliptic_curves extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding supported_point_formats extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding max_fragment_length extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding encrypt_then_mac extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding extended_master_secret extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, total extension length: 96
mbedTLS: |2| 0x5592cfdfc6e8: => write handshake message
mbedTLS: |2| 0x5592cfdfc6e8: => write record
mbedTLS: |3| 0x5592cfdfc6e8: output record: msgtype = 22, version = [3:3], msglen = 159
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: message length: 164, out_left: 164
mbedTLS: |2| 0x5592cfdfc6e8: ssl->f_send() returned 164 (-0xffffff5c)
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= write record
mbedTLS: |2| 0x5592cfdfc6e8: <= write handshake message
mbedTLS: |2| 0x5592cfdfc6e8: <= write client hello
mbedTLS: |2| 0x5592cfdfc6e8: client state: 2
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: => parse server hello
mbedTLS: |2| 0x5592cfdfc6e8: => read record
mbedTLS: |2| 0x5592cfdfc6e8: => fetch input
mbedTLS: |2| 0x5592cfdfc6e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x5592cfdfc6e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x5592cfdfc6e8: ssl->f_recv(_timeout)() returned -26624 (-0x6800)
mbedTLS: |1| 0x5592cfdfc6e8: mbedtls_ssl_fetch_input() returned -26624 (-0x6800)
mbedTLS: |1| 0x5592cfdfc6e8: ssl_get_next_record() returned -26624 (-0x6800)
mbedTLS: |1| 0x5592cfdfc6e8: mbedtls_ssl_read_record() returned -26624 (-0x6800)
mbedTLS: |2| 0x5592cfdfc6e8: <= handshake
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
mbedTLS: |2| 0x5592cfdfc6e8: => free
mbedTLS: |2| 0x5592cfdfc6e8: <= free
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 7 ms backoff.
[DEBUG] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:399] Establishing a TLS session to xxx-ats.iot.us-west-2.amazonaws.com:8883.
[DEBUG] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:347] Configured MbedTLS context.
mbedTLS: |2| 0x5592cfdfc6e8: => handshake
mbedTLS: |2| 0x5592cfdfc6e8: client state: 0
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: client state: 1
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: => write client hello
mbedTLS: |3| 0x5592cfdfc6e8: client hello, max version: [3:3]
mbedTLS: |3| 0x5592cfdfc6e8: client hello, current time: 1643073489
mbedTLS: |3| 0x5592cfdfc6e8: dumping 'client hello, random bytes' (32 bytes)
mbedTLS: |3| 0x5592cfdfc6e8: 0000:  61 ef 4f d1 94 e9 14 31 45 50 04 56 69 12 2c d5  a.O....1EP.Vi.,.
mbedTLS: |3| 0x5592cfdfc6e8: 0010:  30 46 ff 3d cd a8 28 f7 bd c6 75 4e 59 98 ec 6c  0F.=..(...uNY..l
mbedTLS: |3| 0x5592cfdfc6e8: client hello, session id len.: 0
mbedTLS: |3| 0x5592cfdfc6e8: dumping 'client hello, session id' (0 bytes)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc00a (TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc014 (TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc02b (TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc02f (TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc023 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc027 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc009 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, add ciphersuite: 0xc013 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x5592cfdfc6e8: client hello, got 8 ciphersuites (excluding SCSVs)
mbedTLS: |3| 0x5592cfdfc6e8: adding EMPTY_RENEGOTIATION_INFO_SCSV
mbedTLS: |3| 0x5592cfdfc6e8: client hello, compress len.: 1
mbedTLS: |3| 0x5592cfdfc6e8: client hello, compress alg.: 0
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding server name extension: xxx-ats.iot.us-west-2.amazonaws.com
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding signature_algorithms extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding supported_elliptic_curves extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding supported_point_formats extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding max_fragment_length extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding encrypt_then_mac extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, adding extended_master_secret extension
mbedTLS: |3| 0x5592cfdfc6e8: client hello, total extension length: 96
mbedTLS: |2| 0x5592cfdfc6e8: => write handshake message
mbedTLS: |2| 0x5592cfdfc6e8: => write record
mbedTLS: |3| 0x5592cfdfc6e8: output record: msgtype = 22, version = [3:3], msglen = 159
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: message length: 164, out_left: 164
mbedTLS: |2| 0x5592cfdfc6e8: ssl->f_send() returned 164 (-0xffffff5c)
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= write record
mbedTLS: |2| 0x5592cfdfc6e8: <= write handshake message
mbedTLS: |2| 0x5592cfdfc6e8: <= write client hello
mbedTLS: |2| 0x5592cfdfc6e8: client state: 2
mbedTLS: |2| 0x5592cfdfc6e8: => flush output
mbedTLS: |2| 0x5592cfdfc6e8: <= flush output
mbedTLS: |2| 0x5592cfdfc6e8: => parse server hello
mbedTLS: |2| 0x5592cfdfc6e8: => read record
mbedTLS: |2| 0x5592cfdfc6e8: => fetch input
mbedTLS: |2| 0x5592cfdfc6e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x5592cfdfc6e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x5592cfdfc6e8: ssl->f_recv(_timeout)() returned -26624 (-0x6800)
mbedTLS: |1| 0x5592cfdfc6e8: mbedtls_ssl_fetch_input() returned -26624 (-0x6800)
mbedTLS: |1| 0x5592cfdfc6e8: ssl_get_next_record() returned -26624 (-0x6800)
mbedTLS: |1| 0x5592cfdfc6e8: mbedtls_ssl_read_record() returned -26624 (-0x6800)
mbedTLS: |2| 0x5592cfdfc6e8: <= handshake
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
mbedTLS: |2| 0x5592cfdfc6e8: => free
mbedTLS: |2| 0x5592cfdfc6e8: <= free
itwondersteam commented 2 years ago

In addition to above, the same cert tested with MQTT mutual auth demo and is able to get connected to MQTT broker

# ... refer to above for other variables definition ... 
CLIENT_CERT_PATH="$CLAIM_CERT_PATH"
CLIENT_PRIVATE_KEY_PATH="$CLAIM_PRIVATE_KEY_PATH"

cmake -S . -Bbuild \
        -DAWS_IOT_ENDPOINT="$AWS_IOT_ENDPOINT" \
        -DROOT_CA_CERT_PATH="$ROOT_CA_CERT_PATH" \
        -DCLAIM_CERT_PATH="$CLAIM_CERT_PATH" \
        -DCLAIM_PRIVATE_KEY_PATH="$CLAIM_PRIVATE_KEY_PATH" \
        -DCLIENT_CERT_PATH="$CLIENT_CERT_PATH" \
        -DCLIENT_PRIVATE_KEY_PATH="$CLIENT_PRIVATE_KEY_PATH" \
        -DPROVISIONING_TEMPLATE_NAME="$PROVISION_NAME" \
        -DDEVICE_SERIAL_NUMBER="serial-num-testfleet$1"

cd build && make mqtt_demo_mutual_auth && ./bin/mqtt_demo_mutual_auth
[INFO] [DEMO] [mqtt_demo_mutual_auth.c:642] Establishing a TLS session to xxx-ats.iot.us-west-2.amazonaws.com:8883.
[INFO] [MQTT] [core_mqtt.c:885] Packet received. ReceivedBytes=2.
[INFO] [MQTT] [core_mqtt_serializer.c:970] CONNACK session present bit not set.
[INFO] [MQTT] [core_mqtt_serializer.c:912] Connection accepted.
[INFO] [MQTT] [core_mqtt.c:1565] Received MQTT CONNACK successfully from broker.
[INFO] [MQTT] [core_mqtt.c:1831] MQTT connection established with the broker.
[INFO] [DEMO] [mqtt_demo_mutual_auth.c:1130] MQTT connection successfully established with broke
johnrhen commented 2 years ago

Which version of the CSDK are you using? Have you made any modifications to code in the CSDK (other than the ones you made to demo_config.h and the logging macros)? Could you also set MBEDTLS_DEBUG_LOG_LEVEL in mbedtls_pkcs11_posix.h to 4 and post the logs again? This seems to be an issue with mbedTLS, considering the MQTT mutual auth demo successfully connected. We're working on recreating your issue.

As a side note, this probably isn't the root cause of the current issue, but your claim cert policy allows actions for the certificates/create/* resource, whereas it should be allowing actions for the certificates/create-from-csr/* resource. This will result in breakages later down the line. You can fix this by modifying your claim policy to replace these components:

"arn:aws:iot:us-west-2:xxx-aws-account-id:topic/$aws/certificates/create/*",

"arn:aws:iot:us-west-2:xxx-aws-account-id:topicfilter/$aws/certificates/create/*"

with these components: "arn:aws:iot:us-west-2:xxx-aws-account-id:topic/$aws/certificates/create-from-csr/*",

"arn:aws:iot:us-west-2:xxx-aws-account-id:topicfilter/$aws/certificates/create-from-csr/*"

itwondersteam commented 2 years ago

This is a new repo just cloned to try out fleet auto provisioning demo, so it's clean, except for the changed above + enable compiler option -g in CMakeLists. The last commit is currently on

f367d0bca36f99e312906d9500526cab1b45110e Update MbedTLS submodule pointer to v2.28.0

The log after modifying the MBEDTLS_DEBUG_LOG_LEVEL to 4

[INFO] [PKCS11] [core_pkcs11_mbedtls.c:1403] PKCS #11 successfully initialized.
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x3 type object.
[INFO] [PKCS11] [core_pkcs11_pal.c:63] Could not open corePKCS11_Claim_Certificate.dat for reading.
[INFO] [FLEET_PROVISIONING_DEMO] [pkcs11_operations.c:770] Writing certificate into label "Claim Cert".
[INFO] [PKCS11] [core_pkcs11_mbedtls.c:2823] Creating a 0x1 type object.
[INFO] [FLEET_PROVISIONING_DEMO] [fleet_provisioning_with_csr_demo.c:517] Establishing MQTT session with claim certificate...
[DEBUG] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:399] Establishing a TLS session to xxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com:8883.
[DEBUG] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:347] Configured MbedTLS context.
mbedTLS: |2| 0x564444a256e8: => handshake
mbedTLS: |2| 0x564444a256e8: client state: 0
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: client state: 1
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: => write client hello
mbedTLS: |3| 0x564444a256e8: client hello, max version: [3:3]
mbedTLS: |3| 0x564444a256e8: client hello, current time: 1643159509
mbedTLS: |3| 0x564444a256e8: dumping 'client hello, random bytes' (32 bytes)
mbedTLS: |3| 0x564444a256e8: 0000:  61 f0 9f d5 7e ff 9e cf 78 be 03 4a 69 09 cb 68  a...~...x..Ji..h
mbedTLS: |3| 0x564444a256e8: 0010:  12 1a fa 47 2e 59 69 a7 03 e8 33 4a 06 20 90 36  ...G.Yi...3J. .6
mbedTLS: |3| 0x564444a256e8: client hello, session id len.: 0
mbedTLS: |3| 0x564444a256e8: dumping 'client hello, session id' (0 bytes)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc00a (TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc014 (TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc02b (TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc02f (TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc023 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc027 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc009 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc013 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, got 8 ciphersuites (excluding SCSVs)
mbedTLS: |3| 0x564444a256e8: adding EMPTY_RENEGOTIATION_INFO_SCSV
mbedTLS: |3| 0x564444a256e8: client hello, compress len.: 1
mbedTLS: |3| 0x564444a256e8: client hello, compress alg.: 0
mbedTLS: |3| 0x564444a256e8: client hello, adding server name extension: xxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com
mbedTLS: |3| 0x564444a256e8: client hello, adding signature_algorithms extension
mbedTLS: |3| 0x564444a256e8: client hello, adding supported_elliptic_curves extension
mbedTLS: |3| 0x564444a256e8: client hello, adding supported_point_formats extension
mbedTLS: |3| 0x564444a256e8: client hello, adding max_fragment_length extension
mbedTLS: |3| 0x564444a256e8: client hello, adding encrypt_then_mac extension
mbedTLS: |3| 0x564444a256e8: client hello, adding extended_master_secret extension
mbedTLS: |3| 0x564444a256e8: client hello, total extension length: 96
mbedTLS: |2| 0x564444a256e8: => write handshake message
mbedTLS: |2| 0x564444a256e8: => write record
mbedTLS: |3| 0x564444a256e8: output record: msgtype = 22, version = [3:3], msglen = 159
mbedTLS: |4| 0x564444a256e8: dumping 'output record sent to network' (164 bytes)
mbedTLS: |4| 0x564444a256e8: 0000:  16 03 03 00 9f 01 00 00 9b 03 03 61 f0 9f d5 7e  ...........a...~
mbedTLS: |4| 0x564444a256e8: 0010:  ff 9e cf 78 be 03 4a 69 09 cb 68 12 1a fa 47 2e  ...x..Ji..h...G.
mbedTLS: |4| 0x564444a256e8: 0020:  59 69 a7 03 e8 33 4a 06 20 90 36 00 00 12 c0 0a  Yi...3J. .6.....
mbedTLS: |4| 0x564444a256e8: 0030:  c0 14 c0 2b c0 2f c0 23 c0 27 c0 09 c0 13 00 ff  ...+./.#.'......
mbedTLS: |4| 0x564444a256e8: 0040:  01 00 00 60 00 00 00 33 00 31 00 00 2e 61 33 30  ...`...3.1...a30
mbedTLS: |4| 0x564444a256e8: 0050:  xx xx xx xx xx xx xx xx xx xx xx 2d 61 74 73 2e  xxxxxxxxxxx-ats.
mbedTLS: |4| 0x564444a256e8: 0060:  69 6f 74 2e 75 73 2d 77 65 73 74 2d 32 2e 61 6d  iot.us-west-2.am
mbedTLS: |4| 0x564444a256e8: 0070:  61 7a 6f 6e 61 77 73 2e 63 6f 6d 00 0d 00 0a 00  azonaws.com.....
mbedTLS: |4| 0x564444a256e8: 0080:  08 04 03 04 01 03 03 03 01 00 0a 00 04 00 02 00  ................
mbedTLS: |4| 0x564444a256e8: 0090:  17 00 0b 00 02 01 00 00 01 00 01 04 00 16 00 00  ................
mbedTLS: |4| 0x564444a256e8: 00a0:  00 17 00 00                                      ....
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: message length: 164, out_left: 164
mbedTLS: |2| 0x564444a256e8: ssl->f_send() returned 164 (-0xffffff5c)
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: <= write record
mbedTLS: |2| 0x564444a256e8: <= write handshake message
mbedTLS: |2| 0x564444a256e8: <= write client hello
mbedTLS: |2| 0x564444a256e8: client state: 2
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: => parse server hello
mbedTLS: |2| 0x564444a256e8: => read record
mbedTLS: |2| 0x564444a256e8: => fetch input
mbedTLS: |2| 0x564444a256e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x564444a256e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x564444a256e8: ssl->f_recv(_timeout)() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: mbedtls_ssl_fetch_input() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: ssl_get_next_record() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: mbedtls_ssl_read_record() returned -26624 (-0x6800)
mbedTLS: |2| 0x564444a256e8: <= handshake
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
mbedTLS: |2| 0x564444a256e8: => free
mbedTLS: |2| 0x564444a256e8: <= free
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 7 ms backoff.
[DEBUG] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:399] Establishing a TLS session to xxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com:8883.
[DEBUG] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:347] Configured MbedTLS context.
mbedTLS: |2| 0x564444a256e8: => handshake
mbedTLS: |2| 0x564444a256e8: client state: 0
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: client state: 1
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: => write client hello
mbedTLS: |3| 0x564444a256e8: client hello, max version: [3:3]
mbedTLS: |3| 0x564444a256e8: client hello, current time: 1643159509
mbedTLS: |3| 0x564444a256e8: dumping 'client hello, random bytes' (32 bytes)
mbedTLS: |3| 0x564444a256e8: 0000:  61 f0 9f d5 89 0f a6 27 53 00 e1 f6 b3 19 87 a3  a......'S.......
mbedTLS: |3| 0x564444a256e8: 0010:  bb e8 40 c7 51 6d 86 cd ac 32 11 ea e7 ed 30 2a  ..@.Qm...2....0*
mbedTLS: |3| 0x564444a256e8: client hello, session id len.: 0
mbedTLS: |3| 0x564444a256e8: dumping 'client hello, session id' (0 bytes)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc00a (TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc014 (TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc02b (TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc02f (TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc023 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc027 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc009 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc013 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, got 8 ciphersuites (excluding SCSVs)
mbedTLS: |3| 0x564444a256e8: adding EMPTY_RENEGOTIATION_INFO_SCSV
mbedTLS: |3| 0x564444a256e8: client hello, compress len.: 1
mbedTLS: |3| 0x564444a256e8: client hello, compress alg.: 0
mbedTLS: |3| 0x564444a256e8: client hello, adding server name extension: xxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com
mbedTLS: |3| 0x564444a256e8: client hello, adding signature_algorithms extension
mbedTLS: |3| 0x564444a256e8: client hello, adding supported_elliptic_curves extension
mbedTLS: |3| 0x564444a256e8: client hello, adding supported_point_formats extension
mbedTLS: |3| 0x564444a256e8: client hello, adding max_fragment_length extension
mbedTLS: |3| 0x564444a256e8: client hello, adding encrypt_then_mac extension
mbedTLS: |3| 0x564444a256e8: client hello, adding extended_master_secret extension
mbedTLS: |3| 0x564444a256e8: client hello, total extension length: 96
mbedTLS: |2| 0x564444a256e8: => write handshake message
mbedTLS: |2| 0x564444a256e8: => write record
mbedTLS: |3| 0x564444a256e8: output record: msgtype = 22, version = [3:3], msglen = 159
mbedTLS: |4| 0x564444a256e8: dumping 'output record sent to network' (164 bytes)
mbedTLS: |4| 0x564444a256e8: 0000:  16 03 03 00 9f 01 00 00 9b 03 03 61 f0 9f d5 89  ...........a....
mbedTLS: |4| 0x564444a256e8: 0010:  0f a6 27 53 00 e1 f6 b3 19 87 a3 bb e8 40 c7 51  ..'S.........@.Q
mbedTLS: |4| 0x564444a256e8: 0020:  6d 86 cd ac 32 11 ea e7 ed 30 2a 00 00 12 c0 0a  m...2....0*.....
mbedTLS: |4| 0x564444a256e8: 0030:  c0 14 c0 2b c0 2f c0 23 c0 27 c0 09 c0 13 00 ff  ...+./.#.'......
mbedTLS: |4| 0x564444a256e8: 0040:  01 00 00 60 00 00 00 33 00 31 00 00 2e 61 33 30  ...`...3.1...a30
mbedTLS: |4| 0x564444a256e8: 0050:  xx xx xx xx xx xx xx xx xx xx xx 2d 61 74 73 2e  xxxxxxxxxxx-ats.
mbedTLS: |4| 0x564444a256e8: 0060:  69 6f 74 2e 75 73 2d 77 65 73 74 2d 32 2e 61 6d  iot.us-west-2.am
mbedTLS: |4| 0x564444a256e8: 0070:  61 7a 6f 6e 61 77 73 2e 63 6f 6d 00 0d 00 0a 00  azonaws.com.....
mbedTLS: |4| 0x564444a256e8: 0080:  08 04 03 04 01 03 03 03 01 00 0a 00 04 00 02 00  ................
mbedTLS: |4| 0x564444a256e8: 0090:  17 00 0b 00 02 01 00 00 01 00 01 04 00 16 00 00  ................
mbedTLS: |4| 0x564444a256e8: 00a0:  00 17 00 00                                      ....
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: message length: 164, out_left: 164
mbedTLS: |2| 0x564444a256e8: ssl->f_send() returned 164 (-0xffffff5c)
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: <= write record
mbedTLS: |2| 0x564444a256e8: <= write handshake message
mbedTLS: |2| 0x564444a256e8: <= write client hello
mbedTLS: |2| 0x564444a256e8: client state: 2
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: => parse server hello
mbedTLS: |2| 0x564444a256e8: => read record
mbedTLS: |2| 0x564444a256e8: => fetch input
mbedTLS: |2| 0x564444a256e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x564444a256e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x564444a256e8: ssl->f_recv(_timeout)() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: mbedtls_ssl_fetch_input() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: ssl_get_next_record() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: mbedtls_ssl_read_record() returned -26624 (-0x6800)
mbedTLS: |2| 0x564444a256e8: <= handshake
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
mbedTLS: |2| 0x564444a256e8: => free
mbedTLS: |2| 0x564444a256e8: <= free
[WARN] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:426] Connection to the broker failed. Retrying connection after 802 ms backoff.
[DEBUG] [FLEET_PROVISIONING_DEMO] [mqtt_operations.c:399] Establishing a TLS session to xxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com:8883.
[DEBUG] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:347] Configured MbedTLS context.
mbedTLS: |2| 0x564444a256e8: => handshake
mbedTLS: |2| 0x564444a256e8: client state: 0
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: client state: 1
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: => write client hello
mbedTLS: |3| 0x564444a256e8: client hello, max version: [3:3]
mbedTLS: |3| 0x564444a256e8: client hello, current time: 1643159510
mbedTLS: |3| 0x564444a256e8: dumping 'client hello, random bytes' (32 bytes)
mbedTLS: |3| 0x564444a256e8: 0000:  61 f0 9f d6 79 c4 c0 fa 91 1f ae 9d e2 52 60 c7  a...y........R`.
mbedTLS: |3| 0x564444a256e8: 0010:  bf 2c 23 0b 85 08 15 e9 7d e1 66 96 ec e3 6c b7  .,#.....}.f...l.
mbedTLS: |3| 0x564444a256e8: client hello, session id len.: 0
mbedTLS: |3| 0x564444a256e8: dumping 'client hello, session id' (0 bytes)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc00a (TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc014 (TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc02b (TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc02f (TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc023 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc027 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA256)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc009 (TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, add ciphersuite: 0xc013 (TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA)
mbedTLS: |3| 0x564444a256e8: client hello, got 8 ciphersuites (excluding SCSVs)
mbedTLS: |3| 0x564444a256e8: adding EMPTY_RENEGOTIATION_INFO_SCSV
mbedTLS: |3| 0x564444a256e8: client hello, compress len.: 1
mbedTLS: |3| 0x564444a256e8: client hello, compress alg.: 0
mbedTLS: |3| 0x564444a256e8: client hello, adding server name extension: xxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com
mbedTLS: |3| 0x564444a256e8: client hello, adding signature_algorithms extension
mbedTLS: |3| 0x564444a256e8: client hello, adding supported_elliptic_curves extension
mbedTLS: |3| 0x564444a256e8: client hello, adding supported_point_formats extension
mbedTLS: |3| 0x564444a256e8: client hello, adding max_fragment_length extension
mbedTLS: |3| 0x564444a256e8: client hello, adding encrypt_then_mac extension
mbedTLS: |3| 0x564444a256e8: client hello, adding extended_master_secret extension
mbedTLS: |3| 0x564444a256e8: client hello, total extension length: 96
mbedTLS: |2| 0x564444a256e8: => write handshake message
mbedTLS: |2| 0x564444a256e8: => write record
mbedTLS: |3| 0x564444a256e8: output record: msgtype = 22, version = [3:3], msglen = 159
mbedTLS: |4| 0x564444a256e8: dumping 'output record sent to network' (164 bytes)
mbedTLS: |4| 0x564444a256e8: 0000:  16 03 03 00 9f 01 00 00 9b 03 03 61 f0 9f d6 79  ...........a...y
mbedTLS: |4| 0x564444a256e8: 0010:  c4 c0 fa 91 1f ae 9d e2 52 60 c7 bf 2c 23 0b 85  ........R`..,#..
mbedTLS: |4| 0x564444a256e8: 0020:  08 15 e9 7d e1 66 96 ec e3 6c b7 00 00 12 c0 0a  ...}.f...l......
mbedTLS: |4| 0x564444a256e8: 0030:  c0 14 c0 2b c0 2f c0 23 c0 27 c0 09 c0 13 00 ff  ...+./.#.'......
mbedTLS: |4| 0x564444a256e8: 0040:  01 00 00 60 00 00 00 33 00 31 00 00 2e 61 33 30  ...`...3.1...a30
mbedTLS: |4| 0x564444a256e8: 0050:  xx xx xx xx xx xx xx xx xx xx xx 2d 61 74 73 2e  xxxxxxxxxxx-ats.
mbedTLS: |4| 0x564444a256e8: 0060:  69 6f 74 2e 75 73 2d 77 65 73 74 2d 32 2e 61 6d  iot.us-west-2.am
mbedTLS: |4| 0x564444a256e8: 0070:  61 7a 6f 6e 61 77 73 2e 63 6f 6d 00 0d 00 0a 00  azonaws.com.....
mbedTLS: |4| 0x564444a256e8: 0080:  08 04 03 04 01 03 03 03 01 00 0a 00 04 00 02 00  ................
mbedTLS: |4| 0x564444a256e8: 0090:  17 00 0b 00 02 01 00 00 01 00 01 04 00 16 00 00  ................
mbedTLS: |4| 0x564444a256e8: 00a0:  00 17 00 00                                      ....
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: message length: 164, out_left: 164
mbedTLS: |2| 0x564444a256e8: ssl->f_send() returned 164 (-0xffffff5c)
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: <= write record
mbedTLS: |2| 0x564444a256e8: <= write handshake message
mbedTLS: |2| 0x564444a256e8: <= write client hello
mbedTLS: |2| 0x564444a256e8: client state: 2
mbedTLS: |2| 0x564444a256e8: => flush output
mbedTLS: |2| 0x564444a256e8: <= flush output
mbedTLS: |2| 0x564444a256e8: => parse server hello
mbedTLS: |2| 0x564444a256e8: => read record
mbedTLS: |2| 0x564444a256e8: => fetch input
mbedTLS: |2| 0x564444a256e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x564444a256e8: in_left: 0, nb_want: 5
mbedTLS: |2| 0x564444a256e8: ssl->f_recv(_timeout)() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: mbedtls_ssl_fetch_input() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: ssl_get_next_record() returned -26624 (-0x6800)
mbedTLS: |1| 0x564444a256e8: mbedtls_ssl_read_record() returned -26624 (-0x6800)
mbedTLS: |2| 0x564444a256e8: <= handshake
[ERROR] [Transport_MbedTLS_PKCS11] [mbedtls_pkcs11_posix.c:826] Failed to perform TLS handshake: mbedTLSError= SSL - The operation timed out : <No-Low-Level-Code>.
mbedTLS: |2| 0x564444a256e8: => free
mbedTLS: |2| 0x564444a256e8: <= free

Modified Policy. Now MQTT mutual auth demo can subscribe and publish successfully with it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Resource": [
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topic/$aws/certificates/create-from-csr/*",
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topic/$aws/provisioning-templates/xxx-provision-name/provision/*",
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topic/testclient/example/topic"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": [
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topicfilter/$aws/certificates/create-from-csr/*",
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topicfilter/$aws/provisioning-templates/xxx-provision-name/provision/*",
        "arn:aws:iot:us-west-2:xxx-aws-account-id:topicfilter/testclient/example/topic"
      ]
    }
  ]
}
johnrhen commented 2 years ago

Are you able to use Wireshark or another packet analyzer to determine if you're receiving any packets from AWS when you run the Fleet Provisioning demo? This could help us determine if the issue is on the cloud side or if it's with the mbedTLS module. In the meantime, we're continuing to try and recreate your issue.

itwondersteam commented 2 years ago

Captured a portion of the packet in screenshot.

johnrhen commented 2 years ago

I've applied a fix - it seems that we had a low transport timeout in the Fleet Provisioning demo, so on networks with higher ping the demo would fail. Can you confirm that switching to the latest commit resolves your issue?

itwondersteam commented 2 years ago

@johnrhen Yap, latest commit fixes the issue flawlessly! Thank you!