mqtt_demo_mutual_auth failure due to "SSL_connect failed to perform TLS handshake, Failed to establish a TLS connection"

[yunfeilu-dev]

Hi community,

I recently tested this sdk on my arm device and had an issue during mutual authentication.

After checking the packet capture, I found that it's client which initiate the connection closure.

From what I can tell, it seems client is not able to identify the server CA. then I checked if certificates are correctly configured using command openssl s_client -connect custom_endpoint.iot.aws-region.amazonaws.com:8443 -CAfile CA.pem -cert cert.pem -key privateKey.pem from this troubleshooting guidance. After which, I get the following result, which means the certificate is not the issue.

Any idea on what could be the problem?

Some background:

1. Certificate chain returned by server:

   

2. SDK version 202108.00

3. endpoint: use aws default iot endpoint

4. openssh version: 1.1.1h

5. iot core region: cn-northwest-1

6. cross compiler:

       Using built-in specs.
   COLLECT_GCC=arm-oe-linux-gnueabi-gcc
   gcc version 4.9.2 (GCC)

7. hardware

   processor       : 0
   model name      : ARMv7 Processor rev 5 (v7l)
   BogoMIPS        : 38.40
   Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
   CPU implementer : 0x41
   CPU architecture: 7
   CPU variant     : 0x0
   CPU part        : 0xc07
   CPU revision    : 5
   
   Hardware        : Qualcomm Technologies, Inc MDM9607
   Revision        : 0000
   Serial          : 0000000000000000
   Processor       : ARMv7 Processor rev 5 (v7l)

8. os: Linux version 3.18.20 (zlg@ubuntu) (gcc version 4.9.2 (GCC) ) #3 PREEMPT Tue Apr 13 00:40:20 PDT 2021

[jasonpcarroll]

Hi @yunfeilu-dev, Thank you for reaching out. I've reached out to the team to look into this for you.

[muneebahmed10]

Hi @yunfeilu-dev,

Can you try connecting to the MQTT broker with the same certs using another client, such as mosquitto or the AWS IoT Python SDK? E.g. with mosquitto,

mosquitto_pub -h <custom-endpoint>-ats.iot.cn-northwest-1.amazonaws.com.cn --key <private key file>.pem.key --cert <cert file>.pem.crt --cafile AmazonRootCA1.pem -i test-client -q 0 -p 8883 -t test/topic -m 'hello world!'

and change the endpoint, private key, and certificates.

[paulbartell]

@yunfeilu-dev : Looking at your logs a bit closer, it looks like you may be confusing the RootCA certificate with the Device certificate.

In your screenshot of running the SDK, "/data/cert.pem" is used at the RootCA cert and "/data/certificate.pem.crt" is used as the client certificate.

While your call to openssl s_client uses "CA.pem" as the RootCA cert and "cert.pem" as the client certificate. Can you double check that you are using the correct certificate in each case?

The following openssl command can be used to display a human-readable version of a particular certificate file:

openssl x509 -in /path/to/certificate.pem -noout -text 

[yunfeilu-dev]

Hi @yunfeilu-dev,

Can you try connecting to the MQTT broker with the same certs using another client, such as mosquitto or the AWS IoT Python SDK? E.g. with mosquitto,

mosquitto_pub -h <custom-endpoint>-ats.iot.cn-northwest-1.amazonaws.com.cn --key <private key file>.pem.key --cert <cert file>.pem.crt --cafile AmazonRootCA1.pem -i test-client -q 0 -p 8883 -t test/topic -m 'hello world!'

and change the endpoint, private key, and certificates.

Thanks for the suggestion. I tried this command, but got the follow error:

I was able to make a successful call on my linux x86 64bit vm, but not in my arm device. Based on the output error, I assume there is something wrong when I copy and paste all the credentials/certs file from my x86 vm to arm device, but I have double checked all the files are correctly copied to my machine. Any more suggestion on this?

Btw, I also did

openssl x509 -noout -modulus -in certificate.crt | openssl md5
openssl rsa -noout -modulus -in privateKey.key | openssl md5

and the output md5 strings are identical

[yunfeilu-dev]

@yunfeilu-dev : Looking at your logs a bit closer, it looks like you may be confusing the RootCA certificate with the Device certificate.

In your screenshot of running the SDK, "/data/cert.pem" is used at the RootCA cert and "/data/certificate.pem.crt" is used as the client certificate.

While your call to openssl s_client uses "CA.pem" as the RootCA cert and "cert.pem" as the client certificate. Can you double check that you are using the correct certificate in each case?

The following openssl command can be used to display a human-readable version of a particular certificate file:

openssl x509 -in /path/to/certificate.pem -noout -text 

Hello, thanks for the suggestion. I have double checked the device cert/rootCA/private key to ensure that I put the right thing in the right place, but the issue still exists.

[paulbartell]

@yunfeilu-dev , The error you received from your call to mosquitto_pub indicates a mismatch between your private key and certificate.

I noticed that the filenames in your previous message don't quite match up with each other. The easiest way forward is probably re-generating your key / certificate pair so that the names are clear.

Here's an example of doing so using the aws cli:

aws iot create-keys-and-certificate --certificate-pem-outfile device_cert.pem --public-key-outfile device_key.pubkey --private-key-outfile device_key.pem

Note that device certificates issued by AWS IoT Core are not issued by a CA traceable to an ATS Certificate Authority.

If you know the certificate id of your existing certificate (part of the arn), you can get a copy of the certificate from AWS IoT core with the following command:

aws iot describe-certificate --certificate-id=<cert id>

If you don't know the id of your certificate, you can use the following cli command to list all of the certificate-id registered to your account.

aws iot list-certificates

Alternatively, you can create a self-signed key / cert pair and verify that the two match.

# Generate a new RSA key and self signed cert
openssl req -nodes -x509 -newkey rsa:4096 -keyout device_key.pem -out device_cert.pem -sha256 -days 365 -subj "/C=US/ST=WA/L=Seattle/O=Amazon Web Services/CN=MyDeviceName"
Generating a 4096 bit RSA private key
writing new private key to 'device_key.pem'

# Export the public key from the private key:
openssl rsa -in device_key.pem -pubout -out device_key.pubkey

# Export the public key from the certificate:
openssl x509 -in device_cert.pem -noout -pubkey > device_cert.pubkey

# Compare the two public keys
diff device_cert.pubkey device_key.pubkey

# Output pub keys in human-readable form for manual comparison
openssl rsa -pubin -noout -text -in device_cert.pubkey
openssl rsa -pubin -noout -text -in device_key.pubkey

# Optionally, register this certificate with IoT core and use it to authenticate your device.
aws iot register-certificate-without-ca --certificate-pem file://device_cert.pem

You can also test the combination of private key / certificate by signing some random data and verifying it with the public key exported from the certificate:

# Generate some random data:
dd if=/dev/urandom count=1024 | base64 > verifiable_message.txt

# Sign the contents of "verifiable_message.txt" with device_key.pem
openssl dgst -sign device_key.pem -out verifiable_message.sig verifiable_message.txt

# Check against private key
openssl dgst -prverify device_key.pem -signature verifiable_message.sig verifiable_message.txt

# Check against certificate derived private key
openssl dgst -verify device_cert.pubkey -signature verifiable_message.sig verifiable_message.txt

On your device, you can run the follow commands to inspect each of the files:

# for the ca certificate:
openssl x509 -in AmazonRootCA1.pem -noout -text

# For the device certificate:
openssl x509 -in device_cert.pem -noout -text

# For the device private key:
openssl rsa -in device_key.pem -noout -text

More information on the aws cli is available on the AWS website.

[pvyawaha]

@yunfeilu-dev ,

Did Paul's answer helped you resolve the issue or you still have question on this topic?

[abhidixi11]

Hello @yunfeilu-dev, I'm closing this issue. If you still have questions please feel free to open a new issue. Thanks.

[rakesh5283]

Hello,

I am facing same problem while SSL_Connect() API Called @yunfeilu-dev, did you solve the problem ?

If I used following command it works for me and i am able to see the message on AWS IoT Core Test Client window.

mosquitto_pub -h -ats.iot.ap-south-1.amazonaws.com --key privapem.key --cert certipem.crt --cafile RootCA1.pem -q 0 -p 8883 -t testclient/example/topic -m "Jay Swaminarayan Bhagat"

But when using ARM7 hardware and AWS_IoT_Core_embedded_C SDK it stuck to SSL_Connect() API.

[kar-rahul-aws]

Hi @rakesh5283 Can you check if @paulbartell 's answer above will help in resolving the issue you are facing?

[rakesh5283]

@kar-rahul-aws

I am still facing this error . Any clue?