The aws-apigateway-iot construct does not work as expected due to SSL error between AWS services

[StFS]

Deploying the aws-apigateway-iot solutions construct pattern doesn't work because the API Gateway cannot communicate with the IoT Data endpoint due to an SSL error.

Reproduction Steps

Deploy the aws-apigateway-iot pattern to AWS (I deployed to eu-central-1 but I've tried a few regions and they all seem to behave the same).

Try calling the API endpoint and post a message to the topic through the API Gateway "test" UI.

Observe the error that occurs and is displayed in the response (see error log below):

Error Log

[...]
Fri Oct 14 13:11:24 UTC 2022 : Sending request to https://************.iot.eu-central-1.amazonaws.com/topics/foo
Fri Oct 14 13:11:24 UTC 2022 : Execution failed due to configuration error: TLS Server certificate issued after 2019-04-16 and anchored by a distrusted legacy Symantec root CA: CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
Fri Oct 14 13:11:24 UTC 2022 : Method completed with status: 500

Environment

• CDK CLI Version : 2.45.0 (build af1fb7c)
• CDK Framework Version: 2.45.0
• AWS Solutions Constructs Version : 10.0.0
• OS : Mac
• Language : Python

Other

I also did some experiments from my dev machine to verify that the certificate used by the IoT Endpoint was strange:

❯ IOT_ENDPOINT=$(aws iot describe-endpoint | jq -r '.endpointAddress') && echo $IOT_ENDPOINT
a21qilfetw3vhh.iot.eu-central-1.amazonaws.com

❯ curl https://$IOT_ENDPOINT/topics/foo -vvI
*   Trying 3.127.182.63:443...
* Connected to a21qilfetw3vhh.iot.eu-central-1.amazonaws.com (3.127.182.63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This, of course, isn't really a bug with the CDK construct but it seems to result in the example provided in this project not working out of the box. If any workaround is possible, I would like to know about that of course.

So I would think that one of two things needs to happen here:

• Remove the aws-apigateway-iot construct as it does not behave as advertised.
• Document that this is a shortcoming of the construct (not really a solution unless some workaround exists and is documented along with it).

---

This is :bug: Bug Report

[biffgaut]

Thanks, we'll take a look

[StFS]

A colleague of mine pointed me towards testssl.sh which I ran against our IoT endpoint.

I'm attaching the results of that run but the short version is that the chain of trust is incomplete in Linux and Mozilla but okay in Microsoft, Java and Apple.

testssl.report.txt

   Chain of trust               NOT ok: Linux (chain incomplete) Mozilla (chain incomplete)
                                OK: Microsoft Java Apple

I'm not quite sure where to report this to AWS, so can you either do that for me or point me in the right direction?

[StFS]

@biffgaut you can disregard this report (although documentation could possibly be improved slightly).

Our TAM team pointed out that the IoT endpoints come in two varieties. The "plain" one is the one that the aws iot describe-endpoint returns to you and has the broken SSL certificate chain. However, there is another one which has -ats appended to the first part of the FQDN of the IoT endpoint.

This lead me to this blog post explaining where the ATS comes from.

So all I needed to do was to make sure that I added the -ats suffix and things started working.

The aws-apigateway-iot solutions construct pattern uses the ATS version of the endpoint but there is no documentation explaining what that is. I just thought that in the example used, -ats was a part of the regular endpoint hostname and didn't give it another thought and just used what aws iot describe-endpoint reported to me. Turns out that you have to append -ats to that hostname for it to have a proper cert chain that works with apigateway.

[biffgaut]

Reopening to review the docs.

[StFS]

@biffgaut I created a PR if it helps. Maybe you want to fix the wording though.

[georgebearden]

Resolved in #827