Closed stephaneey closed 1 month ago
A little update on this issue. I created my own module and ended up in the same situation. It is clearly a chain validation issue but it seems it is only happening with .NET (the blob storage module is written in .NET as the Temperature Sensor one). So, I crafted my own module and I hooked a custom certificate chain validation using the mqtt settings:
mqttSetting.RemoteCertificateValidationCallback = ValidateServerCertificate;
ITransportSettings[] settings = { mqttSetting };
In the validation method, I print out everything I can about the incoming server (edge gateway) certificate:
private static bool ValidateServerCertificate(
object sender,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors sslPolicyErrors)
{
Console.WriteLine("---------------IN CERT VALIDATION------------------------");
if (sslPolicyErrors == SslPolicyErrors.None)
{
Console.WriteLine("No SSL policy errors.");
return true; // No errors
}
Console.WriteLine("Certificate Information:");
Console.WriteLine($"Subject: {certificate.Subject}");
Console.WriteLine($"Issuer: {certificate.Issuer}");
Console.WriteLine($"Effective Date: {certificate.GetEffectiveDateString()}");
Console.WriteLine($"Expiration Date: {certificate.GetExpirationDateString()}");
Console.WriteLine($"Thumbprint: {certificate.GetCertHashString()}");
Console.WriteLine($"Serial Number: {certificate.GetSerialNumberString()}");
if (chain != null)
{
Console.WriteLine("Chain Information:");
foreach (var element in chain.ChainElements)
{
Console.WriteLine($"Element Certificate Subject: {element.Certificate.Subject}");
Console.WriteLine($"Element Certificate Issuer: {element.Certificate.Issuer}");
Console.WriteLine($"Element Certificate Thumbprint: {element.Certificate.GetCertHashString()}");
Console.WriteLine($"Element Certificate Serial Number: {element.Certificate.GetSerialNumberString()}");
Console.WriteLine("Element Certificate Errors:");
foreach (var status in element.ChainElementStatus)
{
Console.WriteLine($"Status: {status.Status}, Status Information: {status.StatusInformation}");
}
}
}
Console.WriteLine($"SSL Policy Errors: {sslPolicyErrors}");
if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateNameMismatch) != 0)
{
Console.WriteLine("Certificate name mismatch.");
// Handle name mismatch error
}
if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateNotAvailable) != 0)
{
Console.WriteLine("Certificate not available.");
// Handle missing certificate error
}
Console.WriteLine("---------------END OF CERT VALIDATION------------------------");
// just for troubleshooting purposes
return true;
}
and I get this back:
Certificate Information:
Subject: CN=ubuntugtw
Issuer: CN=edgeca.ca
Effective Date: 07/12/2024 05:19:50
Expiration Date: 08/04/2024 15:06:02
Thumbprint: F69844BE1F56A679D8FD0359DE88FF771B348C68
Serial Number: 00
Chain Information:
Element Certificate Subject: CN=ubuntugtw
Element Certificate Issuer: CN=edgeca.ca
Element Certificate Thumbprint: F69844BE1F56A679D8FD0359DE88FF771B348C68
Element Certificate Serial Number: 00
Element Certificate Errors:
Element Certificate Subject: CN=edgeca.ca
Element Certificate Issuer: CN=Azure_IoT_Hub_Intermediate_Cert_Test_Only
Element Certificate Thumbprint: 3EEABC6707475CCA0E7FD1489FDD0606CED56C7E
Element Certificate Serial Number: 1002
Element Certificate Errors:
Status: PartialChain, Status Information: unable to get issuer certificate
SSL Policy Errors: RemoteCertificateChainErrors
---------------END OF CERT VALIDATION------------------------
Which in the end confirms the initial exception I had with the marketplace modules.
However, when I use openssl directly from within the container and connect to the exact same endpoint, it doesn't reveal any problem:
$ openssl s_client --connect ubuntugtw:8883
CONNECTED(00000003)
Can't use SSL_get_servername
depth=3 CN = Azure_IoT_Hub_CA_Cert_Test_Only
verify return:1
depth=2 CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
verify return:1
depth=1 CN = edgeca.ca
verify return:1
depth=0 CN = ubuntugtw
verify return:1
---
Certificate chain
0 s:CN = ubuntugtw
i:CN = edgeca.ca
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 12 05:19:50 2024 GMT; NotAfter: Aug 4 15:06:02 2024 GMT
1 s:CN = edgeca.ca
i:CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 5 15:06:02 2024 GMT; NotAfter: Aug 4 15:06:02 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEGTCCAgGgAwIBAgIBADANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDDAllZGdl
Y2EuY2EwHhcNMjQwNzEyMDUxOTUwWhcNMjQwODA0MTUwNjAyWjAUMRIwEAYDVQQD
DAl1YnVudHVndHcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8ixi/
cW6b3Ce5XcOh5Y16sDE8L8tJIOoXYCZwl4TmI/7LEv7/EIzOYMi0xdP5gxXY6CUT
NFgRyQp+x6gtSX03ukeCeUhAouh6sRAKqpwJVh0F5WoJYHQQl2Y/lBgOH04QGO4Q
md3LsKFNQk1NIBNVeIaU/oiIKv/nfNx48eUneYtHiPAcZEytKSwpX9mTeBxRrmUH
dewjdPkGo2iwgRXJSw5QKyGeqPTppbRVZELzUJjj/BJAiWueft0WgdG45f2bqH73
CwO8yp2mda1kw3KSZoQ+jSpfUCQfnX2bdLbXZ1AL/x3owyZbNyCnI4NRseP0q1R6
mxUbdjD1bUKOToQnAgMBAAGjdjB0MBMGA1UdJQQMMAoGCCsGAQUFBwMBMB0GA1Ud
EQQWMBSCCXVidW50dWd0d4IHZWRnZUh1YjAdBgNVHQ4EFgQUdyDNHXXHcLJvCMz/
dEKxkJyJaTwwHwYDVR0jBBgwFoAULoayDfbj0FA1Hfk8A3xCFj1kJ3kwDQYJKoZI
hvcNAQELBQADggIBABvHy+TjkqUZR9TW9R1I1wACLxkzrbMmxjqeadZD1z+VCZhn
xvwE6uoDECCLa/JE7E5qj1iPQCGy64CpK6yMRun9SB66hyU9jDNf3ypTSJbWXLhj
BQ4u9OZIp39+pnHLt/4NGYSxMyQP4Bhqf/pYV0tdXay7bjOFVUd27fKwwg8KxLM8
BQTAHsA9I1NyG6936kZcNBvanBCAZpLqzKBCkYc4QRkULHYLLqsWDI5zpYtqT5Nc
ccvIn5ga8AgeaYbivbg9NlzkQ87rgtLdBRVo+uDihJlojagqi8A4WRUqWeprBhZF
WTEqE5Ky33iSJcjrv5m1VP7MJkvhNTQYKKolAEVz0tskctM+1tzFAGlU6XcTytHJ
YOr9TvNrxMyUYfJ3KD3pTlKNBPTa4c0Kd02o3+z5g4+KPU4I6VN6eimVjkWIFx4O
3RXA6D3oirs59JMS8y1hVKn26yNhT6sVcYdNyAIBG/3wtJnTtlA8POZCFQxd49II
F+/O3RunM2a4HS91UqAaTg96yaFA5l1O76Pt6m6/3tCrhoCnYxhQuiK02tnE5VSD
LyH9SKRHTsmyIEKe9qkrRohDbUwWaAScsAvEil+0GbCcFARC4lhGrJmvCuB1rDnn
ffpjcW5PUuhO7o04/dV9NOq/DEQeKMwWE8UBwi+o8FQMSXEASVBHmCnOnHkd
-----END CERTIFICATE-----
subject=CN = ubuntugtw
issuer=CN = edgeca.ca
---
No client certificate CA names sent
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3058 bytes and written 407 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: BE9EE5B0A07F531F6983FB8D15FE1FA81D3DBDCF5204186F09744284EC2DB5C1
Session-ID-ctx:
Resumption PSK: E0242A120E4873D8223AE6FBB51C060A4B0942FB7DE5284C0AEC5D73C9D0534DB0DD1D2A130B3BA7E141A82010F96DFF
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - d7 dd 77 40 51 07 99 35-c2 86 2b 7a 29 21 c7 a2 ..w@Q..5..+z)!..
0010 - c0 af bd 85 a1 87 05 61-1d 66 eb 26 57 4b 36 bf .......a.f.&WK6.
0020 - 0b 07 73 c2 bc 1c e6 99-55 78 01 5b 78 0d ba eb ..s.....Ux.[x...
0030 - 4c 56 9f d3 5e 43 68 ad-d8 bb 62 82 29 b6 7b 7d LV..^Ch...b.).{}
0040 - 56 ed c5 a1 da 8e 20 51-26 5b a0 25 59 27 5b e0 V..... Q&[.%Y'[.
0050 - 3b e8 a5 0d 94 2e e0 59-8e ea 71 a6 11 d6 46 b8 ;......Y..q...F.
0060 - c4 7e 6b 6b cb 5c 3f c5-e4 23 db e7 1b fb a3 9f .~kk.\?..#......
0070 - 87 d2 26 57 c8 e6 13 d7-d1 ee d9 58 1a d6 67 6d ..&W.......X..gm
0080 - 80 30 dc d4 e7 9f 3e d5-9a 9c ce cb 3d f7 13 5b .0....>.....=..[
0090 - 61 6b 55 02 29 cb a1 2c-c8 a1 d2 44 39 d9 c2 dc akU.)..,...D9...
00a0 - 0f 41 cb 46 b7 69 24 58-ba 5f 91 f5 d5 34 1c d0 .A.F.i$X._...4..
00b0 - f4 48 ee 90 26 66 70 11-95 00 6e f5 fb 2d dc e5 .H..&fp...n..-..
00c0 - 86 c4 5f 5f 68 9b 46 35-7d d0 46 72 56 ff 05 5d ..__h.F5}.FrV..]
00d0 - 6f e1 a7 ea f3 37 5b e9-5d bb 9b 29 54 4b a4 54 o....7[.]..)TK.T
00e0 - 48 56 ba b3 f5 6c 08 99-6c 95 bb b8 73 0a 42 7e HV...l..l...s.B~
Start Time: 1720772115
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 5DAD4589FD05C632AC12A82C86BA3C047DEC53B2F6A9F6B7B03AE819179E69CE
Session-ID-ctx:
Resumption PSK: 31157DF2A906FA9FE0C925C88EC59F8C1A726B9205C7C2BBC0ABA28964E66636BDB9E3DD3DF95B50D6278B09843496FE
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - d7 dd 77 40 51 07 99 35-c2 86 2b 7a 29 21 c7 a2 ..w@Q..5..+z)!..
0010 - 7d b0 a7 25 26 28 6b 2f-d9 0e 9a fa 19 1c be 88 }..%&(k/........
0020 - 29 24 c5 e3 42 0b 72 6b-5a 7b 6a 20 c6 2a a8 c4 )$..B.rkZ{j .*..
0030 - cd b2 ed e6 76 15 02 29-27 a3 13 6f 9d ac dc 13 ....v..)'..o....
0040 - fc 04 e4 22 21 85 ef 24-09 38 d9 14 61 3c 83 9a ..."!..$.8..a<..
0050 - de 85 fa 5f ed 7e 69 6f-10 ab 93 a6 1c 1f e2 14 ..._.~io........
0060 - cd 93 8f 4e de 4f 66 68-4b 0b 07 41 1e 14 19 9e ...N.OfhK..A....
0070 - 77 c4 97 23 c4 bd cc 05-e2 48 cb 1e 93 a6 8d 38 w..#.....H.....8
0080 - 7a b7 b9 18 0e 2c 99 be-42 6e 00 00 4d ed 5e 17 z....,..Bn..M.^.
0090 - ee d8 b9 f1 0b 60 8c b3-95 9f 57 91 d5 28 f8 b7 .....`....W..(..
00a0 - 09 f2 c9 18 e4 1c f6 d4-62 68 bf e4 13 af c8 02 ........bh......
00b0 - f6 b5 6a 48 5f d3 67 a0-ab 28 3d ad a8 ac c8 11 ..jH_.g..(=.....
00c0 - 8b fd 23 78 4c c7 d9 92-60 3d ff eb 3e c7 d0 73 ..#xL...`=..>..s
00d0 - 0c e5 68 0e ad 35 31 23-e2 87 16 e2 71 83 f2 bd ..h..51#....q...
00e0 - 48 1d 3f 09 1d 39 96 a7-11 41 8c 30 25 f0 70 8d H.?..9...A.0%.p.
Start Time: 1720772115
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
Similarly, curl -v also doesn't have any problem:
$ curl -v https://ubuntugtw
* Trying 172.18.0.3:443...
* Connected to ubuntugtw (172.18.0.3) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=ubuntugtw
* start date: Jul 12 05:19:50 2024 GMT
* expire date: Aug 4 15:06:02 2024 GMT
* subjectAltName: host "ubuntugtw" matched cert's "ubuntugtw"
* issuer: CN=edgeca.ca
* SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: ubuntugtw]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x6530206e2ce0)
> GET / HTTP/2
> Host: ubuntugtw
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 404
< date: Fri, 12 Jul 2024 08:25:41 GMT
< server: Kestrel
< content-length: 0
< x-content-type-options: nosniff
<
* Connection #0 to host ubuntugtw left intact
So, I'm not sure how to fix this!
@stephaneey The edgehub server root certificate would be in the trust bundle that is provided to the modules, not in the os trusted roots, so it is surprising that the curl/openssl requests succeeded.
This looks to be an issue with the modules not reading the trust bundle. Did you ever try simulated temperature sensor without any custom validation code? I don't think your custom validation logic is referencing it at all.
The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
Here's some info on the trust bundle, and note in the example when openssl -connect is used it passes in the trust bundle file: https://learn.microsoft.com/en-us/azure/iot-edge/iot-edge-certs?view=iotedge-1.5#self-signed-root-ca-specificity
Hi @lfitchett ,
Thanks for your answer but let me clarify a few things:
The link you referred to is talking about the Device, not about a custom/marketplace module (container). As stated earlier, I got that part working.
The only reason why I have my own RemoteCertificateValidationCallack is to debug and troubleshoot. As stated earlier I dump as much info as possible about the problematic certificate.
openssl and curl succeed because I have injected the certificates into the cert store of the container by crafting my own container image. Here is an extract of my Dockerfile:
COPY iot-edge-device-ca-edgeca.cert.crt /usr/local/share/ca-certificates/iot-edge-device-ca-edgeca.cert.crt
COPY azure-iot-test-only.intermediate-full-chain.cert.crt /usr/local/share/ca-certificates/azure-iot-test-only.intermediate-full-chain.cert.crt
COPY azure-iot-test-only.root.ca.cert.crt /usr/local/share/ca-certificates/azure-iot-test-only.root.ca.cert.crt
RUN apt-get update \
&& apt-get install -y ca-certificates \
&& update-ca-certificates --fresh\
&& rm -rf /var/lib/apt/lists/* \
&& chown -R app:app /app
So, the only real questions are:
Thanks Best Regards
Hi @lfitchett,
Although the article you mentioned is talking about the device, not modules, you still pointed me to the right direction. I already tried to do what was done in the article but there was an important flaw in my config: my trusted bundle was not valid. So, in my config.toml, I did this (shown in the first part of the issue):
trust_bundle_cert = "file:///var/secrets/azure-iot-test-only.root.ca.cert.pem"
where I should have been doing this:
trust_bundle_cert = "file:///var/secrets/iot-edge-device-identity-edgedevice-full-chain.cert.pem"
My bundle was actually not a bundle....I changed this in the config.toml, followed by sudo iotedge config apply and voilà! So, long story short, no need to inject any cert or to pass any CA to the module container, just use the right bundle.
I got probably confused by the fact that everything was working fine...where we could have expected that passing a bundle that is not actually a bundle would have caused the iotedge check to fail or at least to warn about this. In any case, thanks!
Cheers
Expected Behavior
ModuleClient should be able to connect to IoTHub without issue.
Current Behavior
TLS Exception is encountered with whatever module. Sending regular messages from child devices through the IoT Edge Runtime in Transparent Gateway mode works, only the module part does not.
Here is a sample exception from the Temperature Sensor module:
Steps to Reproduce
Provide a detailed set of steps to reproduce the bug.
Context (Environment)
Output of
iotedge check
Output of
iotedge system status
Device Information
Runtime Versions
docker version
]:Logs
Here is the entire support bundle zip for the past hour that includes the above mentioned exception but I couldn't find anything else relevant.
https://seyiot.blob.core.windows.net/logs/support_bundle_2024_07_11_05_46_40_UTC.zip
Additional Information
Certificates are not expired. I've seen many issues similar to this one where the Edge CA renewal was not scheduled. I'm not in this situation (as visible in the attached support bundle).