stephaneey
commented
1 month ago A little update on this issue. I created my own module and ended up in the same situation. It is clearly a chain validation issue but it seems it is only happening with .NET (the blob storage module is written in .NET as the Temperature Sensor one). So, I crafted my own module and I hooked a custom certificate chain validation using the mqtt settings:
mqttSetting.RemoteCertificateValidationCallback = ValidateServerCertificate;
ITransportSettings[] settings = { mqttSetting };In the validation method, I print out everything I can about the incoming server (edge gateway) certificate:
private static bool ValidateServerCertificate(
object sender,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors sslPolicyErrors)
{
Console.WriteLine("---------------IN CERT VALIDATION------------------------");
if (sslPolicyErrors == SslPolicyErrors.None)
{
Console.WriteLine("No SSL policy errors.");
return true; // No errors
}
Console.WriteLine("Certificate Information:");
Console.WriteLine($"Subject: {certificate.Subject}");
Console.WriteLine($"Issuer: {certificate.Issuer}");
Console.WriteLine($"Effective Date: {certificate.GetEffectiveDateString()}");
Console.WriteLine($"Expiration Date: {certificate.GetExpirationDateString()}");
Console.WriteLine($"Thumbprint: {certificate.GetCertHashString()}");
Console.WriteLine($"Serial Number: {certificate.GetSerialNumberString()}");
if (chain != null)
{
Console.WriteLine("Chain Information:");
foreach (var element in chain.ChainElements)
{
Console.WriteLine($"Element Certificate Subject: {element.Certificate.Subject}");
Console.WriteLine($"Element Certificate Issuer: {element.Certificate.Issuer}");
Console.WriteLine($"Element Certificate Thumbprint: {element.Certificate.GetCertHashString()}");
Console.WriteLine($"Element Certificate Serial Number: {element.Certificate.GetSerialNumberString()}");
Console.WriteLine("Element Certificate Errors:");
foreach (var status in element.ChainElementStatus)
{
Console.WriteLine($"Status: {status.Status}, Status Information: {status.StatusInformation}");
}
}
}
Console.WriteLine($"SSL Policy Errors: {sslPolicyErrors}");
if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateNameMismatch) != 0)
{
Console.WriteLine("Certificate name mismatch.");
// Handle name mismatch error
}
if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateNotAvailable) != 0)
{
Console.WriteLine("Certificate not available.");
// Handle missing certificate error
}
Console.WriteLine("---------------END OF CERT VALIDATION------------------------");
// just for troubleshooting purposes
return true;
}and I get this back:
Certificate Information:
Subject: CN=ubuntugtw
Issuer: CN=edgeca.ca
Effective Date: 07/12/2024 05:19:50
Expiration Date: 08/04/2024 15:06:02
Thumbprint: F69844BE1F56A679D8FD0359DE88FF771B348C68
Serial Number: 00
Chain Information:
Element Certificate Subject: CN=ubuntugtw
Element Certificate Issuer: CN=edgeca.ca
Element Certificate Thumbprint: F69844BE1F56A679D8FD0359DE88FF771B348C68
Element Certificate Serial Number: 00
Element Certificate Errors:
Element Certificate Subject: CN=edgeca.ca
Element Certificate Issuer: CN=Azure_IoT_Hub_Intermediate_Cert_Test_Only
Element Certificate Thumbprint: 3EEABC6707475CCA0E7FD1489FDD0606CED56C7E
Element Certificate Serial Number: 1002
Element Certificate Errors:
Status: PartialChain, Status Information: unable to get issuer certificate
SSL Policy Errors: RemoteCertificateChainErrors
---------------END OF CERT VALIDATION------------------------Which in the end confirms the initial exception I had with the marketplace modules.
However, when I use openssl directly from within the container and connect to the exact same endpoint, it doesn't reveal any problem:
$ openssl s_client --connect ubuntugtw:8883
CONNECTED(00000003)
Can't use SSL_get_servername
depth=3 CN = Azure_IoT_Hub_CA_Cert_Test_Only
verify return:1
depth=2 CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
verify return:1
depth=1 CN = edgeca.ca
verify return:1
depth=0 CN = ubuntugtw
verify return:1
---
Certificate chain
0 s:CN = ubuntugtw
i:CN = edgeca.ca
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 12 05:19:50 2024 GMT; NotAfter: Aug 4 15:06:02 2024 GMT
1 s:CN = edgeca.ca
i:CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 5 15:06:02 2024 GMT; NotAfter: Aug 4 15:06:02 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEGTCCAgGgAwIBAgIBADANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDDAllZGdl
Y2EuY2EwHhcNMjQwNzEyMDUxOTUwWhcNMjQwODA0MTUwNjAyWjAUMRIwEAYDVQQD
DAl1YnVudHVndHcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8ixi/
cW6b3Ce5XcOh5Y16sDE8L8tJIOoXYCZwl4TmI/7LEv7/EIzOYMi0xdP5gxXY6CUT
NFgRyQp+x6gtSX03ukeCeUhAouh6sRAKqpwJVh0F5WoJYHQQl2Y/lBgOH04QGO4Q
md3LsKFNQk1NIBNVeIaU/oiIKv/nfNx48eUneYtHiPAcZEytKSwpX9mTeBxRrmUH
dewjdPkGo2iwgRXJSw5QKyGeqPTppbRVZELzUJjj/BJAiWueft0WgdG45f2bqH73
CwO8yp2mda1kw3KSZoQ+jSpfUCQfnX2bdLbXZ1AL/x3owyZbNyCnI4NRseP0q1R6
mxUbdjD1bUKOToQnAgMBAAGjdjB0MBMGA1UdJQQMMAoGCCsGAQUFBwMBMB0GA1Ud
EQQWMBSCCXVidW50dWd0d4IHZWRnZUh1YjAdBgNVHQ4EFgQUdyDNHXXHcLJvCMz/
dEKxkJyJaTwwHwYDVR0jBBgwFoAULoayDfbj0FA1Hfk8A3xCFj1kJ3kwDQYJKoZI
hvcNAQELBQADggIBABvHy+TjkqUZR9TW9R1I1wACLxkzrbMmxjqeadZD1z+VCZhn
xvwE6uoDECCLa/JE7E5qj1iPQCGy64CpK6yMRun9SB66hyU9jDNf3ypTSJbWXLhj
BQ4u9OZIp39+pnHLt/4NGYSxMyQP4Bhqf/pYV0tdXay7bjOFVUd27fKwwg8KxLM8
BQTAHsA9I1NyG6936kZcNBvanBCAZpLqzKBCkYc4QRkULHYLLqsWDI5zpYtqT5Nc
ccvIn5ga8AgeaYbivbg9NlzkQ87rgtLdBRVo+uDihJlojagqi8A4WRUqWeprBhZF
WTEqE5Ky33iSJcjrv5m1VP7MJkvhNTQYKKolAEVz0tskctM+1tzFAGlU6XcTytHJ
YOr9TvNrxMyUYfJ3KD3pTlKNBPTa4c0Kd02o3+z5g4+KPU4I6VN6eimVjkWIFx4O
3RXA6D3oirs59JMS8y1hVKn26yNhT6sVcYdNyAIBG/3wtJnTtlA8POZCFQxd49II
F+/O3RunM2a4HS91UqAaTg96yaFA5l1O76Pt6m6/3tCrhoCnYxhQuiK02tnE5VSD
LyH9SKRHTsmyIEKe9qkrRohDbUwWaAScsAvEil+0GbCcFARC4lhGrJmvCuB1rDnn
ffpjcW5PUuhO7o04/dV9NOq/DEQeKMwWE8UBwi+o8FQMSXEASVBHmCnOnHkd
-----END CERTIFICATE-----
subject=CN = ubuntugtw
issuer=CN = edgeca.ca
---
No client certificate CA names sent
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3058 bytes and written 407 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: BE9EE5B0A07F531F6983FB8D15FE1FA81D3DBDCF5204186F09744284EC2DB5C1
Session-ID-ctx:
Resumption PSK: E0242A120E4873D8223AE6FBB51C060A4B0942FB7DE5284C0AEC5D73C9D0534DB0DD1D2A130B3BA7E141A82010F96DFF
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - d7 dd 77 40 51 07 99 35-c2 86 2b 7a 29 21 c7 a2 ..w@Q..5..+z)!..
0010 - c0 af bd 85 a1 87 05 61-1d 66 eb 26 57 4b 36 bf .......a.f.&WK6.
0020 - 0b 07 73 c2 bc 1c e6 99-55 78 01 5b 78 0d ba eb ..s.....Ux.[x...
0030 - 4c 56 9f d3 5e 43 68 ad-d8 bb 62 82 29 b6 7b 7d LV..^Ch...b.).{}
0040 - 56 ed c5 a1 da 8e 20 51-26 5b a0 25 59 27 5b e0 V..... Q&[.%Y'[.
0050 - 3b e8 a5 0d 94 2e e0 59-8e ea 71 a6 11 d6 46 b8 ;......Y..q...F.
0060 - c4 7e 6b 6b cb 5c 3f c5-e4 23 db e7 1b fb a3 9f .~kk.\?..#......
0070 - 87 d2 26 57 c8 e6 13 d7-d1 ee d9 58 1a d6 67 6d ..&W.......X..gm
0080 - 80 30 dc d4 e7 9f 3e d5-9a 9c ce cb 3d f7 13 5b .0....>.....=..[
0090 - 61 6b 55 02 29 cb a1 2c-c8 a1 d2 44 39 d9 c2 dc akU.)..,...D9...
00a0 - 0f 41 cb 46 b7 69 24 58-ba 5f 91 f5 d5 34 1c d0 .A.F.i$X._...4..
00b0 - f4 48 ee 90 26 66 70 11-95 00 6e f5 fb 2d dc e5 .H..&fp...n..-..
00c0 - 86 c4 5f 5f 68 9b 46 35-7d d0 46 72 56 ff 05 5d ..__h.F5}.FrV..]
00d0 - 6f e1 a7 ea f3 37 5b e9-5d bb 9b 29 54 4b a4 54 o....7[.]..)TK.T
00e0 - 48 56 ba b3 f5 6c 08 99-6c 95 bb b8 73 0a 42 7e HV...l..l...s.B~
Start Time: 1720772115
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 5DAD4589FD05C632AC12A82C86BA3C047DEC53B2F6A9F6B7B03AE819179E69CE
Session-ID-ctx:
Resumption PSK: 31157DF2A906FA9FE0C925C88EC59F8C1A726B9205C7C2BBC0ABA28964E66636BDB9E3DD3DF95B50D6278B09843496FE
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - d7 dd 77 40 51 07 99 35-c2 86 2b 7a 29 21 c7 a2 ..w@Q..5..+z)!..
0010 - 7d b0 a7 25 26 28 6b 2f-d9 0e 9a fa 19 1c be 88 }..%&(k/........
0020 - 29 24 c5 e3 42 0b 72 6b-5a 7b 6a 20 c6 2a a8 c4 )$..B.rkZ{j .*..
0030 - cd b2 ed e6 76 15 02 29-27 a3 13 6f 9d ac dc 13 ....v..)'..o....
0040 - fc 04 e4 22 21 85 ef 24-09 38 d9 14 61 3c 83 9a ..."!..$.8..a<..
0050 - de 85 fa 5f ed 7e 69 6f-10 ab 93 a6 1c 1f e2 14 ..._.~io........
0060 - cd 93 8f 4e de 4f 66 68-4b 0b 07 41 1e 14 19 9e ...N.OfhK..A....
0070 - 77 c4 97 23 c4 bd cc 05-e2 48 cb 1e 93 a6 8d 38 w..#.....H.....8
0080 - 7a b7 b9 18 0e 2c 99 be-42 6e 00 00 4d ed 5e 17 z....,..Bn..M.^.
0090 - ee d8 b9 f1 0b 60 8c b3-95 9f 57 91 d5 28 f8 b7 .....`....W..(..
00a0 - 09 f2 c9 18 e4 1c f6 d4-62 68 bf e4 13 af c8 02 ........bh......
00b0 - f6 b5 6a 48 5f d3 67 a0-ab 28 3d ad a8 ac c8 11 ..jH_.g..(=.....
00c0 - 8b fd 23 78 4c c7 d9 92-60 3d ff eb 3e c7 d0 73 ..#xL...`=..>..s
00d0 - 0c e5 68 0e ad 35 31 23-e2 87 16 e2 71 83 f2 bd ..h..51#....q...
00e0 - 48 1d 3f 09 1d 39 96 a7-11 41 8c 30 25 f0 70 8d H.?..9...A.0%.p.
Start Time: 1720772115
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCKSimilarly, curl -v also doesn't have any problem:
$ curl -v https://ubuntugtw
* Trying 172.18.0.3:443...
* Connected to ubuntugtw (172.18.0.3) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=ubuntugtw
* start date: Jul 12 05:19:50 2024 GMT
* expire date: Aug 4 15:06:02 2024 GMT
* subjectAltName: host "ubuntugtw" matched cert's "ubuntugtw"
* issuer: CN=edgeca.ca
* SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: ubuntugtw]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x6530206e2ce0)
> GET / HTTP/2
> Host: ubuntugtw
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 404
< date: Fri, 12 Jul 2024 08:25:41 GMT
< server: Kestrel
< content-length: 0
< x-content-type-options: nosniff
<
* Connection #0 to host ubuntugtw left intact
So, I'm not sure how to fix this!
lfitchett
Expected Behavior
ModuleClient should be able to connect to IoTHub without issue.
Current Behavior
TLS Exception is encountered with whatever module. Sending regular messages from child devices through the IoT Edge Runtime in Transparent Gateway mode works, only the module part does not.
Here is a sample exception from the Temperature Sensor module:
Steps to Reproduce
Provide a detailed set of steps to reproduce the bug.
Context (Environment)
Output of
iotedge checkOutput of
iotedge system statusDevice Information
Runtime Versions
docker version]:Logs
Here is the entire support bundle zip for the past hour that includes the above mentioned exception but I couldn't find anything else relevant.
https://seyiot.blob.core.windows.net/logs/support_bundle_2024_07_11_05_46_40_UTC.zip
Additional Information
Certificates are not expired. I've seen many issues similar to this one where the Edge CA renewal was not scheduled. I'm not in this situation (as visible in the attached support bundle).