dart-lang / http

A composable API for making HTTP requests in Dart.
https://pub.dev/packages/http
BSD 3-Clause "New" or "Revised" License
1.01k stars 351 forks source link

HandshakeException: Handshake error in client (OS Error: CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:359)) #627

Open mryogi1976 opened 2 years ago

mryogi1976 commented 2 years ago

HandshakeException: Handshake error in client (OS Error: CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:359))

I have 4 apps live on Google Play Store from last few months , yesterday all stopped working because of above error.

I did not understand the cause of above error and no solution found til now.

Could anybody suggest. I am using http: ^0.13.3 in my one of the app , It comes in http.post method.

itmammoth commented 2 years ago

Same problem on Android 6/7

erikmompean commented 2 years ago

Same with Huawei Android 7

yalscript commented 2 years ago

Same problem here since the 30th of September for our android 6/7 devices. Is it related to this: internet-disconnect-sept-30?

Has this problem a solution without allowing all the certificates on our app requests (because this is a development patch, not a production solution)?

ben-xx commented 2 years ago

Definitely related to the expiration of issuer certificate from Digital Signature Trust (DST Root CA X3) which expired on Sept. 30, 2021 (which LetsEncrypt certificates were using as one of the signers on the trust chain).

If I'm understanding the situation correctly, Android 7 was before LetsEncrypt had their own Certificate Authority (Internet Security Research Group - ISRG Root X1) that was well accepted and distributed their own CA certificate. So Android 7 doesn't have ISRG_Root_X1.pem certificate in Trusted credentials (Settings > Security > Certificate Management > Trusted credentials). If you check your device, this cert should be missing.

We can add User credentials to install that missing ISRG_Root_X1.pem certificate, but Flutter/http.Client won't use that.

I guessing one possible solution is to create a custom http client using IOClient from this package & HttpClient from dart:io which can take a SecurityContext which can include a custom trusted certificate authority (i.e. ISRG Root X1).

That guess is coming from response from Nate Bosch: https://github.com/dart-lang/http/issues/223#issuecomment-452129527

cpswan commented 2 years ago

I just hit this without any Android device involvement. Just one Dart app on Linux talking to another.

The workaround I've put in place it to delete the final cert in LetsEncrypt fullchain.pem which is signed by the expired DST Root CA X3.

e.g. if I take my original fullchain.pem:

-----BEGIN CERTIFICATE-----
MIIFJjCCBA6gAwIBAgISA2DCk9/LeIgu/nxulPl6LHBIMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEwMDYxMTQ4MTZaFw0yMjAxMDQxMTQ4MTVaMBsxGTAXBgNVBAMT
EGNpY2QxLmF0c2lnbi53dGYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQDJCCTgbbedcg1MlpsRSEkuJkasLt/5TAKXpwl078wb0ZGv2qf4wC017folRO/C
XhMpKIQg64f1hRaR0YDVyYxNlnoOGsPgKDNtTTc5qQnL/df1ZJtcskBp3d2JTPbp
Mq01BTw0phrx2uCTJ3GAV1BlTvsbpFscH1ymG9xzVgt6PSX3jNY4zNxfPkUyXgQ+
hx1qfsdgs8qyyP0JQZpZfhZdsY1LWiMhxK8t0UZMYq35p2LjtOxSQScNVN2KK4bv
v1emqn3odkG9Kf4yY8Dhzupv6aJNjT4hISnZTLPqktbs2vTn/6oADyNL6/cGX8YD
zpuCsSmYzSvNYqQxvvfAEJcHAgMBAAGjggJLMIICRzAOBgNVHQ8BAf8EBAMCBaAw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYD
VR0OBBYEFC4Ij3YXbRKz8eQmcZWYOe9/ccCLMB8GA1UdIwQYMBaAFBQusxe3WFbL
rlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDov
L3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5v
cmcvMBsGA1UdEQQUMBKCEGNpY2QxLmF0c2lnbi53dGYwTAYDVR0gBEUwQzAIBgZn
gQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5s
ZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdQDfpV6raIJP
H2yt7rhfTj5a6s2iEqRqXo47EsAgRFwqcwAAAXxVpbdaAAAEAwBGMEQCIBe2NoTw
OHFJMQ5AhgWUJmpfnxX+PFNRrkCb1qE2At/eAiAaickGhy8owlnnzFJIbArc382E
VNN2PY7H435XKEOymgB3AEalVet1+pEgMLWiiWn0830RLEF0vv1JuIWr8vxw/m1H
AAABfFWlt4MAAAQDAEgwRgIhALIMgUmi7WSBv+nx5OX7CmaWL7zoSmduuEoN6Kj8
q9j6AiEAvL5zN1F06NqcYFruYuktU/PLb9rjRrPyYlkaqc1C69gwDQYJKoZIhvcN
AQELBQADggEBAETViBMF1OnksWUu0YllivFLT8OB2Krw8081yRuQY9oicC8fRv/a
178vO9c3sCF3yQ+yacU2IH2dH0xrfdsBJziDhrol0RLMDLRUUKMMF5dd202bljqC
+et6YQVuMkAb+AiZg227FHg7VIQWV7hUlqH7e6ma8wlGcAjDUNm/gL/xHzqyX7jP
h8OHXHKEZZkmnhSIAr/6DYm97gZ3Bfa8DNQKny3Y0B2nKHdIY+HJyvLd+9kyW2kd
XBPuJA6WUtPGTnhvEMuKv7oPyQxepf3YE94J8U8clVUXrN8nvUn8pOZ3i96Ty7uC
r4ZSSEeGRWJtRu89Ywg45NQmi79llKYImgc=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAwTzELMAkGA1UE
BhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2VhcmNoIEdyb3VwMRUwEwYDVQQD
EwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQG
EwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMT
DElTUkcgUm9vdCBYMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54r
Vygch77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+0TM8ukj1
3Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6UA5/TR5d8mUgjU+g4rk8K
b4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sWT8KOEUt+zwvo/7V3LvSye0rgTBIlDHCN
Aymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyHB5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ
4Q7e2RCOFvu396j3x+UCB5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf
1b0SHzUvKBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWnOlFu
hjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTnjh8BCNAw1FtxNrQH
usEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbwqHyGO0aoSCqI3Haadr8faqU9GY/r
OPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CIrU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4G
A1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY
9umbbjANBgkqhkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ3BebYhtF8GaV
0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KKNFtY2PwByVS5uCbMiogziUwt
hDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJw
TdwJx4nLCgdNbOhdjsnvzqvHu7UrTkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nx
e5AW0wdeRlN8NwdCjNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZA
JzVcoyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq4RgqsahD
YVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPAmRGunUHBcnWEvgJBQl9n
JEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57demyPxgcYxn/eR44/KJ4EBs+lVDR3veyJ
m+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

it has 3 certs in it:

  1. The cert.pem for my host (in this case cicd1.atsign.wtf): Common Name: cicd1.atsign.wtf Subject Alternative Names: cicd1.atsign.wtf Valid From: October 6, 2021 Valid To: January 4, 2022 Issuer: R3, Let's Encrypt Write review of Let's Encrypt Serial Number: 0360c293dfcb78882efe7c6e94f97a2c7048

  2. The LetsEncrypt Intermediate CA R3: Common Name: R3 Organization: Let's Encrypt Country: US Valid From: September 3, 2020 Valid To: September 15, 2025 Issuer: ISRG Root X1, Internet Security Research Group Serial Number: 912b084acf0c18a753f6d62e25a75f5a

  3. The ISRG Root CA X1 signed by DST Root CA X3: Common Name: ISRG Root X1 Organization: Internet Security Research Group Country: US Valid From: January 20, 2021 Valid To: September 30, 2024 Issuer: DST Root CA X3, Digital Signature Trust Co. Serial Number: 4001772137d4e942b8ee76aa3c640ab7

That last cert seems to be causing the problem, as it's signed by DST Root CA X3, which expired on 30 Sep 2021. Once it's deleted then Dart uses the ISRG Root CA X1 (self signed) that's in cacerts, and all is happy again.

I think (in this case at least) the root cause is a LetsEncrypt/certbot issue - why they're still adding a cert that's been signed by an expired CA is a mystery.

Edited to add...

Certbot has a flag to deal with this, so certbot renew --force-renewal --preferred-chain "ISRG Root X1" will generate a fullcert.pem without a DST X3 signed cert. But... at some stage LetsEncrypt is scheduled to move to their X2 CA, and so updating cron jobs to use a flag pinned to X1 will just be more trouble down the line.

Also the reason they're adding the ISRG X1 signed by DST X3 is for compatibility with older Android builds :/

Lessons From An Internet Outage - Issues Caused By Let’s Encrypt DST Root CA X3 Expiration provides a good rundown of the issues at hand.

Edited some more...

We're using an alternative workaround of removing the expired DST CA Root X3 from our cacerts, and thankfully it seems that Dart is smart enough to walk back up the cert chain to find a root CA that's valid.

ben-xx commented 2 years ago

@cpswan Thanks for the detailed post.

Does your Android device have the ISRG Root X1 CA certificate installed as a Trusted credential?

For my Android 7 test device, the deletion of the DST Root CA X3 from fullchain.pem on my web host only changed the details of the CERTIFICATE_VERIFY_FAILED error from "certificate has expired" to "local certificate not found". Which makes sense if the Android device does not have ISRG Root X1 certificate installed. (Which would be common for Android devices below 7.1.x).

I did have some success testing when supplying an ISRG Root X1 certificate & using package:http's IOClient (which can take a SecurityContext) along with dart:io's HttpClient.

But, this will only solve https communications that are done with that customized Dart http client. Using Chrome on an Android 7 device (that does not have ISRG Root X1 trusted cert) to visit a domain with a Let's Encrypt SSL certificate with either ISRG Root X1 or DST Root CA X3 will still result in a certificate could not be found or expired error.

Let's Encrypt has talked about how Android 7 can ignore the expiration of the DST Root CA X3 expiration in this post, but that's not what I'm seeing on an actual Android 7 device.

cpswan commented 2 years ago

@ben-xx my stuff isn't using Android (though it's a set of services that might have Android clients and we're now braced for any issues croping up with Android 6/7).

We've been cut by the other edge of the Letsencrypt compromise by continuing to sign their root CA with the expired DST X3 CA. It's not a panacea for early Android compatibility, and it's causing collateral damage to other apps that are picky about chains of trust.

DiaaEddin commented 2 years ago

I have the same error on flutter windows application. even when using HttpClient from dart:io . it is working on some machines but not on other machines. I have no Idea why but this is a serious issue. Are there any workarounds?

DiaaEddin commented 2 years ago

I tried solving the problem but still getting the same error. you can check it at this issue . So what am I doing wrong?

ben-xx commented 2 years ago

@DiaaEddin I just posted an example on Stack of supplying Let's Encrypt's root certificate to dart:io IOClient via a HttpClient with a SecurityContext which should help on older Android devices that are missing the ISRG Root X1 certificate.

DiaaEddin commented 2 years ago

@ben-xx Thanks for the reply. As you can see in my issue I am getting CERT_ALREADY_IN_HASH_TABLE error. still the client throws Cert expired exception. you can check the code and the output at the linked issue in my comment

ben-xx commented 2 years ago

@DiaaEddin That's an interesting situation.

I wonder if the systems where you're still seeing the issue are using OpenSSL 1.0.2, which will always use the expired trust chain & thus will always fail, as described in this post, in Workaround 3 (which is linked from this LE post).

I'm definitely no expert in this, so please take this with a grain (or heap) of salt, but I think one of the solutions to that particular issue is to simply remove the DST Root X3 certificate from the fullchain.pem file from the hosting server's Let's Encrypt SSL cert bundle. Normally there would be three sets of certificates in fullchain.pem, with the third (bottom) one being the DST cert.

Doing this (as far as I understand) removes the expired DST cert from the chain so even on systems using OpenSSL 1.0.2, the trust chain is no longer "expired".

Maybe?

I'm unsure whether you need to also remove the expired DST Root X3 certificate from the hosting OS. For my Linux based servers, I did, just to be safe. (As described here and here.)

DiaaEddin commented 2 years ago

@ben-xx Note that I am testing on https://valid-isrgrootx1.letsencrypt.org which is a subdomain under letsencrypt.org , so I do not have any control over the server besids, they are the certificate issuer so thier server configuration should be the standard and I think it is better to take thier approach, witch leaves us with the client side workaround. Any ideas how to implement the mentioned solution on a dart client?

DiaaEddin commented 2 years ago

BTW, On my server I am using nginxproxy/nginx-proxy docker image to automatically request and update my certificates. do you have any resources on how to implement that server side workaround in this case?

DiaaEddin commented 2 years ago

@ben-xx I just figured that I need to remove the expired certificate from the list. So is it possible to remove a certificate from SecurityContext? or should I create a new empty SecurityContext and provide the valid list from mozilla like https://github.com/dart-lang/root_certificates ?

ben-xx commented 2 years ago

@DiaaEddin I'm also using nginx-proxy, but I'm not having any issues making https connections to LetsEncrypt SSL protected sites it's serving from old Android devices, so long as I supply the ISRG Root X1 trusted cert to HttpClient via SecurityContext on Flutter. If I don't supply the ISRG Root X1 cert, I do get CERTIFICATE_VERIFY_FAILED error on that same HttpClient when I try to make a get or post request.

So, the only thing I needed to do (client-side) was supply the ISRG Root X1 cert for use by Flutter's HttpClient (since the underlying Android device is old, doesn't get updates & doesn't have that cert). I did not need to remove (i.e. turn off) the DST Root X3 (expired) cert from the Android device's Trusted Credentials list. I did not need to do anything on the nginx-proxy sites. They're still using the Let's Encrypt SSL certs with the expired cross-sign DST cert in their fullchain.pem. I'm assuming this doesn't affect my setup because none of my servers are using openssl version 1.0.2, so the client & server are both able to use the short trust chain of just ISRG Root X1 and they both just ignore the long trust chain which includes the DST Root X3 (expired).

DiaaEddin commented 2 years ago

@ben-xx I think that openssl version 1.0.2 is an old version of openssl client to connect to secured sites. so your servers have an up to date version of openssl client so it is possible to make successful requests to other servers as clients. even in the question on stackoverflow link that you provided the OP was using his server as client and the answers was solving the issue for the client. But just to be sure. could you give me a link to your website to test my code against it. Or maybe try yourself connecting to it using a client with the same situation as mine. I am really getting frusterated about this issue and it seems to me that the only soluation is to provide the whole root certificates list to an empty SecurityContext though I do not think that this is a practical solution

DiaaEddin commented 2 years ago

@ben-xx It looks like the dart sdk have the same problem as openssl version 1.0.2, and my issue is the same as this one. Dart sdk stops and throws an error on the first valid certificate if its expired and does not complete checking even if there are valid certificates after the expired one. The problem got fixed but the fix did not get shipped to the stable channel yet.

CritterAlert commented 2 years ago

i had the same issue on an android 6 device. odd thing is, it's only contained to my dart app (in development) on affected OS version devices. i fired up chrome & opera on 4.2.2, 4.4.2, 4.4.4, 6.0 and they worked against the same URL on these devices without error, the browsers aren't failing for this expired issue against the same URL. on 7.1.1 and above, i found no issue via dart app, i hear it's 7.0 and below the exception happens. for security reasons the recommendation is to fix it on the server and not the client by the user via a workaround. i'm using nginx and removing the last entry in the chain goes from expired to local cert issue exception (it was recommended fix, doesn't work for dart). the only way to fix this was to dump letsencrypt and get a free ZeroSSL cert. installed, restarted nginx and issue went away.
they offer more via their ACME (certbot) integration, i made the mistake doing it via their website directly, you only get 3 certs per 90 days with no subdomains, each subdomain is a cert. apparently the command line tools offer letsencrypt functionality via ZeroSSL but not via their free site offering.

DiaaEddin commented 2 years ago

@CritterAlert Have you tried this
If it did not work please try building your app with the latest dart sdk on the development channel. I think that the recommended approach is to update the client not removing the old cert from the server so your website still have compatapility with older devices. Note that this is a bug in older clients not in letsencrypt.

CritterAlert commented 2 years ago

I think that the recommended approach is to update the client not removing the old cert from the server so your website still have compatapility with older devices. Note that this is a bug in older clients not in letsencrypt.

the browsers on the old devices didn't fail, they worked against the same URL. i disagree, so do security experts. this happened at the server level (unless there is an sdk issue, i did reference this), because a certificate in the chain expired (30th September), the application is doing what it is designed and reporting a security exception and only contained to letsencrypt certificates on older devices. the self signed letsencrypt fix doesn't work either. going forward, letsencrypt certificates now have a compatibility issue on older devices that zerossl doesn't. if you have a flutter application with tens of thousands or more users connecting against a server using an expired certificate like this, it can be corrected instantly at the server instead of rolling out an update (which by the way you should only be using stable release builds in production environments). rolling out an update can take time to propagate to devices. server side fix is instant for all. i just switched SSL provider, installed new certificates and it fixed it instantly on the server. you pay for what you get, it's a false economy. i'm seriously looking at going back to paid commercial SSL offerings.

DiaaEddin commented 2 years ago

@CritterAlert Can you provide any references to back your claims? I am no security expert either but from what I understand. the problem is that older devices does not trust ISRG ROOT X1 so it is not included in their chain. so letsencrypt used DST Root CA X3 in their chain to keep compatability with these devices. but DST Root CA X3 expired about a month ago but some of the new devices that have ISRG root X1 certificate have bug in their clients that causes them to refuse certificates signed with DST Root CA X3 because it is expired even though it is also signed with ISRG ROOT X1. the other issue in older devices is obviously that they do not have ISRG ROOT X1 in their trusted chain so you need to add it yourself by updating your client. ISRG ROOT X1 is a valid root ca. and it is accepted in mozilla and other modern browsers. So I think the client should be updated. also If I understand the situation correctlly other root certificates will expire at some point and a client update will be required eventually.

DiaaEddin commented 2 years ago

BTW, if your are going to use a paid service what certificate issuer would you recommend?

CritterAlert commented 2 years ago

> @CritterAlert Can you provide any references to back your claims? I am no security expert either but from what I understand. the problem is that older devices does not trust ISRG ROOT X1 so it is not included in their chain. so letsencrypt used DST Root CA X3 in their chain to keep compatability with these devices. but DST Root CA X3 expired about a month ago but some of the new devices that have ISRG root X1 certificate have bug in their clients that causes them to refuse certificates signed with DST Root CA X3 because it is expired even though it is also signed with ISRG ROOT X1. the other issue in older devices is obviously that they do not have ISRG ROOT X1 in their trusted chain so you need to add it yourself by updating your client. ISRG ROOT X1 is a valid root ca. and it is accepted in mozilla and other modern browsers. So I think the client should be updated. also If I understand the situation correctlly other root certificates will expire at some point and a client update will be required eventually.

I've been in the IT profession for 30 years, you don't code your way out of a problem unless you have no control over the environment around you, especially if it can be resolved at the source. it took me less than 600 seconds to resolve the issue without having to write a single line of code, by creating an account with ZeroSSL, requesting an SSL, authenticating via my DNS and downloading, installing and restarting nginx and instantly user clients were connecting without issue.

Reading one article, one company have 4 apps that started failing on the 30th Sept, the instant solution was to make the problem go away without having open up an application, by going to another SSL provider.

This one article i found, they say the same thing.. fix it at the server (in their conclusion), as it was the certificate on the server that was the cause of this issue, to resolve it, resolve it there (which was dump letsencrypt).

https://www.catchpoint.com/blog/lessons-from-an-internet-outage-issues-caused-by-lets-encrypt-dst-root-ca-x3-expiration

There is argument of PWA over native apps, if you rollout a change via PWA, everyone gets it on the next reload, you don't with native, there is no guarantee when and if phones will get the updates. I've met users who've run out of space and apps haven't been updated in months (failing due to insufficient space (they fill them with photos and don't update to the cloud and don't delete the device copy)), they're still using the old buggy apps. Resolving on the server is taking the same approach, restart and everyone affected gets the fix.

The point is, 3 hours (10,800 seconds) reading and trying what was being recommended and nothing worked.. and i thought.. enough.. under 600 seconds later my problem (letsencrypt) went away. All certificates will expire eventually, some have 2035 on their expiration date.

CritterAlert commented 2 years ago

BTW, if your are going to use a paid service what certificate issuer would you recommend?

cheapsslsecurity offer rapid sll or comodo postive ssl for up to 6 years (with reissues).. no more 90 day refresh (sometimes i need to reboot the server because nginx fails to restart). They start from a few bucks a year if bought for 6 years. Get a trial first and test it (often via the sites directly, not the reseller). You'll find these CA's are likely on the old devices, i read the older devices don't get CA updates, these providers have been around a long time. I've used both in the past without issue, rapid first.

DiaaEddin commented 2 years ago

There is argument of PWA over native apps, if you rollout a change via PWA, everyone gets it on the next reload, you don't with native, there is no guarantee when and if phones will get the updates. I've met users who've run out of space and apps haven't been updated in months (failing due to insufficient space (they fill them with photos and don't update to the cloud and don't delete the device copy)), they're still using the old buggy apps. Resolving on the server is taking the same approach, restart and everyone affected gets the fix.

Yes you are absolutely right from business point of view. in production you should opt for the quick fix whenever you can.

I've been in the IT profession for 30 years, you don't code your way out of a problem unless you have no control over the environment around you, especially if it can be resolved at the source. it took me less than 600 seconds to resolve the issue without having to write a single line of code, by creating an account with ZeroSSL, requesting an SSL, authenticating via my DNS and downloading, installing and restarting nginx and instantly user clients were connecting without issue.

Also after checking ZeroSSL they seems to me like a better alternative to letsencypt when using acme client, and they have paid plans so I know how they make their money, this is another reason to consider moving to their service.

cheapsslsecurity offer rapid sll or comodo postive ssl for up to 6 years (with reissues).. no more 90 day refresh (sometimes i need to reboot the server because nginx fails to restart). They start from a few bucks a year if bought for 6 years. Get a trial first and test it (often via the sites directly, not the reseller). You'll find these CA's are likely on the old devices, i read the older devices don't get CA updates, these providers have been around a long time. I've used both in the past without issue, rapid first.

I checked cheapsslsecurity, to me their site feels a bit distracting and cheap, I do not feel comfortable buying from them. and their prices are very expensive comparing to ZeroSSL premium plan, especially when buying certificates for multiple domains.

it took me less than 600 seconds to resolve the issue without having to write a single line of code, by creating an account with ZeroSSL, requesting an SSL, authenticating via my DNS and downloading, installing and restarting nginx and instantly user clients were connecting without issue.

that is why I try to use nginx-proxy/acme-companion whenever I can, the default configurations uses letsencrypt but they have a ZeroSSL page on their wiki.

CritterAlert commented 2 years ago

I switched my servers away from letsencrypt last night, not just the development server.

Also after checking ZeroSSL they seems to me like a better alternative to letsencypt when using acme client, and they have paid plans so I know how they make their money, this is another reason to consider moving to their service.

From ZeroSSL's website you only have 3 single domain's (sub domain is counted as a single domain issue) per 90 days, if you revoke one it counts as one. It solved my problem pretty much instantly. After a bit of further digging i found their ACME partner scripts allow a lot more directly.

Their ZeroSSL Bot ACME script doesn't appear from their reference to give you control over multi domain & it appears to wrap around letsencrypt when i tried it, i couldn't find much on configuring it for ZeroSSL (you'd think out of the box would work), like their website, they don't appear to offer multiple sub domain certificates.

I switched to their one of their trusted ACME partner scripts where i can get get multi sub domain per certificate issue:

https://github.com/acmesh-official/acme.sh

Out of the box it worked. A bit of trial and error, it likes to create everything under ~/.acme.sh/ including certificates.

To put certs under a specific location i used: --cert-home /etc/nginx/ssl (had to create an ssl directory) or alternative /etc/ssl

acme.sh --issue --standalone --cert-home /etc/nginx/ssl -d domain.com -d www.domain.com -d sub1.domain.com

the first listed -d becomes the the directory name under the --cert-home path.

The problem is, without spending too much time looking for a solution, you have to stop nginx and restart (like letsencrypt)... then a quick check, 3 issued domains under one certificate.

Then i manually modified my nginx conf files to point to:

ssl_certificate      /etc/nginx/ssl/domain.com/fullchain.cer;
ssl_certificate_key  /etc/nginx/ssl/domain.com/domain.com.key;

I kept the original options-ssl-nginx.conf

include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot

Checked my browser, 2029 & 2030 appear to be their expiration dates in the full chain, that's breathing space for another 8/9 years.

ssllabs test gave an A

and flutter app now connects and returns data without an error.

I checked cheapsslsecurity, to me their site feels a bit distracting and cheap, I do not feel comfortable buying from them. and their prices are very expensive comparing to ZeroSSL premium plan, especially when buying certificates for multiple domains.

They are cheap, they're a broker for getting them at a fraction of a price rather than going directly and paying full price. I used them for years before i discovered SSL for free. I didn't have the issues I've faced with free. I've had reissues via them when there was a call to do so. never had an issue with cheapsslsecurity, all certificates are issued behind the scenes by the providers they're fronting for, they're all valid.

cpswan commented 2 years ago

Since this thread has taken a turn towards alternatives to LetsEncrypt, my 2¢...

The issues I noted above happened in our CI environment where we use vanilla Certbot with LetsEncrypt. Our other environments weren't affected, as we use ZeroSSL in them.

In the past we've generated certs on ZeroSSL using Certbot, but it's too slow for our needs, so we've switched to using ZeroSSL's REST API. As I was finding my way around the API I wished there were some examples, which is why I wrote ZeroSSL API – The missing examples.

bubnenkoff commented 2 years ago

Upgrade to latest beta dart\flutter helped.

CritterAlert commented 2 years ago

Upgrade to latest beta dart\flutter helped.

that won't happen at companies, you don't run beta software in production environments, especially when resolution was a lot simpler....then you have to distribute your app via the app store, and that can take days to propagate to devices (check the release date on the android app store to the actual date when it appears in your update list, 3 days i've seen) and not guaranteed to reach all users due to what i've highlighted above. in the meantime your 1 star reviews star mounting up as your app stopped working.

it took less than 600 seconds to resolve by switching to alternative SSL provider on the server, reload nginx config (nginx -s reload) and all users got the update on the next connection. no app rollout required.

it will help once it reaches the stable channel and production ready for the future. i'm surprised since flutter compiles to native it didn't compile the latest openssl library with it. golang builds an independent binary.

prilepskiy commented 2 years ago

Upgrade to latest beta dart\flutter helped.

It didn't help for me (with Flutter 2.7.0-3.0.pre) So can we wait for any fixes or have to use some workarounds like there?

ben-xx commented 2 years ago

I don't think there are any fixes coming from neither Google nor Apple (for underlying Android/iOS system software), since this issue affects devices that are already no longer supported and are not receiving updates for Android/iOS.

On the Flutter side, there is a change incoming #47432, but I'm not sure it resolves this particular issue, where a device is so old, it does not have an ISRG Root X1 certificate at all. If I'm not mistaken, #47432 only fixes the issue where both ISRG Root X1 cert & DST Root X3 cert are present on the underlying Android/iOS device but Flutter is not using the alternate/short trust path (only ISRG cert) and fails to accept the long trust path (ISRG + DST) since DST cert is expired and stops looking for alternate paths that are OK.

See https://dart-review.googlesource.com/c/sdk/+/211160 for more info on the incoming patch mentioned in #47432 to Dart VM that should arrive with Flutter 2.14 whenever that arrives (I'm guessing first half 2022?).

ShivamPage commented 2 years ago

Hello all, I found this workaround:

Solution:

In your main.dart file please add the following: class MyHttpOverrides extends HttpOverrides { @override HttpClient createHttpClient(SecurityContext context) { return super.createHttpClient(context) ..badCertificateCallback = (X509Certificate cert, String host, int port) => true; } }

And then add this line to your main method: HttpOverrides.global = new MyHttpOverrides();

Silfalion commented 2 years ago

Hello all, I found this workaround:

Solution:

In your main.dart file please add the following: class MyHttpOverrides extends HttpOverrides { @override HttpClient createHttpClient(SecurityContext context) { return super.createHttpClient(context) ..badCertificateCallback = (X509Certificate cert, String host, int port) => true; } }

And then add this line to your main method: HttpOverrides.global = new MyHttpOverrides();

If I'm not wrong this should be the solution but is not safe to use in a production environment.

tudor07 commented 2 years ago

Above workaround is not safe for production use. Any proper solution?

CritterAlert commented 2 years ago

Above workaround is not safe for production use. Any proper solution?

it's what i did with zero code change, dump letsencrypt SSL in favour of a paid certificate or free ZeroSSL which i switched to without error. you get what you pay for!

Silfalion commented 2 years ago

Above workaround is not safe for production use. Any proper solution?

Ended up validating the problematic host names manually. If that helps.

tudor07 commented 2 years ago

Ended up validating the problematic host names manually. If that helps.

Can you expand a bit onto this? I would appreciate it.

Silfalion commented 2 years ago

Ended up validating the problematic host names manually. If that helps.

Can you expand a bit onto this? I would appreciate it.

Hey sorry for the late response, used an answer along these lines:

https://stackoverflow.com/a/61120486/8568263

Instead of returning just true, you compare the host parameter to the domain you need or if you many you can put them in a list and test the containment then return true if it fulfills the condition.

TJ-coding commented 2 years ago

Is there any update on this issue?

udroi commented 2 years ago

I faced the same error, in my case i was testing my code on localhost, so i just removed https and replaced it with just http which just resolved the error for me..

hussamDana92 commented 2 years ago

any update on this issue?

alihaidersiddique commented 2 years ago

I Used http:// instead of https://. And it worked

Get Outlook for Androidhttps://aka.ms/AAb9ysg


From: hussamDana92 @.> Sent: Thursday, July 28, 2022 1:53:59 PM To: dart-lang/http @.> Cc: Muhammad Ali Haider @.>; Comment @.> Subject: Re: [dart-lang/http] HandshakeException: Handshake error in client (OS Error: CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:359)) (#627)

any update on this issue?

— Reply to this email directly, view it on GitHubhttps://github.com/dart-lang/http/issues/627#issuecomment-1197857179, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHIBJJHFTW6DSDLUG4SZRQTVWJDCPANCNFSM5FHWY2VQ. You are receiving this because you commented.Message ID: @.***>

saravananmnm commented 2 years ago

I am also faced this issue. It's due to ssl certificate. Try to make upgrade or change the ssl certificate from backend side.

After ssl changes from backend side its worked for me.

MenaRaafat commented 2 years ago

I am having this problem only on my android devices (8 + 10 + 11) but not on iPhones. I do not want to override ssl in my code.

CritterAlert commented 1 year ago

I switched my servers away from letsencrypt last night, not just the development server.

Also after checking ZeroSSL they seems to me like a better alternative to letsencypt when using acme client, and they have paid plans so I know how they make their money, this is another reason to consider moving to their service.

From ZeroSSL's website you only have 3 single domain's (sub domain is counted as a single domain issue) per 90 days, if you revoke one it counts as one. It solved my problem pretty much instantly. After a bit of further digging i found their ACME partner scripts allow a lot more directly.

Their ZeroSSL Bot ACME script doesn't appear from their reference to give you control over multi domain & it appears to wrap around letsencrypt when i tried it, i couldn't find much on configuring it for ZeroSSL (you'd think out of the box would work), like their website, they don't appear to offer multiple sub domain certificates.

I switched to their one of their trusted ACME partner scripts where i can get get multi sub domain per certificate issue:

https://github.com/acmesh-official/acme.sh

Out of the box it worked. A bit of trial and error, it likes to create everything under ~/.acme.sh/ including certificates.

To put certs under a specific location i used: --cert-home /etc/nginx/ssl (had to create an ssl directory) or alternative /etc/ssl

acme.sh --issue --standalone --cert-home /etc/nginx/ssl -d domain.com -d www.domain.com -d sub1.domain.com

the first listed -d becomes the the directory name under the --cert-home path.

The problem is, without spending too much time looking for a solution, you have to stop nginx and restart (like letsencrypt)... then a quick check, 3 issued domains under one certificate.

Then i manually modified my nginx conf files to point to:

ssl_certificate      /etc/nginx/ssl/domain.com/fullchain.cer;
ssl_certificate_key  /etc/nginx/ssl/domain.com/domain.com.key;

I kept the original options-ssl-nginx.conf

include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot

Checked my browser, 2029 & 2030 appear to be their expiration dates in the full chain, that's breathing space for another 8/9 years.

ssllabs test gave an A

and flutter app now connects and returns data without an error.

I checked cheapsslsecurity, to me their site feels a bit distracting and cheap, I do not feel comfortable buying from them. and their prices are very expensive comparing to ZeroSSL premium plan, especially when buying certificates for multiple domains.

They are cheap, they're a broker for getting them at a fraction of a price rather than going directly and paying full price. I used them for years before i discovered SSL for free. I didn't have the issues I've faced with free. I've had reissues via them when there was a call to do so. never had an issue with cheapsslsecurity, all certificates are issued behind the scenes by the providers they're fronting for, they're all valid.

The simplest solution was to do nothing on the code side (not code work around hacks) and dump letsencrpyt in favour of an another SSL provider, i've been using the free ZeroSSL option since without any issue. My problem instantly went away when i switched SSL vendor, i use the apps and website via the same domain SSL (supports multiple subdomain).

ahmeedev commented 1 year ago

Any Solution?

eximius313 commented 1 year ago

For me the problem suddenly went away, so perhaps it might be because of this or this reason.

jeffrey-schmitz commented 1 year ago

My issue was due to the wifi I was connected to messing with the handshake process, though I'm not exactly sure how. It may have been reporting an incorrect clock time to my OS. If you're seeing this, try a different connection before digging too deep. I was going nuts trying to solve this.

ghost commented 1 year ago

My issue was due to the wifi I was connected to messing with the handshake process, though I'm not exactly sure how. It may have been reporting an incorrect clock time to my OS. If you're seeing this, try a different connection before digging too deep. I was going nuts trying to solve this.

This helped me in solving it. In my case, disabling VPN worked.