[Windows] Support root certificates that are installed lazily

[loic-sharma]

Problem

Windows installs a subset of trusted root certificates lazily. Dart's certificate verification should trigger these root certificates' installation if necessary.

This affects all Let's Encrypt certificates.

See:

1. https://community.letsencrypt.org/t/isrg-root-lazy-loading-problem-missing-from-random-updated-windows-10-versions/141550
2. https://community.letsencrypt.org/t/root-cert-not-included-in-windows-10/177956

Repro

On a brand new Windows box (or use Windows sandbox), run this app:

import 'package:http/http.dart' as http;

void main() async {
  var url = Uri.parse('https://letsencrypt.org/');
  var response = await http.get(url);
  print('Response status: ${response.statusCode}');
  print('Response body: ${response.body}');
}

It results in the following error:

Unhandled exception:
HandshakeException: Handshake error in client (OS Error:
        CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(../../third_party/boringssl/src/ssl/handshake.cc:393))
#0      _SecureFilterImpl._handshake (dart:io-patch/secure_socket_patch.dart:99:46)
#1      _SecureFilterImpl.handshake (dart:io-patch/secure_socket_patch.dart:142:25)
#2      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:915:54)
#3      _RawSecureSocket._tryFilter (dart:io/secure_socket.dart:1044:19)
<asynchronous suspension>

Dart SDK Version: Dart SDK version: 2.19.6 (stable) (Tue Mar 28 13:41:04 2023 +0000) on "windows_x64"

Workaround

Use PowerShell to access the affected endpoint:

Invoke-WebRequest https://letsencrypt.org

This causes Windows to install the trusted root certificate. The Dart app will now work as expected without producing a CERTIFICATE_VERIFY_FAILED error.

[a-siva]

//cc @brianquinlan

[alexei-g-aloteq]

It happens for us not only with LetsEncrypt certificates.

Also affected any other certificate chains not loaded onto the Windows trusted store. Namely, calls to something like https://x.cloudfunctions.net (Google cloud) also fails.

But if I try to fetch this address with a curl (like 'curl -v https://x.cloudfunctions.net') - the lazy loading triggers and Windows fetches the root certificate. After that, requests from dart begins to work.

[alexei-g-aloteq]

Hello? Anyone tried to reproduce it?

[penguisnt]

Hi, I'm having the same issue with needing to make requests to Google APIs. My Firebase Auth login was failing, until I called the identitytoolkit API endpoint directly from powershell. Would love to understand what is going on here.

[alexei-g-aloteq]

Another observation.

Many developers do not see this issue with ISRG Root X1 (aka Letsencrypt) certs because have Spotify installed. Being installed (often automatically by Windows), even not being used, Spotify make requests to their servers at least on automatic update installations through the Windows update.

The thing is: Since Spotify uses X1 certificates and triggers the ISRG Root X1 root certificate to load. Then your requests to an endpoint secured with Letsencrypt certificate start to work. Bot not for the others.

Before trying to reproduce this issue, first disable Spotify (better - uninstall), then remove ISRG Root X1 from trusted root CAs folder and try.

If you see that ISRG Root X1 cert is reappeared - it seems that an another application just made a native network request which triggered the lazy root certificate download. It will not happen if you making similar request from Dark runtime.

[ntkme]

Ironically, this even impacts https://pub.dev, that a fresh installation of dart-sdk on GitHub's Hosted Windows on Arm runner fails to run dart pub get out of box. We had to add the following workaround in dart-sass' GitHub Actions:

    - run: Invoke-WebRequest https://pub.dev
      if: runner.os == 'Windows'
      shell: powershell

I think the root of the problem is Microsoft's TLS implementation a.k.a Schannel would lazily initialize the trusted CA certificates, but dart is built with BoringSSL which would only read the already initialized trusted CA.

This isn't an easy thing to fix, unless dart-sdk adds support of using Schannel on Windows, or ships its own default CA bundle like Chrome.

[escamoteur]

Ah, just ran into this too in our case on an accelerate endpoint from S3