WINHTTP: unknown certificate check failure (12)

[jeroen]

We have a Windows 2008 (= Vista) server, which has all of the TLS 1.2 windows updates installed. I am sure TLS 1.2 works: the server can reach Github over HTTPS via native Windows applications such as Internet Explorer, or with libcurl using the schannel TLS back-end.

However on the same server when trying to clone from Github with libgit2 we see:

libgit2 error in git_clone: unknown certificate check failure (12)

This happens with libgit2 0.27.8 and 0.28.1 built with mingw-w64 compiles and WINHTTP=ON. The problem does not appear on more recent versions of Windows.

Reproduction steps

Build libgit2 with WINHTTP=ON. Then try to clone something from Github on Windows 2008.

Expected behavior

Clone succeeds.

Actual behavior

Clone fails with error: unknown certificate check failure.

Version of libgit2 (release number or SHA1)

0.27.8 and 0.28.1

Operating system(s) tested

Windows 2008, Windows Vista.

[ethomson]

"unknown certificate check failure" isn't an error message that would be produced when there's a problem with TLS incompatibility. The problem here is the server's certificate.

Having said that, I'm not at all sure why WinHTTP would have a problem coping with GitHub's certificate. Instead, do you have a proxy on your network?

[jeroen]

There is no proxy. Judging from a quick search, this error started popping up when GH moved to TLS 1.2. I think this may actually be the same problem as https://github.com/libgit2/libgit2/issues/4663

[jeroen]

Also I tried disabling certificate checking:

static int skip_cert_check(git_cert *cert, int valid, const char *host, void *payload){
  return 0;
}

static void set_default_callbacks(git_remote_callbacks *callbacks){
  callbacks->credentials = auth_callback;
  callbacks->certificate_check = skip_cert_check;
}

Now, instead of an error I get a segfault when cloning over https on Vista. Works fine on other systems.

[ethomson]

There is no proxy. Judging from a quick search, this error started popping up when GH moved to TLS 1.2. I think this may actually be the same problem as #4663

Which was fixed by #4550.

Note that you may be seeing false positives from testing with Internet Explorer due to group policy affecting IE and WinHTTP differently. eg: https://community.qualys.com/thread/17465-tls-12-for-2008-non-r2

Now, instead of an error I get a segfault when cloning over https on Vista.

Can you be more specific?

[jeroen]

Which was fixed by #4550.

Hmm are you sure? https://github.com/libgit2/libgit2/pull/4550 is from Feb 2018 and https://github.com/libgit2/libgit2/issues/4663 is from May 2018 (and references that commit as a potential culprit).

Can you be more specific?

It's difficult for me to get a backtrace on Windows because I'm only building bindings, not libgit2 itself. What happens is that when I set a certificate_check handler, I am no longer seeing the error message but insteadgit_clone() function segfaults without ever calling the callback.

I'll try to get better tooling to get a backtrace...

[ethomson]

Hmm are you sure? #4550 is from Feb 2018 and #4663 is from May 2018 (and references that commit as the potential problem).

Well, no, I'm definitely not sure. 😀

But by explicitly enabling TLS1.2, we are able to negotiate TLS 1.2 on older Windows systems that have the patches installed and have the proper settings configured in the registry. That we've had only one report of an issue in the year since we did this - and that there was no follow-up - suggests to me that #4663 was a local / environmental problem. But indeed it's impossible for me to tell.

So I think that there are a few options here:

1. Your system is not actually letting us enable TLS 1.2. You could test this by checking the return code of WinHttpSetOption. eg, in winhttp.c line 787:

   if (!WinHttpSetOption(t->session,
       WINHTTP_OPTION_SECURE_PROTOCOLS,
       &protocols,
       sizeof(protocols))) {
           git_error_set(GIT_ERROR_OS, "failed to set secure protocols");
           goto on_error;
   }

   (We don't do this by default since one could very legitimately be using an unpatched Windows machine and only wanting to connect to hosts that speak TLS 1.1 and don't want to break them.)

2. There's an incompatibility or subtlety with mingw. We build and test with mingw, so I'm skeptical about this one, but it's definitely a possibility. You could test this by building with MSVC instead of mingw to determine if there's any success in that route.

My suspicion is that we're trying, but not able - to enable TLS 1.2. For the sake of completeness (and, again, because Internet Explorer and WinHTTP use different settings here) can you provide me with the value of these registry keys?

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp\DefaultSecureProtocols
HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp\DefaultSecureProtocols

[jeroen]

Thanks for the response. This is a build server so I cannot easily run regedit, but afaict from running some scripts there is no DefaultSecureProtocols. I only see SecureProtocols=0xA80 (outside of WinHttp):

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\SecureProtocols

If you think setting DefaultSecureProtocols is likely to fix the situation I could request the sysadmin to do so. However I am a bit skeptical this is really the issue because the main application that runs on that server uses InternetOpenUrl() which works as expected for connecting to github.com.

I'll try to build a version using the patched winhttp.c above. Does libgit2 not consult GetLastError() when WinHttpSetOption() fails to get an informative error?

[ethomson]

If you think setting DefaultSecureProtocols is likely to fix the situation I could request the sysadmin to do so.

I do. Please follow the steps at https://support.microsoft.com/en-gb/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-default-secure-protocols-in-wi to enable TLS 1.2 for WinHTTP. I'd encourage you to use the "Easy fix" method and run the helper application.

However I am a bit skeptical this is really the issue because the main application that runs on that server uses InternetOpenUrl() which works as expected for connecting to github.com.

InternetOpenUrl uses WinINet, while we use WinHTTP. These are two different systems that do behave differently. While indeed I cannot guarantee that this will solve your problem, I do suspect that it will.

I'll try to build a version using the patched winhttp.c above. Does libgit2 not consult GetLastError() when WinHttpSetOption() fails to get an informative error?

We do not raise an error when we can't use WinHttpSetOption to enable TLS1.2. Again, one could very legitimately be using an unpatched Windows machine and only wanting to connect to hosts that speak TLS 1.1 and don't want to break them.

[jeroen]

Hmmm that article only mentions support for Windows 7 / Windows Server 2008 R2 but not Windows Vista / 2008. Either way, I'll try to persuade the sysadmin to have a look at that fix...

[tiennou]

I've been thinking that we're missing an option for enforcing a TLS protocol level, so maybe that should be added ?

[ethomson]

@jeroen Sorry to nag, since I realize that working with other teams can take some time to get settings updated. But I am curious if you have any update here.

[jeroen]

I have not heard back from my sysadmin yet :-(

[ethomson]

@jeroen any update?

[ethomson]

Closing this; let me know if you're still having issues after validating with your system administrator that the correct patches are installed.

[jeroen]

I finally was able to sit down with the sysadmin and test it, and it worked! Thanks again for your help!

[ethomson]

Sweet, glad it worked!

[andreasfranke]

@ethomson Unfortunately, despite all my efforts, I still have the problem with the "unknown certificate check failure".

I am running Gitea 1.17.2 as my Git server. The libgit2 client is running on Windows 2012

What I have done so far. Update to libgit2 v1.5 Set registry patches for DefaultSecureProtocols and SecureProtocols to the appropriate values (client machine).

From my desktop machine (Windows 10) I can start the client without any problem.