C# .NET Framework issue WinHttpException: Error 12175 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR

[p1x31]

Description

Hi! I was asked to copy the original issue from grpc repo in here.

I have implemented a simple gRPC client in C# .NET Framework (WinHttpHandler) and a gRPC server in Python3 (grpcio) for testing.

I'm getting this error with .net framework client and python server

Grpc.Core.RpcException
  HResult=0x80131500
  Message=Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. WinHttpException: Error 12175 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A security error occurred'.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.")
  Source=mscorlib
  StackTrace:
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Grpc.Net.Client.Internal.GrpcCall`2.<GetResponseHeadersCoreAsync>d__72.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Program.<<Main>$>d__0.MoveNext() in C:\Users\*\GrpcClientTest\GrpcClientTest\Program.cs:line 53

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
HttpRequestException: An error occurred while sending the request.

Inner Exception 2:
WinHttpException: Error 12175 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A security error occurred.

.net framework client to .net framework server works fine .net 8 client to python server works fine

What did you do?

If possible, provide a recipe for reproducing the error. Try being specific and include code snippets if helpful.

following WinHttpHandler official example.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

Reproduction Steps

Client

using var channel = GrpcChannel.ForAddress("https://localhost:7215", new GrpcChannelOptions
{
    HttpHandler = new WinHttpHandler()
});

var client = new Greeter.GreeterClient(channel);
Console.Write("name: ");
var name = Console.ReadLine();
var reply = await client.SayHelloAsync(new HelloRequest { Name = name });
Console.WriteLine($"server: {reply.Message}");
Console.ReadKey();

Server:

"""The Python AsyncIO implementation of the GRPC greet.Greeter server."""

import asyncio
import logging

import grpc
import greet_pb2
import greet_pb2_grpc

class Greeter(greet_pb2_grpc.GreeterServicer):
    async def SayHello(
        self,
        request: greet_pb2.HelloRequest,
        context: grpc.aio.ServicerContext,
    ) -> greet_pb2.HelloReply:
        return greet_pb2.HelloReply(message="Hello, %s!" % request.name)

async def serve() -> None:
    server = grpc.aio.server()
    greet_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
    listen_addr = "[::]:7215"
    server.add_secure_port(listen_addr)
    logging.info("Starting server on %s", listen_addr)
    await server.start()
    await server.wait_for_termination()

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(serve())

proto:
syntax = "proto3";

option csharp_namespace = "GrpcClientTest";

package greet;

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply);
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings.
message HelloReply {
  string message = 1;
}

I have tried:

/*AppContext.SetSwitch(
 "System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
*/

/*var httpHandler = new HttpClientHandler
{
    ServerCertificateCustomValidationCallback =
            HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
};
*/

Expected behavior

What did you expect to see?

a returned string value from my gRPC method call

Actual behavior

What did you see instead?

HttpRequestException

Regression?

works with .net 8 client .net framework client to .net framework server works fine

Known Workarounds

No response

Configuration

What version of gRPC and what language are you using?

Grpc.Net.Client 2.62.0 C# .NET Framework 4.8.1

What operating system (Linux, Windows,...) and version?

Windows 11 Pro 23H2

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

Visual Studio 2022 project; C# .NET Framework 4.8.1

c# greeter client example, python greeter server example

Other information

Please refer to: https://github.com/grpc/grpc/issues/36658 https://github.com/grpc/grpc/issues/29659 https://github.com/grpc/grpc-dotnet/issues/1733 https://github.com/grpc/grpc-dotnet/issues/2439

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

[p1x31]

cc: JamesNK @JamesNK

[rzikm]

.NET Framework is no longer in active development, we are unlikely to do any fixes there unless it is for a security vulnerability.

Does the problem reproduce with .NET Core?

[p1x31]

Hi, @rzikm No, it doesn't reproduce, .NET 8 client works fine with python server. it says on the microsoft website :

gRPC client is fully supported on Windows 11 or later.

Isn't .NET Framework 4.8.1 continued to also be supported? It might be a security vulnerability or TLS not properly configured.

Error 12175 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A security error occurred'.

Maybe there are some workarounds to mimic .NET Core behaviour?

[huoyaoyuan]

Which http handler are you using for .NET Core?

WinHttpHandler is an external component provided by Windows. It's the default for .NET Framework, and also available for .NET Core. It would be surprising if WinHttpHandler behaves differently under .NET Framework and .NET Core.

"System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);

httpHandler = new HttpClientHandler { ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator };

SocketsHttpHandler is the default handler in .NET Core. It's not available for .NET Framework.

HttpClientHandler is a facade that delegates to the default http handler. It doesn't has its own behavior and depends on underlying default implementation.

[rzikm]

Isn't .NET Framework 4.8.1 continued to also be supported?

Only critical security fixes.

It might be a security vulnerability or TLS not properly configured.

TLS configuration would be something done at the target system, not in .NET source code.

Security vulnerabilities usually look differently than inability to connect to a server. Some examples would be making remote process crash or leak data, or ability to execute arbitrary code on remote machine. Such errors should also not be reported openly on GitHub.

According to the documentation at https://learn.microsoft.com/en-us/windows/win32/winhttp/error-messages, The error says that something went wrong when verifying the server certificate at TLS layer. What sort of certificate is it? If it is some locally generated (i.e. self-signed) by the GRPC server code not explicitly configured as trusted on the (client) machine, then this sounds like correct and expected behavior.

HttpClientHandler.DangerousAcceptAnyServerCertificateValidator not working may be due to some sequence of events received from http.sys not being handled properly (my guess only). Even if that was so, it would not be considered severe enough to warrant servicing .NET Framework.

If you could reproduce this using WinHttpHandler on .NET Core, then we would fix in for future .NET Core release. What would be useful for us would be the entire self-contained project which reproduces the issue (as a zip) so that we don't need to reverse-engineer it from description.

[antonfirsov]

I find it weird that the error only reproduces on .NET Framework, since the underlying WinHttp logic should be more or less the same. We keep hitting similar issues in CI, see #69238. It would be nice to improve our diagnostics and provide more info about the underlying TLS issue by inspecting and logging lpvStatusInformation.

Regardless, I agree that this is not high priority for now, so triaging to Future.

@p1x31 it might be possible to troubleshoot this by capturing WinHttp traces:

- netsh trace start scenario=InternetClient_dbg capture=yes correlation=disabled report=disabled maxSize=1024
- <repro>
- netsh trace stop
- Inspect NetTrace.etl

Does it reveal anything interesting?

[p1x31]

Hi @huoyaoyuan!

Which http handler are you using for .NET Core?

I was using the default SocketsHttpHandler so .NET Core client with python server or .NET Core server works as expected.

However when configuring WinHttpHandler on .NET Core client with python server I was able to reproduce this error Therefore you are right that WinHttpHandler behaves in the same manner under both .NET Framework and .NET Core.

@rzikm sorry for confusing you, the problem doesn't reproduce with the default SocketsHttpHandler here is the minimal reproducible example with .NET Core Client / python server: link

I can confirm that the .NET Core Client to .NET Framework server works fine and backwards .NET Framework Client to .NET Core Server also works fine

I'm guessing for now it might be python problem here

[huoyaoyuan]

WinHttpHandler is indeed external. Its implementation does not belong to .NET. All we can do is providing suggestions to configure it, via .NET api or other ways.

[rzikm]

I'm guessing for now it might be python problem here

I think so as well, this is the output when I try to connect with OpenSSL s_client

8Service  grpcdotnetClient  main                                                                 14:27:17 
❯ openssl s_client -connect "[::1]:7215"
Connecting to ::1
CONNECTED(0000017C)
346B0000:error:0A00010B:SSL routines:tls_validate_record_header:wrong version number:ssl\record\methods\tlsany_meth.c:81:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 299 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

so the server is trying to establish an unencrypted connection, which I suppose WinHttpHandler refuses. Using a server certificate on the server should get rid of the issue

[p1x31]

I have configured python server to work with TLS and now getting this error

I have managed to solve this by converting root certificates to pkcs12 file format which I believe is preferred format for windows (other formats didn't work)

Exact steps I followed:

Configure WinHttpHandler to use TLS 1.3 in the client

Generate certs for the python server:

openssl req -x509 -nodes -newkey rsa:4096 -keyout ca.key -out ca.pem -subj /O=me

openssl req -nodes -newkey rsa:4096 -keyout server.key -out server.csr -subj /CN=localhost

openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -set_serial 1 -out server.pem

this will generate

1. ca.key is a private key.
2. ca.pem is a public certificate.

server:

1. server.key is the server’s private key.
2. server.csr is an intermediate file.
3. server.pem is the server’s public certificate.

And converting root certificates to pfx with openssl openssl pkcs12 -inkey ca.key -in ca.pem -export -out ca.pfx

on every client host "trust" the cert.pfx: doubleclick it and follow the wizard, the only important thing here is to put it into 'Trusted Root Certification Authorities'

Thanks everyone @rzikm @huoyaoyuan @antonfirsov for your in depth expertise and generous support. Without your knowledge I wouldn't ve be able to come up with the solution.

Complete working example here