Unable to connect gRPC from Android <= 9 (using .NET MAUI) to Linux Azure WebApp API. Repository for reproduction attached.

czmirek commented 1 year ago

What version of gRPC and what language are you using?

C# .NET 7 proto3 Visual Studio Version 17.5.5 Grpc.Net.Client 2.53.0 Grpc.Tools 2.54.0 Google.Protobuf 3.23.0

What operating system (Linux, Windows,...) and version?

Server = Azure WebApp running on linux Failing clients = Android 9 and older Successful Clients = Android 10 and newer

What runtime / compiler are you using (e.g. .NET Core SDK version `dotnet --info`)

.NET SDK: Version: 7.0.203 Commit: 5b005c19f5

Runtime Environment: OS Name: Windows OS Version: 10.0.22621 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\7.0.203\

Host: Version: 7.0.5 Architecture: x64 Commit: 8042d61b17

.NET SDKs installed: 6.0.311 [C:\Program Files\dotnet\sdk] 6.0.400-preview.22330.6 [C:\Program Files\dotnet\sdk] 7.0.100 [C:\Program Files\dotnet\sdk] 7.0.203 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: Not found

Learn more: https://aka.ms/dotnet/info

Download .NET: https://aka.ms/dotnet/download

What did you do?

I have prepared a repository with a Visual Studio for you to test here: https://github.com/czmirek/maui_droid_linuxazapp_grpc_bug

Create an Linux Azure WebApp for .NET 7
Configure the web app to enable gRPC
- Add env variable HTTP20_ONLY_PORT = 8585
- Set Http Version = 2.0
- Set Http 2.0 Proxy = On
Deply project GrpcWebApi on the WebApp.
Get the webapp URL from Azure Portal and replace the string in MainPage.xaml.cs
- const string testUrl = "https://YOURAPP.azurewebsites.net";
Download Android 10 emulator
Run MauiTest in Android 10. You'll see the following screen:

Push "Test HTTP/2". You'll get dialog with message "TEST". = HTTP/2 works.
Push "Test gRPC". You'll get dialog with message "Hello TEST". = gRPC works
Repeat the same for Android 9 or less
Push "Test HTTP/2". You'll get dialog with message "TEST". = HTTP/2 works.
Push "Test gRPC". You'll get the following exception.

Error starting gRPC call. HttpRequestException: Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while unable to estabilish HTTP/2 connection.

What did you expect to see?

For Android 9 or lower:

Push "Test gRPC". You'll get dialog with message "Hello TEST". = gRPC works

What did you see instead?

Exception is thrown.

Error starting gRPC call. HttpRequestException: Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while unable to estabilish HTTP/2 connection.

Anything else we should know about your project / environment?

If you try it locally (after disabling HTTPS certificate checks) over HTTP then it works even with Android 9 and lower versions!
This makes me suspicious of a bug related to TLS.

JamesNK commented 1 year ago

Is this code resulting in a HTTP/2 request? https://github.com/czmirek/maui_droid_linuxazapp_grpc_bug/blob/83816204207fe245837ae22396601acf7a87ba31/MauiTest/MainPage.xaml.cs#L34-L43

It's probably HTTP/1.1. You can check by looking at HttpResponseMessage.Version.

Edit: On further thought, it might be HTTP/2. Note that the gRPC channel uses SocketsHttpHandler internally, and it might have different behavior than the default HTTP handler. HttpClient client = new HttpClient(new SocketsHttpHandler()) would be more equivalent with gRPC.

JamesNK commented 1 year ago

@steveisok Do you know if there are certain versions of Android where HTTP/2 is supported with SocketsHttpHandler? If not, could you recommend someone who would know?

simonrozsival commented 1 year ago

@JamesNK I will look into this issue next week. It's possible we already fixed this problem with some recent improvements in .NET 8.

simonrozsival commented 1 year ago

I looked into the issue and I have a few notes:

I can reproduce the problem on Android 9.
The "Test HTTP/2" does HTTP 1.1 requests. The HttpClient uses the native AndroidMessageHandler from Xamarin under the hood by default and it can't do HTTP/2 requests.
When I change the code of "Test HTTP/2" to make plain HTTP/2 requests without gRPC, I get the same error which was reported in this issue. It's therefore unrelated to gRPC and it's a dotnet/runtime issue and I think we should transfer it to the runtime repo.
I get the same error on the latest .NET main branch.

Simplified repro:

var handler = new SocketsHttpHandler();
var client = new HttpClient(handler)
{
    DefaultRequestVersion = new Version(2, 0),
    DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact
};

var response = await client.GetAsync("https://www.github.com/"); // -- throws on Android 9
Console.WriteLine(response);

I haven't been able to narrow down what exactly is going on with Android 9 (and older) yet.

DosangGu commented 7 months ago

@JamesNK When do you expect this issue to be fixed? Or are there any alternative ways to avoid this?

Sevenish commented 4 months ago

Any updates on this? Been looking for a workaround / solution for multiple days now. We are stuck at the moment.

Sevenish commented 4 months ago

After allot of digging through the dotnet runtime it eventually comes down to this function

https://developer.android.com/reference/javax/net/ssl/SSLParameters#setApplicationProtocols(java.lang.String[])

This function is defined from api level 29

which is called here in the dotnet framework:

https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Android.cs

https://github.com/dotnet/runtime/blob/main/src/native/libs/System.Security.Cryptography.Native.Android/pal_sslstream.c#L1050

with further definitions in:

https://github.com/dotnet/runtime/blob/main/src/native/libs/System.Security.Cryptography.Native.Android/pal_ssl.c#L55 and https://github.com/dotnet/runtime/blob/main/src/native/libs/System.Security.Cryptography.Native.Android/pal_jni.c#L83

When this function is not present (below api 29), ALPN does not work. http/2 is not set in the handshake.

simonrozsival commented 4 months ago

@Sevenish thanks for the detailed investigation and explanation. I looked through Android API docs and I can't find any alternative APIs that could be used on older versions. I wonder if we might use some private java APIs that would allow us to implement ALPN on older Android versions.

Sevenish commented 4 months ago

OkHttp uses BoringSLL

If you look through their issues history they where struggling the the native jave SSL and support for ALPN and http/2 in the past. They opted for using BoringSSL on android and support ALPN http/2 from android 5 and up

https://square.github.io/okhttp/security/security_providers/

OkHttp also mentions Conscrypt which uses BoringSSL https://source.android.com/docs/core/ota/modular-system/conscrypt

It seems android made a switch to using conscrypt under the hood in android 10 (api 29) https://source.android.com/docs/core/ota/modular-system/conscrypt

Conscrypt works from api level 9 and up https://github.com/google/conscrypt

conscrypt is where the actual implementation of the missing function is located: https://github.com/google/conscrypt/blob/master/common/src/main/java/org/conscrypt/SSLParametersImpl.java

I did read that conscrypt used to be installed on older android versions with google play services with a feature called 'Google Play Services Dynamic Security Provider'

grpc / grpc-dotnet

Unable to connect gRPC from Android <= 9 (using .NET MAUI) to Linux Azure WebApp API. Repository for reproduction attached. #2126

What version of gRPC and what language are you using?

What operating system (Linux, Windows,...) and version?