dotnet / android

.NET for Android provides open-source bindings of the Android SDK for use with .NET managed languages such as C#
MIT License
1.89k stars 524 forks source link

HttpClient and AcceptEncoding gzip on Android stoped working after latest SR4 release. #8928

Open PureWeen opened 1 month ago

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @JoacimWall on Thursday, April 11, 2024 6:58:03 PM

Description

After I updated to latest Dotnet SDK and workloads on my dev machine my app start to return error on some API calls. The same thing happened in Azure Pipeline even though I didn't make any changes to the script.

The app works flawlessly under Ios. So the unchanged code that target MAUI 8.0.6( Also tested target 8.0.20) return the same error.

if I just check de content of the return it's still gzip I think. var test = result.Content.ReadAsStringAsync() on Ios = "{\"authentication\":{\"logintoken\":\ on Android = "�\b\0\0\0\0\0\0��n�8��0��g�T����-ǒc[�,-)Qe�\"ɲ�A�f�a`^l)'��N��3h����!�(�>�����UHҊz��Y�y�s'�

//Joacim

Steps to Reproduce

If I check the logs of the Mac build host in Azure pipeline Pool: Azure Pipelines Image: macOS-13 Agent: Azure Pipelines 3 Started: Today at 20:01 Duration: 10m 39s

I see that Successfully installed .NET Core sdk version 8.0.201. Creating global tool path and pre-pending to PATH. Finishing: Use .Net 8 and the dotnet workload install maui Now installs newer versions of the workload and other parts than it did before, Maybe the error is there somewhere

Link to public reproduction project repository

https://github.com/JoacimWall/Joacim_Bug_Report/tree/main/GzipBug

Version with bug

8.0.20 SR4

Is this a regression from previous behavior?

Yes, this used to work in .NET MAUI

Last version that worked well

8.0.6 SR1

Affected platforms

Android

Affected platform versions

No response

Did you find any workaround?

Yes turn of the gzip for Android

if IOS

    this.client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));

endif

Relevant log output

Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.net7.iossimulator-x86 version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.Mono.ios-arm version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.net7.ios-arm version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.net7.ios-arm64 version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-arm64 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.net7.iossimulator-arm64 version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.net7.iossimulator-x64 version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.Mono.iossimulator-x86 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.net7.iossimulator-x86 version 7.0.18...
Installing pack Microsoft.MacCatalyst.Sdk version 17.2.8053...
Writing workload pack installation record for Microsoft.MacCatalyst.Sdk.net8 version 17.2.8053...
Installing pack Microsoft.MacCatalyst.Sdk version 16.4.7142...
Writing workload pack installation record for Microsoft.MacCatalyst.Sdk.net7 version 16.4.7142...
Installing pack Microsoft.MacCatalyst.Ref version 17.2.8053...
Writing workload pack installation record for Microsoft.MacCatalyst.Ref version 17.2.8053...
Installing pack Microsoft.MacCatalyst.Runtime.maccatalyst-x64 version 17.2.8053...
Writing workload pack installation record for Microsoft.MacCatalyst.Runtime.maccatalyst-x64 version 17.2.8053...
Installing pack Microsoft.MacCatalyst.Runtime.maccatalyst-arm64 version 17.2.8053...
Writing workload pack installation record for Microsoft.MacCatalyst.Runtime.maccatalyst-arm64 version 17.2.8053...
Installing pack Microsoft.MacCatalyst.Templates version 17.2.8053...
Writing workload pack installation record for Microsoft.MacCatalyst.Templates.net8 version 17.2.8053...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-arm64 version 8.0.4...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.maccatalyst-arm64 version 8.0.4...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64 version 8.0.4...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.maccatalyst-x64 version 8.0.4...
Installing pack Microsoft.NETCore.App.Runtime.Mono.maccatalyst-arm64 version 8.0.4...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.maccatalyst-arm64 version 8.0.4...
Installing pack Microsoft.NETCore.App.Runtime.Mono.maccatalyst-x64 version 8.0.4...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.maccatalyst-x64 version 8.0.4...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-arm64 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.net7.maccatalyst-arm64 version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.net7.maccatalyst-x64 version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.Mono.maccatalyst-arm64 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.net7.maccatalyst-arm64 version 7.0.18...
Installing pack Microsoft.NETCore.App.Runtime.Mono.maccatalyst-x64 version 7.0.18...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.net7.maccatalyst-x64 version 7.0.18...
Garbage collecting for SDK feature band(s) 8.0.200...

Successfully installed workload(s) maui.

Finishing: Install Maui workload
PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @PureWeen on Thursday, April 11, 2024 10:18:56 PM

@jonathanpeppers thoughts?

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @jonathanpeppers on Friday, April 12, 2024 2:15:19 AM

@JoacimWall can you share more complete sample code?

You might be able to test with: https://httpbin.org/

@simonrozsival @grendello could this be related to some of the changes here:

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @JoacimWall on Friday, April 12, 2024 6:25:07 AM

@JoacimWall can you share more complete sample code? You might be able to test with: https://httpbin.org/ @simonrozsival @grendello could this be related to some of the changes here:

Hi I am not responsible for the backend so I can't share the code for that. I have now shared how the front end logic works and a dummy backend but this is not the real backend.

https://github.com/JoacimWall/Joacim_Bug_Report/tree/main/GzipBug

I ma running on a Mac and i have not bin able to get the simulator to accept connect to local server.

Hi I have now published a dummy API on Azure (the code is in the repo also). if I debug on real Android device(samsung sm-g96OF (API29)) I get the error now. If you use iOS the api works. //Joacim

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @simonrozsival on Friday, April 12, 2024 7:27:32 AM

@jonathanpeppers @grendello This change is probably relevant: https://github.com/xamarin/xamarin-android/pull/8753

@JoacimWall could you please try setting handler.AutomaticDecompression = DecompressionMethods.None; or handler.AutomaticDecompression = DecompressionMethods.GZip; here: https://github.com/JoacimWall/Joacim_Bug_Report/blob/main/GzipBug/GzipClient/HttpsClientHandlerService.cs#L9?

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @jonathanpeppers on Friday, April 12, 2024 1:40:41 PM

It seems like the BCL doesn't have automatic decompression by default:

So, was AndroidMessageHandler auto decompressing in the past?

It does seem like we should match BCL behavior, just unfortunate it changed in a service release.

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @JoacimWall on Thursday, May 2, 2024 5:53:32 AM

Hello I do not understand what you need for more information. I think that I can't do more to solve this bug. I reported this bug when something that worked in the previous version of MAUI now stopped working, so that others that encounter the same error could know what the issue is. We have 3 MAUI customer apps in production that are still on version 8.0.6 because there are new bugs in the new SRs that mean we cannot update.

//Regards Joacim

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @simonrozsival on Thursday, May 2, 2024 8:45:08 AM

@JoacimWall I noticed that in the GzipBug repro project you set the default Accept-Encoding headers here: https://github.com/JoacimWall/Joacim_Bug_Report/blob/main/GzipBug/GzipClient/RestClient.cs#L68-L70. The better way to do this is by setting the AutomaticDecompression property:

var handler = new HttpClientHandler
{
    AutomaticDecompression = DecompressionMethods.Gzip | DecompressionMethods.Deflate,
};
var client = new HttpClient(handler);
// ...

When I change the default AcceptEncoding headers for AutomaticDecompression, the repro app works correctly on Android. Does this solve your issue?

I'm slightly confused that you report that this used to work on Android in MAUI 8.0.6. I thought I might have broken the default behavior in https://github.com/xamarin/xamarin-android/pull/8753 but after revisiting it, I can't find a way that this could have been affected.

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @JoacimWall on Thursday, May 2, 2024 9:42:24 AM

@JoacimWall I noticed that in the GzipBug repro project you set the default Accept-Encoding headers here: https://github.com/JoacimWall/Joacim_Bug_Report/blob/main/GzipBug/GzipClient/RestClient.cs#L68-L70. The better way to do this is by setting the AutomaticDecompression property:

var handler = new HttpClientHandler
{
    AutomaticDecompression = DecompressionMethods.Gzip | DecompressionMethods.Deflate,
};
var client = new HttpClient(handler);
// ...

When I change the default AcceptEncoding headers for AutomaticDecompression, the repro app works correctly on Android. Does this solve your issue?

I'm slightly confused that you report that this used to work on Android in MAUI 8.0.6. I thought I might have broken the default behavior in xamarin/xamarin-android#8753 but after revisiting it, I can't find a way that this could have been affected.

Hi As I write above, this is from a production app where everything works, then we update to the latest version of MAUI no code changes and it stops working, so some type of breaking change. It's good that you can get around this by defining a specific HttpHandler or remove the GZIP row. But as you can see, this is not done in the example code, instead we run the standard declaration new Httpclient(); I think that I am not the only one that has built an app where we just do var client = new HttpClient(); and try to add
this.client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));

I spent some hours to find what it was that generate the error create I sample and report this issue. So others could skip spending the hours to find the root issue.

//Joacim

PureWeen commented 1 month ago

Issue moved from dotnet/maui#21786


From @simonrozsival on Thursday, May 2, 2024 12:58:27 PM

@JoacimWall thanks for the additional details.

@grendello do you have an idea if we could have broken this with https://github.com/xamarin/xamarin-android/pull/8753 or what else could have caused this regression?

grendello commented 1 month ago

@PureWeen yes, I think the change might have broken the sample code. In particular, this line might be overriding the headers, if the AutomaticDecompression is set by some code to DecompressionMethods.None. This would force us to set the identity encoding, thus overwriting the headers set by the sample code.

simonrozsival commented 1 month ago

@grendello I'm having problems reproducing the original working solution. I'm not getting any automatic decompression when I set just the default headers. The Mono.Android.dll that's being used is "34.0.1.78; git-rev-head:87e4a6d; git-branch:release/8.0.1xx" so it shouldn't be affected by #8753.

jamesingreersc commented 1 month ago

I had the exact same problem, http response was just gibberish. Had to roll VS back to previous version which fixed it.

AlleSchonWeg commented 1 month ago

I also have the same problem. My code is from a old XF App, which i triy to migrate. My question is: What is correct/new way to enable compression with MAUI for Android and iOS?

This:

var handler = new HttpClientHandler {
                AutomaticDecompression = DecompressionMethods.GZip,
            };
var client = new HttpClient(handler);

This handler add the AcceptEncoding Header for outgoing requests?

Thank you

simonrozsival commented 1 month ago

@AlleSchonWeg yes, this is the preferred way. The handler will automatically add the appropriate encoding headers and if the server responds with compressed data, it will automatically decompress it.

AlleSchonWeg commented 1 month ago

@simonrozsival Thank you. I tested it with Android:

var response = client .GetAsync("https://httpbingo.org/headers").Result;
var resultString = client.Content.ReadAsStringAsync().Result;

This was the reponse:

{
  "headers": {
    "Accept": [
      "application/json"
    ],
    "Accept-Encoding": [
      "gzip"
    ],
    "Accept-Language": [
      "en"
    ],
    "Host": [
      "httpbingo.org"
    ],
    "User-Agent": [
      "XXX0 (DEV; Phone; Google; sdk_gphone64_x86_64; Android 14)"
    ],
    "Via": [
      "1.1 fly.io"
    ],
    "X-Forwarded-For": [
      "YYY, ZZZ"
    ],
    "X-Forwarded-Port": [
      "443"
    ],
    "X-Forwarded-Proto": [
      "https"
    ],
    "X-Forwarded-Ssl": [
      "on"
    ],
    "X-Request-Start": [
      "t=1716456090916385"
    ]
  }
}

The Accept-Encoding was successfully set. Can't test iOS.

simonrozsival commented 1 month ago

@grendello @PureWeen I have tried again and I wasn't able to get the original described behavior to work. I'm not sure what caused this problem and how to fix it. Moreover, I'm not sure if we want to even support this behavior of enabling automatic decompression by setting the Accept-Encoding header instead of setting the AutomaticDecompression property.

grendello commented 1 month ago

@simonrozsival I looked at it a bit too and I have absolutely no clue why it worked before and stopped now. Clearly, it's some change in the http handler inheritance tree or thereabouts, but I think I agree with you that the way to do it by setting the property is the correct one. Especially that we're dealing with a "proxy" handler in this case, and most of the magic somehow happens in the Java backend we have no control over. As far as I can tell, we did nothing special before to make the scenario work, so I wouldn't exclude the possibility that the behavior might be different also because of a different target SDK when the app is built.

All in all, I'm in favor of documenting the issue and specifying what is the correct and supported way to achieve the desired outcome.

simonrozsival commented 4 days ago

All in all, I'm in favor of documenting the issue and specifying what is the correct and supported way to achieve the desired outcome.

Where would be a good place to document this? https://learn.microsoft.com/en-us/dotnet/maui/data-cloud/rest?view=net-maui-8.0 ?