HttpClient on Android sends wrong content-type header with ByteArrayContent

[rockfordlhotka]

Description

In theory, the content-type header should be automatically set based on the type used to set the Content property of an http request. As a result, the content-type for a ByteArrayContent object should match up to some binary content type value.

      var client = new HttpClient();
      var content = new ByteArrayContent(serialized);
      var httpResponse = await client.PostAsync(
        "<server url>",
        content);

In practice this is what happens:

Platform	Behavior
Windows	content-type is not set at all
Android	content-type is set to application/x-www-form-urlencoded

The Windows behavior, while incorrect, at least works.

The Android behavior doesn't work at all, because the aspnetcore server doesn't properly take the binary payload and make it available in Request.Body like it should.

The result is that we can't post binary data from a MAUI app on Android to aspnetcore. Accounts of this exist on stackoverflow and in CSLA.

Steps to Reproduce

1. Create an Android MAUI app and a Console app with code like this:

      var content = new ByteArrayContent(serialized);
      var httpResponse = await client.PostAsync(
        "<server url>",
        content);

2. Create an ASP.NET Core web site with a controller like this:

    [HttpPost]
    public virtual async Task PostAsync([FromQuery] string operation)
    {
        byte[] data;
        using (var requestBodyBuffer = new MemoryStream())
        {
          await Request.Body.CopyToAsync(requestBodyBuffer).ConfigureAwait(false);
          Request.Body.Position = 0;
          data = requestBodyBuffer.ToArray();
        }
    }

3. Run the web server
4. Run the console app and it works - data has data
5. Run the Android app and it fails - data is empty
6. Use something like ngrok to see what's going over the wire - the exact same binary payload goes over the wire in both cases; the http content-type value is set wrong from Android and not at all from Windows

Link to public reproduction project repository

No response

Version with bug

8.0.20 SR4

Is this a regression from previous behavior?

Not sure, did not test other versions

Last version that worked well

Unknown/Other

Affected platforms

Android

Affected platform versions

No response

Did you find any workaround?

The workaround is to base64 encode the binary data and send it to the server using a StringContent object instead of the broken ByteArrayContent type.

Obviously the server endpoint has to accept and decode the string from Android and the binary payload from all other platforms.

Relevant log output

No response

[rockfordlhotka]

More information, as I still think this is a bug.

This is with help from @dotMorten

It turns out that the default HttpClient handler on Android is one that (according to the docs) is from .NET Core 2.0 and earlier: HttpClientHandler

My guess is that it is too old to understand the new Content property behaviors in .NET 8 - though that's just a guess.

The workaround is to configure the HttpClient service (or instance) to use a different handler. So in the MauiProgram.cs file define the HttpClient service like this:

  builder.Services.AddScoped<HttpClient>((p) => new HttpClient(new SocketsHttpHandler()));

This way the HttpClient used by the app will properly handle the ByteArrayContent content object type.

The reason I still think it is a bug, is that the exact same code using HttpClient acts differently on Android from other platforms.

Or if it isn't a bug, there should be much better documentation, as I spent hours searching the web. Finally had to break down and use evil Twitter to get an answer 😢

[drasticactions]

@jonathanpeppers Would this be a runtime issue? I don't think anything in the MAUI UI repo influences this.

[rockfordlhotka]

@drasticactions it is odd that this only happens in Android, and not Windows - so something is different in the Xamarin/MAUI Android environment compared to others.

[dotMorten]

I wonder if you see the same issue on Windows if you use the HttpClientHandler there?

[drasticactions]

@drasticactions it is odd that this only happens in Android, and not Windows - so something is different in the Xamarin/MAUI Android environment compared to others.

It's not odd. The implementation of HttpClient is different between the platforms, since they generally interoperate with the native HTTP stacks and not with the managed .NET versions. So the platform you're running on can have differences depending on what changes were made in the runtime.

Also, Xamarin.Android runs on Mono, .NET Android runs on .NET via Mono, and there could be differences between the two depending on what has changed.

[jonathanpeppers]

@simonrozsival or @grendello might be able to take a look.

@rockfordlhotka is there an example endpoint we could test this at: https://httpbin.org/

[simonrozsival]

@rockfordlhotka the underlying Java HTTP library will not allow you to send content without specifying some Content-Type header. If you don't provide any (and ByteArrayContent doesn't have any default content type), it will use the default one you're seeing. Fortunately, it's easy to set any content type you want:

var content = new ByteArrayContent(serialized);
content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");

Or if it isn't a bug, there should be much better documentation, as I spent hours searching the web. Finally had to break down and use evil Twitter to get an answer 😢

You're right that the documentation of differences between different platforms is currently lacking.

I wonder if you see the same issue on Windows if you use the HttpClientHandler there?

AFAIK HttpClientHandler on Windows uses SocketsHttpHandler under the hood by default.

[rockfordlhotka]

@simonrozsival or @grendello might be able to take a look.

@rockfordlhotka is there an example endpoint we could test this at: https://httpbin.org/

I created a simple solution that demonstrates the issue:

https://github.com/rockfordlhotka/AndroidBinaryPost

In MainProgram.cs you can set the server address (I used ngrok as you'll see).

Also in MainProgram.cs you can see where the HttpClient service is created. The default HttpClient is what fails on Android, but works elsewhere. The other uses the socket handler and works on Android and Windows.

If we decide this is (somehow) intended behavior, then it is a failure of documentation to tell everyone that the default behavior of HttpClient is chaotic evil (to use a D&D term).

[rockfordlhotka]

@rockfordlhotka the underlying Java HTTP library will not allow you to send content without specifying some Content-Type header. If you don't provide any (and ByteArrayContent doesn't have any default content type), it will use the default one you're seeing. Fortunately, it's easy to set any content type you want:

var content = new ByteArrayContent(serialized);
content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");

Or if it isn't a bug, there should be much better documentation, as I spent hours searching the web. Finally had to break down and use evil Twitter to get an answer 😢

You're right that the documentation of differences between different platforms is currently lacking.

I wonder if you see the same issue on Windows if you use the HttpClientHandler there?

AFAIK HttpClientHandler on Windows uses SocketsHttpHandler under the hood by default.

This doesn't work by the way. I tried it this weekend, and

content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");

Has absolutely no impact on the header actually passed over the network, which continues to be application/x-www-form-urlencoded.

[simonrozsival]

This doesn't work by the way. I tried it this weekend, and

content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");

Has absolutely no impact on the header actually passed over the network, which continues to be application/x-www-form-urlencoded.

@rockfordlhotka This is interesting. In my case the header is correctly set with both the AndroidMessageHandler and SocketsHttpHandler. Would you mind sharing a full repro project that I could try? I wonder what prevents the handler to correctly set the header in your case.

My repro is based on dotnet new maui (.NET 8) with the following change to MainPage.cs

public partial class MainPage : ContentPage
{
    Task? _task;

    public MainPage()
    {
        InitializeComponent();
    }

    private void OnCounterClicked(object sender, EventArgs e)
    {
        _task = Task.Run(TestByteArrayContentWithCustomContentTypeHeader);
    }

    private async Task TestByteArrayContentWithCustomContentTypeHeader()
    {
        var content = new ByteArrayContent([1, 2, 3, 4, 5, 6]);
        content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
        var client = new HttpClient();
        // var client = new HttpClient(new SocketsHttpHandler());
        var httpResponse = await client.PostAsync("https://httpbin.org/post", content);
        var response = await httpResponse.Content.ReadAsStringAsync();

        Console.WriteLine($"--- Request ---");
        Console.WriteLine(httpResponse.RequestMessage);
        Console.WriteLine($"--- Response ---");
        Console.WriteLine(httpResponse);
        Console.WriteLine($"--- Response content ---");
        Console.WriteLine(response);
    }
}

This is the output I'm seeing in logcat with AndroidMessageHandler:

--- Request ---
Method: POST, RequestUri: 'https://httpbin.org/post', Version: 1.1, Content: System.Net.Http.ByteArrayContent, Headers:
{
  Content-Type: application/octet-stream
  Content-Length: 6
}
--- Response ---
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Access-Control-Allow-Credentials: true
  Access-Control-Allow-Origin: *
  Connection: keep-alive
  Date: Mon, 22 Apr 2024 09:26:35 GMT
  Server: gunicorn/19.9.0
  X-Android-Received-Millis: 1713777995247
  X-Android-Response-Source: NETWORK 200
  X-Android-Selected-Protocol: http/1.1
  X-Android-Sent-Millis: 1713777995114
  Content-Length: 498
  Content-Type: application/json
}
--- Response content ---
{
  "args": {}, 
  "data": "\u0001\u0002\u0003\u0004\u0005\u0006", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Content-Length": "6", 
    "Content-Type": "application/octet-stream", 
    "Host": "httpbin.org", 
    "User-Agent": "Dalvik/2.1.0 (Linux; U; Android 14; SM-S911B Build/UP1A.231005.007)", 
    "X-Amzn-Trace-Id": "Root=1-66262d4b-1bcd65cf4189ab557de82280"
  }, 
  "json": null, 
  "origin": "***", 
  "url": "https://httpbin.org/post"
}
^C

and this one I'm getting with SocketsHttpHandler:

--- Request ---
Method: POST, RequestUri: 'https://httpbin.org/post', Version: 1.1, Content: System.Net.Http.ByteArrayContent, Headers:
{
  Content-Type: application/octet-stream
  Content-Length: 6
}
--- Response ---
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
  Date: Mon, 22 Apr 2024 09:31:18 GMT
  Connection: keep-alive
  Server: gunicorn/19.9.0
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Credentials: true
  Content-Type: application/json
  Content-Length: 376
}
--- Response content ---
{
  "args": {}, 
  "data": "\u0001\u0002\u0003\u0004\u0005\u0006", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Content-Length": "6", 
    "Content-Type": "application/octet-stream", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-66262e66-53d7c74532f13bca38139e66"
  }, 
  "json": null, 
  "origin": "***", 
  "url": "https://httpbin.org/post"
}

[rockfordlhotka]

@simonrozsival ok, I re-read your post.

So why does PostAsync work for you and fail for me? I was able to explicitly set the content-type when using SendAsync, but not PostAsync.

Here's the branch I created that demonstrates the issue: https://github.com/rockfordlhotka/AndroidBinaryPost/tree/explicit-type

Most notably, the commented block here fails, and the code here works.

[simonrozsival]

PostAsync is a very thin wrapper around SendAsync, I can't see how that would change the behavior this significantly:

https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs#L387-L401

Can you, just as a sanity check, run the code against https://httpbin.org/post?

[rockfordlhotka]

@simonrozsival I did the sanity check and it worked. So I switched back to my server and it worked. Clearly I messed something up in my original test run 😳

So I think my takeaway from this whole thing is that the default content type for ByteArrayContent is random, and (imo) the HttpClient API (or an analyzer) should fail the build if someone writes code that doesn't explicitly set the content-type for the content.

Sure, it'll work in Windows and Linux, so most people never encounter this issue. But realistically, something is broken here imo. Docs, API, analyzers. People shouldn't fall into a hole where bing/google can't help and only @dotMorten can save the day (as cool as he is!).

[simonrozsival]

I'm glad you got it working in the end 😄

But realistically, something is broken here imo. Docs, API, analyzers. People shouldn't fall into a hole where bing/google can't help and only @dotMorten can save the day (as cool as he is!).

I agree, it shouldn't be this hard and confusing.

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

[vitek-karas]

@simonrozsival is this a docs issue? If so, please move it to the docs repo.

[simonrozsival]

@rockfordlhotka @vitek-karas closing in favor of https://github.com/dotnet/docs/issues/41683