HttpMessageContent breaks on dotnet core 2.1 on Mac

[aliostad]

[Originally created here https://github.com/dotnet/corefx/issues/31918 but was closed there]

Hi,

[NOT ENTIRELY SURE IF THIS IS ASP.NET OR COREFX]

I have an HTTP Caching library for .NET and I use HttpMessageContent class to help me serialise and deseralise the messages. This has been working throughout including .NET Core 2.0 but it seems to have been broken by .NET Core 2.1 on Mac:

System.InvalidOperationException: Error parsing HTTP message header byte 697 of message System.Byte[].
   at System.Net.Http.HttpContentMessageExtensions.ReadAsHttpResponseMessageAsyncCore(HttpContent content, Int32 bufferSize, Int32 maxHeaderSize, CancellationToken cancellationToken)
   at CacheCow.Client.MessageContentHttpMessageSerializer.DeserializeToResponseAsync(Stream stream)
   at CacheCowCrashRepro.TestCacheStore.AddOrUpdateAsync(CacheKey key, HttpResponseMessage response) in C:\Users\james\Documents\Projects\CacheCowCrashRepro\CacheCowCrashRepro\Program.cs:line 45
   at CacheCow.Client.CachingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at CacheCowCrashRepro.Program.Main(String[] args) in C:\Users\james\Documents\Projects\CacheCowCrashRepro\CacheCowCrashRepro\Program.cs:line 19} System.Exception {System.InvalidOperationException

Here is the repro code. Works with netcoreapp2.0 but breaks with netcoreapp2.1.

Project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <LangVersion>latest</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNet.WebApi.Client" Version="5.2.6" />
  </ItemGroup>
</Project>

Program.cs:

using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;

namespace CacheCowCrashRepro
{
    class Program
    {
        static async Task Main(string[] args)
        {
            try
            {
                var serializer = new MessageContentHttpMessageSerializer();
                var client = new HttpClient();
                var request = new HttpRequestMessage(HttpMethod.Get, new Uri("https://google.com"));
                var response = await client.SendAsync(request);
                var ms = new MemoryStream();
                await serializer.SerializeAsync(response, ms);
                ms.Position = 0;
                // var bytes = ms.ToArray();
                // File.WriteAllBytes("response.bin", bytes); // to store 
                var r2 = await serializer.DeserializeToResponseAsync(ms);
                Console.WriteLine(response);
            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }
        }
    }

    public class MessageContentHttpMessageSerializer
    {
        private bool _bufferContent;

        public MessageContentHttpMessageSerializer()
            : this(true)
        {

        }

        public MessageContentHttpMessageSerializer(bool bufferContent)
        {
            _bufferContent = bufferContent;
        }

        public async Task SerializeAsync(HttpResponseMessage response, Stream stream)
        {
            if (response.Content != null)
            {
                if (_bufferContent)
                    await response.Content.LoadIntoBufferAsync();
            }

            var httpMessageContent = new HttpMessageContent(response);
            var buffer = await httpMessageContent.ReadAsByteArrayAsync();
            stream.Write(buffer, 0, buffer.Length);
        }

        public async Task SerializeAsync(HttpRequestMessage request, Stream stream)
        {
            if (request.Content != null && _bufferContent)
            {
                await request.Content.LoadIntoBufferAsync();
            }

            var httpMessageContent = new HttpMessageContent(request);
            var buffer = await httpMessageContent.ReadAsByteArrayAsync();
            stream.Write(buffer, 0, buffer.Length);
        }

        public async Task<HttpResponseMessage> DeserializeToResponseAsync(Stream stream)
        {
            var response = new HttpResponseMessage();
            response.Content = new StreamContent(stream);
            response.Content.Headers.Add("Content-Type", "application/http;msgtype=response");
            var responseMessage = await response.Content.ReadAsHttpResponseMessageAsync();
            if (responseMessage.Content != null && _bufferContent)
                await responseMessage.Content.LoadIntoBufferAsync();
            return responseMessage;
        }

        public async Task<HttpRequestMessage> DeserializeToRequestAsync(Stream stream)
        {
            var request = new HttpRequestMessage();
            request.Content = new StreamContent(stream);
            request.Content.Headers.Add("Content-Type", "application/http;msgtype=request");
            var requestMessage = await request.Content.ReadAsHttpRequestMessageAsync();
            if (requestMessage.Content != null && _bufferContent)
                await requestMessage.Content.LoadIntoBufferAsync();
            return requestMessage;
        }
    }
}

Here is the response I get which is nothing special. The only thing I notice is that there is no ContentLength header and encoding is chunked but looking at the message, I could not see a chunked encoding, the response is all in one block - maybe I missed.

response.bin

[mkArtakMSFT]

@aliostad we'll have a look and get back to you.

[danroth27]

This has been working throughout including .NET Core 2.0 but it seems to have been broken by .NET Core 2.1 on Mac.

@aliostad Broken how? Are you getting some sort of error?

[dougbu]

@aliostad the response.bin trace you provided seems to contain the HTML for a Google search form. We'll need more information along the lines of @danroth27's questions to investigate further.

[aliostad]

@danroth27 sorry I thought I put the error message but I have not. OK updated the issue.

[aliostad]

@dougbu the point of that file is that the payload is NOT chunked encoding and I have a feeling it might be somehow related.

I have given you repro code, just run it and see.

[dougbu]

@aliostad I can't get to this immediately but will test things out within a week.

[aliostad]

Any news? It is more than a week.

[dougbu]

My apologies @aliostad. Due to other priorities, I'm not exactly sure when I'll get back to this investigation. Will let you know as soon as I've had a chance to look at this.

[xela30]

Any updates, guys? BTW, it doesn't work on Windows either.

[aliostad]

Yes, please at least tell us if you can confirm it is a bug at your side?

[dougbu]

I should have some time to look at this issue next week.

[dougbu]

@aliostad I'm about to reactivate dotnet/corefx#31918 because this appears to be a break between .NET Core 2.0 and 2.1. w.r.t. how they parse date headers. In 2.0, Expires: -1 is treated as valid. In 2.1, the parsers throw a FormatException on this header. I'll add more information to dotnet/corefx#31918.

Note the issue doesn't seem to have anything to do with whether the response is chunked.