Microsoft.AspNetCore.TestHost.TestServer - Client hangs when writing large data synchronously

[tkvalvik]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

The client produced by TestServer.CreateClient() hangs while writing requests, if the written is sufficiently larger than the buffersize to the underwlying writer, and it is written synchronously.

Following minimal example with a custom content class:

public class CustomContent : HttpContent
    {
        private readonly string _content;

        public MessageContent(string content)
        {
            _content = content;
        }

        protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
        {
            var streamWriter = new StreamWriter(stream);
            streamWriter.Write(_content); // never return from here
            await streamWriter.FlushAsync();
        }

        protected override bool TryComputeLength(out long length)
        {
            length = -1;
            return false;
        }
    }

And the following testcode using the default WebApi-template, but the subject being tested doesn't matter here:

            var server = new Microsoft.AspNetCore.TestHost.TestServer(
                new WebHostBuilder()
                    .UseTestServer()
                    .UseStartup(typeof(WeatherForecastController).Assembly.FullName)
                );

            await server.Host.StartAsync();

            var client = server.CreateClient();
            var content = string.Join("", Enumerable.Repeat("a", 100000));

            var res = await client.PostAsync("http://localhost/AnythingGoesWontBeHit", new CustomContent(content));

This will never complete. This is an issue specifically because this is how Newtonsoft.Json.JsonTextWriter writes to a TextWriter, causing a custom HttpContent using newtonsoft to lock up the test suite if too large test data is used.

The issue goes away if the buffer size for the StreamWriter is increased. How much is uknown. It does not need to be larger than the message, but too small will cause it to lock up.

Expected Behavior

The request data should be written even if it is significantly larger than the StreamWriter's buffer.

Steps To Reproduce

1. Create a TestHost for a minimal server.
2. Run the code above.

Exceptions (if any)

No response

.NET Version

6.0.100

Anything else?

Tested with the current dotnet new webapi-template and Microsoft.AspNetCore.TestHost 6.0.2.

Seems to apply to 3.1.x versions as well.

[Tratcher]

If you attach the debugger and pause the app, what is the full callstack for the blocked call?

The SerializeToStreamAsync is writing into a Pipe, which limits how much data it will buffer before the receiving app must consume some. https://github.com/dotnet/aspnetcore/blob/c65e5ae81e5e77eca595395156ab1ba8a4011937/src/Hosting/TestHost/src/HttpContextBuilder.cs#L43

[tkvalvik]

Full stacktrace:

    [Async] System.IO.Pipelines.dll!System.IO.Pipelines.PipeWriterStream.GetFlushResultAsTask.__AwaitTask|28_0(System.Threading.Tasks.ValueTask<System.IO.Pipelines.FlushResult> valueTask) Unknown
    System.IO.Pipelines.dll!System.IO.Pipelines.PipeWriterStream.Write(byte[] buffer, int offset, int count)    Unknown
    System.Private.CoreLib.dll!System.IO.Stream.Write(System.ReadOnlySpan<byte> buffer) Unknown
>   System.Private.CoreLib.dll!System.IO.StreamWriter.Flush(bool flushStream, bool flushEncoder)    Unknown
    System.Private.CoreLib.dll!System.IO.StreamWriter.Write(string value)   Unknown
    Tests.dll!Tests.MessageContent.SerializeToStreamAsync(System.IO.Stream stream, System.Net.TransportContext context) Line 61 C#

[Tratcher]

There should be more to that stack. I think it looks something like this: https://github.com/dotnet/aspnetcore/blob/1c32a6df6ebd96300df6b96ec8a1ea718c74780d/src/Hosting/TestHost/src/ClientHandler.cs#L70-L87 https://github.com/dotnet/aspnetcore/blob/1c32a6df6ebd96300df6b96ec8a1ea718c74780d/src/Hosting/TestHost/src/HttpContextBuilder.cs#L78

If you can't make the CustomContent fully async, try adding a await Task.Yield(); at the start of SerializeToStreamAsync to avoid synchronously blocking the caller.

[tkvalvik]

This is not a support request for a workaround. It is a report that the client supplied by the test host seems to have a defect that causes tests to block indefinably where a request against a real HttpClient would always succeed. The example I gave was a minimal example to reproduce the issue, not the actual production code that showed the problem for us.

The production code that revealed the problem does several things that would not contribute to this bug report, but it ends up sometimes wrapping the stream in a Newtonsoft.Json.JsonTextWriter. Newtonsoft.Json writes synchronously, so it would require Stream.Write to work and not have hidden limits on data sizes, and failure modes that are hard to figure out and debug.

The supplied stacktrace is what Visual Studio is presenting to me.

[Tratcher]

I suggested that workaround to verify our understanding of the issue, and to unblock your development scenario. If it works then we can consider incorporating a similar fix into the product.

[tkvalvik]

I will have to check when I get into the office in the morning. (Evening now, my timezone) It's easy enough to reproduce though. dotnet new webapi and dotnet new nunit. Add to same solution, add the TestHost-package and paste my code above into the test.

Our specific issue was already worked around by making the buffer size of the TextWriter configurable, and setting it to a large value in the testcases that have large test data.

[adityamandaleeka]

Triage: given that there's a workaround, moving this to backlog. If we confirm that the suggested workaround works, we might take a PR for it.

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[ztatw]

I think I have a similar issue, here is the windbg screenshot, this is a test case, send some guids to the api, when I tried send 2200 guids, it hangs, when I decreased to 1600, it works fine, actually 1700 will hangs the test case.

I also tried to replace with a JsonContent.Create(myRequest), it works for me.

I can not reproduce with a new repo.