HTTP/2 multiplexing performance issues

[avendel]

Update: I have only been able to reproduce the behavior with requests blocking for several seconds under certain network conditions. So this might only partially be a problem with ASP.NET Core.

Writing to the response body stream with HttpContext.Response.Body.WriteAsync blocks other concurrent requests.

When server code writes data to the response stream in multiple writes - e.g. a large download - and the client calls the same code for more data, these calls will take an unexpectedly long time to finish.

• This is mostly a problem with IIS hosting. Kestrel performs much better, although not as fast as with HTTP/1.1.
• This is not only a problem with IIS. I cannot reproduce it with .NET Framework.
• This is only reproducible with HTTP/2.
• A workaround in IIS is to either use Response.BodyWriter.WriteAsync or to issue a Response.Body.FlushAsync after each write. This will result in response times comparable to those with Kestrel, although not as fast as with HTTP/1.1.
• I can reproduce this on .NET Core 3.1 and 5.0.
• I cannot reproduce the worst observed behavior under all network conditions.

Here is a small repro for testing. Consider the following server code, wich I think is a supported way of producing response content.

app.UseRouting();
app.UseEndpoints(endpoints =>
{
    endpoints.MapGet("/data", async context =>
    {
        int length = int.Parse(context.Request.Query["length"]);
        int written = 0;
        while (written < length)
        {
            var chunk = new byte[Math.Min(length - written, 4096)];
            await context.Response.Body.WriteAsync(chunk.AsMemory());
            written += chunk.Length;
        }
    });
});

The FileResultExecutorBase uses a similar approach for writing the response, so the behavior can be reproduced with MVC actions returning FileStreamResult or FileContentResult.

Here is some client code for testing in a browser:

<script language="javascript">
    function fetchData() {
        for (let i = 0; i < 5; i++) {
            fetch("data?length=102400")
                .then(response => response.arrayBuffer()
                .then(content => console.log(content.byteLength)));
        }
    }
</script>
<a href="data?length=1073741824">Download 1GB</a>
<button onclick="fetchData()">Fetch 100KB</button>

Steps to reproduce

1. Click Download 1GB to start downloading data.
2. Click Fetch 100KB to start five concurrent requests for a smaller amount of data.

Observed behavior

The five requests for the 100KB take an unreasonably long time to finish, not simply explainable by the bandwidth usage of the large download. Over a 100Mbit network with a 17ms ping round trip time, the first one takes a couple of seconds, and the last around 7 seconds, in Edge. In FireFox, they don't seem to finish until I cancel the large download.

With HTTP/1.1, they finish within a few tens of ms. In Kestrel (HTTP/2), they finish within a few hundred milliseconds.

Using a 80KB buffer in the repro has the same results.

[BrennanConroy]

Triage: We believe this is an issue with the internal pipewriter and its weird behavior with flushes/cancelpendingreads.

[ghost]

Thanks for contacting us.

We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[adityamandaleeka]

Triage: there is a chance this may have helped: https://github.com/dotnet/aspnetcore/pull/34733

Need to investigate.

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.