HttpClient can't handle 307 or 308 status codes for large POST payloads

carl-di-ortus commented 1 year ago

Description

We have a HAProxy setup that handles various domains and subpaths. For some paths we do a redirect with HTTP 307 status code, because we need the client to repeat POST request to a different endpoint. HttpClient throws exception when trying to send request without even getting the redirect location. Same behavior observed if redirect response returns with 308 status code.

Request always passes successfully when compiled to target 4.8 framework. Request passes with small payloads on .NET7, fails on large payloads (tried a valid JSON 202759 chars long). Didn't try to find the exact threshold.

Might be related to #47706

Reproduction Steps

using System;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp2
{
    internal class RedirectPost
    {
        static async Task Main(string[] args)
        {
            var url = "https://demo.addosign.net/echo?q=query"; // returns HTTP 307 redirect to https://postman-echo.com/post?q=query

            //var json = "{\"var\": \"value\"}"; // small payload - always success

            var filename = "C:\\Users\\username\\example.json"; // 202759 chars minimized valid JSON
            string json;
            using (var sr = new StreamReader(filename))
            {
                json = sr.ReadToEnd();
                Console.WriteLine(json.Length);
            }

            try
            {
                using (var client = new HttpClient())
                {
                    var response = await client.PostAsync(url, new StringContent(json, Encoding.UTF8, "application/json"));
                    var body = await response.Content.ReadAsStringAsync();

                    if (response.IsSuccessStatusCode)
                    {
                        Console.WriteLine($"success, body length {body.Length}");
                    }
                    else
                    {
                        Console.WriteLine(body);
                        Console.WriteLine(response.StatusCode);

                    }
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
        }
    }
}

Expected behavior

Request should correctly redirect to final location and send POST request there.

Actual behavior

Large payloads throws exception

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|153_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn)
   at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source, Boolean async)
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ConsoleApp2.RedirectPost.Main(String[] args) in C:\Users\username\source\repos\sandboxes\test-project\ConsoleApp1\RedirectPost.cs:line 28

Regression?

Haven't tested a lot of build targets, but I suspect it's a regression since reimplementing HttpClientHandler back in .NET Core 2.1.

Known Workarounds

No response

Configuration

No response

Other information

Attaching WSL curl command outputs here for reference. Curl never fails with small or large payloads.

> curl -vL --data "{\"var\":\"value\"}" --header "Content-Type: application/json" -X POST https://demo.addosign.net/echo?q=query

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 94.137.136.108:443...
* Connected to demo.addosign.net (94.137.136.108) port 443 (#2)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=DK; ST=K�benhavn; L=K�benhavn; O=twoday A/S; CN=*.addosign.net
*  start date: Dec 13 09:49:02 2022 GMT
*  expire date: Jan 14 09:49:01 2024 GMT
*  subjectAltName: host "demo.addosign.net" matched cert's "*.addosign.net"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /echo?q=query HTTP/1.1
> Host: demo.addosign.net
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 15
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 307 Temporary Redirect
< content-length: 0
* Please rewind output before next send
< location: https://postman-echo.com/post?q=query
< cache-control: no-cache
< connection: close
<
* Closing connection 2
* TLSv1.3 (IN), TLS alert, close notify (256):
* TLSv1.3 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://postman-echo.com/post?q=query'
... <cropped>

curl -vL --data "@example.json" --header "Content-Type: application/json" -X POST https://demo.addosign.net/echo

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 94.137.136.108:443...
* Connected to demo.addosign.net (94.137.136.108) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=DK; ST=K�benhavn; L=K�benhavn; O=twoday A/S; CN=*.addosign.net
*  start date: Dec 13 09:49:02 2022 GMT
*  expire date: Jan 14 09:49:01 2024 GMT
*  subjectAltName: host "demo.addosign.net" matched cert's "*.addosign.net"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /echo HTTP/1.1
> Host: demo.addosign.net
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 203829
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 307 Temporary Redirect
< content-length: 0
* Please rewind output before next send
< location: https://postman-echo.com/post?
< cache-control: no-cache
< connection: close
* Keep sending data to get tossed away
<
* we are done reading and this is set to close, stop send
* Closing connection 0
* TLSv1.3 (IN), TLS alert, close notify (256):
* TLSv1.3 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://postman-echo.com/post'
... <cropped>

The only thing that differs with large requests are these lines

* Keep sending data to get tossed away
<
* we are done reading and this is set to close, stop send

But it works correctly even with it.

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

### Description We have a HAProxy setup that handles various domains and subpaths. For some paths we do a redirect with HTTP 307 status code, because we need the client to repeat POST request to a different endpoint. HttpClient throws exception when trying to send request without even getting the redirect location. Same behavior observed if redirect response returns with 308 status code. Request always passes successfully when compiled to target 4.8 framework. Request passes with small payloads on .NET7, fails on large payloads (tried a valid JSON 202759 chars long). Didn't try to find the exact threshold. Might be related to #47706 ### Reproduction Steps ``` c# using System; using System.Net.Http; using System.Text; using System.Threading.Tasks; namespace ConsoleApp2 { internal class RedirectPost { static async Task Main(string[] args) { var url = "https://demo.addosign.net/echo?q=query"; // returns HTTP 307 redirect to https://postman-echo.com/post?q=query //var json = "{\"var\": \"value\"}"; // small payload - always success var filename = "C:\\Users\\username\\example.json"; // 202759 chars minimized valid JSON string json; using (var sr = new StreamReader(filename)) { json = sr.ReadToEnd(); Console.WriteLine(json.Length); } try { using (var client = new HttpClient()) { var response = await client.PostAsync(url, new StringContent(json, Encoding.UTF8, "application/json")); var body = await response.Content.ReadAsStringAsync(); if (response.IsSuccessStatusCode) { Console.WriteLine($"success, body length {body.Length}"); } else { Console.WriteLine(body); Console.WriteLine(response.StatusCode); } } } catch (Exception e) { Console.WriteLine(e); } } } } ``` ### Expected behavior Request should correctly redirect to final location and send POST request there. ### Actual behavior Large payloads throws exception ``` System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.. ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.g__CompleteWriteAsync|153_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source, Boolean async) at System.Net.Http.HttpContent.g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at ConsoleApp2.RedirectPost.Main(String[] args) in C:\Users\username\source\repos\sandboxes\test-project\ConsoleApp1\RedirectPost.cs:line 28 ``` ### Regression? Haven't tested a lot of build targets, but I suspect it's a regression since [reimplementing HttpClientHandler back in .NET Core 2.1](https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler?view=net-5.0#remarks). ### Known Workarounds _No response_ ### Configuration _No response_ ### Other information _No response_

Author:	carl-di-ortus
Assignees:	-
Labels:	`area-System.Net.Http`
Milestone:	-

wfurt commented 1 year ago

Do you have packet capture? From the exception it seems like the server is also shutting down the connection. This may be good to enable Expect100Continue.

carl-di-ortus commented 1 year ago

I'm sorry, I'm not that skilled to understand packet capturing.

Yes server is closing the connection when returning redirect.

By "enable Expect100Continue" do you mean this ServicePointManager.Expect100Continue? It looks like the default value is already true. I tried setting this true/false - no effect, still throwing same exception.

Then I tried ServicePointManager.FindServicePoint() method to return ServicePoint instance, even though it says it's deprecated, and setting Expect100Continue there - no effect.

Then I tried increasing SocketsHttpHandler.Expect100ContinueTimeout value and passing this handler to client - no effect.

Then while debugging, I noticed HttpClient.DefaultRequestHeaders.ExpectContinue property - which by default is null. I manually set this to true:

using (var client = new HttpClient())
{
    client.DefaultRequestHeaders.ExpectContinue = true;
    var response = await client.PostAsync(url, new StringContent(json, Encoding.UTF8, "application/json"));

And suddenly it all worked! I'm not really happy with this approach, I would call it a workaround at best, since documentation for ServicePointManager.Expect100Continue says it already defaulting to true, but inside HttpClient.DefaultRequestHeaders this header is nullable boolean defaulting to null.

Our clients (and probably a lot of other people around the world) wouldn't know when they are receiving normal response, when they are being redirected to receive response there. Documentation on HttpClient.DefaultRequestHeaders.ExpectContinue is very scarce, not comparing to what ServicePointManager has.

wfurt commented 1 year ago

ServicePointManager is obsolete and does nothing for HttpClient in .NET Core. The documentation discrepancy comes with age - .NET Framework is around for decades and many articles were written before .NET Core even existed so they can be misleading.

Behavior of the 100Continue is described in HTTP RFC and there are just different ways how to set it on the request.

The problem is server closing connection in middle of the request. HttpClient sees that as IO error and aborts the processing. Perhaps something we can improve in the future to improve compatibility with .NET Framework.

dotnet / runtime