Why Response.BodyString contains HeaderText?

[AxisRay]

dotnet --info
.NET SDK
 Version:   6.0.101
 Commit:    ef49f6213a

运行时环境:
 OS Name:     Windows
 OS Version:  10.0.22000
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.101\

Host (useful for support):
  Version: 6.0.1
  Commit:  3a25a7f1cc

.NET SDKs installed:
  6.0.101 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

    private async Task onAfterResponse(object sender, SessionEventArgs e)
    {
        if (!e.HttpClient.Request.Url.Contains("getmsg"))
            return;
        var text = await e.GetResponseBodyAsString();
        var respJson = JsonNode.Parse(text);
        if(respJson == null || respJson["data"] == null)
            return;
        Console.WriteLine($"Data: {respJson["data"].ToJsonString()}");
    }

[honfika]

Please specify how to reproduce the problem. I checked it with some request, but I was not able to reproduce it.

[AxisRay]

like #547 when I switched from AfterResponse event to BeforeResponse event, the problem is gone.

I will try to find a way to reproduce it this weekend.

[AxisRay]

public class MoaProxy
{
    private readonly ProxyServer proxyServer;
    private ExplicitProxyEndPoint explicitEndPoint;
    public MoaProxy(IPAddress address, int port)
    {
        explicitEndPoint = new ExplicitProxyEndPoint(address, port);
        proxyServer = new ProxyServer(userTrustRootCertificate: false);
        proxyServer.ExceptionFunc = exception =>
        {
            Console.WriteLine(exception.Message + ": " + exception.InnerException?.Message);
        };
        proxyServer.BeforeResponse += onBeforeResponse;
        proxyServer.AfterResponse += onAfterResponse;
        proxyServer.ReuseSocket = false;
    }

    public void StartProxy()
    {
        proxyServer.AddEndPoint(explicitEndPoint);
        proxyServer.Start();
    }

    private async Task onBeforeResponse(object sender, SessionEventArgs e)
    {
        //everything will be fine , if you uncomment this
        //var bodyString = await e.GetResponseBodyAsString();
        //Console.WriteLine("onBeforeRequest" + bodyString);
    }

    private async Task onAfterResponse(object sender, SessionEventArgs e)
    {
        var bodyString = await e.GetResponseBodyAsString();
        Console.WriteLine("onAfterResponse" + bodyString);
    }
}

[AxisRay]

Seems that we cannot use GetResponseBodyAsString or GetResponseBody in AfterResponse. Beacause the connection has already been released.

                           // Otherwise it will keep authenticating per session.
                            if (EnableConnectionPool && connection != null
                                    && !connection.IsWinAuthenticated)
                            {
                                await tcpConnectionFactory.Release(connection);          //released!!!!
                                connection = null;
                            }
                        }
                        catch (Exception e) when (!(e is ProxyHttpException))
                        {
                            throw new ProxyHttpException("Error occured whilst handling session request", e, args);
                        }
                    }
                    catch (Exception e)
                    {
                        args.Exception = e;
                        closeServerConnection = true;
                        throw;
                    }
                    finally
                    {
                        Console.WriteLine(args.HttpClient.Response);
                        await onAfterResponse(args);
                        args.Dispose();

[honfika]

Yes, that is true. GetResponseBody* should be used in BeforeResponse event.

By default the response body is not preserved. Imagine, that there is a 2GB video or something like that. It would need a lot of memory. This is by design.

If you need the resposne body in after respose, set the KeepBody property to true in the before response event.

[honfika]

Anyway I'll try to reproduce the original problem. The header in the body string.... it sould not be there even in the AfterRsponse event.

[honfika]

I still can't reproduce the header problem. I get an exception when I call GetResponseBodyAsString in AfterResponse... which is expected.

Edit: I reproduced it. Sometimes happens... probably reads the next response header.

[honfika]

It should be fixed.. now you should receive an exception when you call the GetResponseBodyAsString in the AfterResponse event. (when there was a body, and it was not read in the beforerequest eventhandler)

[AxisRay]

Thanks! call GetResponseBody in AfterResponse is not safe. it will cause various exception.

Edit: I reproduced it. Sometimes happens... probably reads the next response header.

yeah, like the issue mentioned

[honfika]

Various exceptions? Could you please specify what type of exceptions?

[AxisRay]

yeah, various exceptions. But I think the reason is same.

For example: Exception thrown in user event: Invalid chunk length: 'Date: Sat, 22 Jan 2022 09:32:22 GMT' Error occured whilst handling session request: Invalid http status code. Titanium.Web.Proxy.Http.HeaderCollectionException thrown in user event: Data received after stream end Titanium.Web.Proxy.Http.HeaderCollectionError occured whilst handling session request: The ReadAsync method cannot be called when another read operation is pending.

System.ObjectDisposedException: Cannot access a disposed object.

[honfika]

But this is an internal exception. Do you receive any non expected exception as a user of the library? Can you still reproduce the wrong respinse body?

[honfika]

Also please include the various exception messages.

[AxisRay]

the app didn't work properly through the proxy, when I call GetResponseBody in AfterResponse . something was broken.

probably reads the next response header.

i think it's the reason. it breaked the http stream.

[honfika]

please put the GetResponseBodyAsString to a try-catch.

It is normal, that it throws an exception, since it is not allowed to call GetResponseBodyAsString in AfterResponse when you are not reading the body in BeforeResponse.

[AxisRay]

I see. it works well when i change to BeforeResponse. so it's time to close. thanks for your help