Stream from GetStreamAsync stops writing to buffer after short period of time

domneedham commented 6 months ago

Steps to Reproduce

Create a web request with httpClient.GetStreamAsync (I use Server Sent Events so this is designed to stay open as long as required)
Pass the stream into a stream reader and read line by line with ReadLineAsync
Observe after a few minutes, the ReadLineAsync hangs as nothing is getting written to the stream to read from.

Expected Behavior

The events received are written to the stream so they can be read.

Actual Behavior

The events received are written to the stream for a few minutes, then it stops.

Note: Reopening the stream can parse up to where it got to and beyond, for another few minutes, then repeat.

Environment

Version information

``` Visual Studio Community 2022 for Mac Version 17.6.11 (build 400) Installation UUID: 4bfe3bc1-7555-4e5e-8bb5-785daefa6edb Runtime .NET 7.0.3 (64-bit) Architecture: Arm64 Microsoft.macOS.Sdk 13.1.1007; git-rev-head:8afca776a0a96613dfb7200e0917bb57f9ed5583; git-branch:release/7.0.1xx-xcode14.2 Roslyn (Language Service) 4.6.0-3.23180.6+99e956e42697a6dd886d1e12478ea2b27cceacfa NuGet Version: 6.4.0.117 .NET SDK (Arm64) SDK: /usr/local/share/dotnet/sdk/8.0.204/Sdks SDK Versions: 8.0.204 8.0.201 8.0.101 7.0.315 7.0.314 MSBuild SDKs: /Applications/Visual Studio.app/Contents/MonoBundle/MSBuild/Current/bin/Sdks .NET Runtime (Arm64) Runtime: /usr/local/share/dotnet/dotnet Runtime Versions: 8.0.4 8.0.2 8.0.1 7.0.18 7.0.17 Xamarin.Profiler Version: 1.8.0.49 Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler Updater Version: 11 Apple Developer Tools Xcode: 15.3 22618 Build: 15E204a Xamarin.Mac Version: 9.3.0.23 Visual Studio Community Hash: 9defd91b3 Branch: xcode14.3 Build date: 2023-10-23 16:14:59-0400 Xamarin.iOS Version: 16.4.0.23 Visual Studio Community Hash: 9defd91b3 Branch: xcode14.3 Build date: 2023-10-23 16:15:00-0400 Xamarin.Android Version: 13.2.2.0 (Visual Studio Community) Commit: xamarin-android/d17-5/45b0e14 Android SDK: /Users/dom/Library/Developer/Xamarin/android-sdk-macosx Supported Android versions: 13.0 (API level 33) SDK Command-line Tools Version: 7.0 SDK Platform Tools Version: 34.0.3 SDK Build Tools Version: 32.0.0 Build Information: Mono: d9a6e87 Java.Interop: xamarin/java.interop/d17-5@149d70fe SQLite: xamarin/sqlite/3.40.1@68c69d8 Xamarin.Android Tools: xamarin/xamarin-android-tools/d17-5@ca1552d Microsoft Build of OpenJDK Java SDK: /Library/Java/JavaVirtualMachines/microsoft-11.jdk 11.0.20.1 Android Designer EPL code available here: https://github.com/xamarin/AndroidDesigner.EPL Eclipse Temurin JDK Java SDK: /Library/Java/JavaVirtualMachines/temurin-8.jdk 1.8.0.302 Android Designer EPL code available here: https://github.com/xamarin/AndroidDesigner.EPL Android SDK Manager Version: 17.6.0.50 Hash: a715dca Branch: HEAD Build date: 2024-04-05 09:48:29 UTC Android Device Manager Version: 0.0.0.1309 Hash: 06e3e77 Branch: HEAD Build date: 2024-04-05 09:48:29 UTC Xamarin Designer Version: 17.6.3.9 Hash: 2648399ae8 Branch: remotes/origin/d17-6 Build date: 2024-04-05 09:48:23 UTC Build Information Release ID: 1706110400 Git revision: 5d3d895c30788d39f3fb9931002e2961d22b60fd Build date: 2024-04-05 09:46:37+00 Build branch: release-17.6 Build lane: release-17.6 Operating System Mac OS X 14.4.1 Darwin 23.4.0 Darwin Kernel Version 23.4.0 Fri Mar 15 00:10:42 PDT 2024 root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Enabled user installed extensions .NET Core Extensions 0.7 ```

Build Logs

NGT.NGPlatform_Debug_Build_2024-04-19T09_40_32.5707610Z.msbuild.log.zip

Example Project (If Possible)

I can't share an example project, as I can't provide the SSE stream.

The code to get and read the stream is as follows:

private async Task<IAsyncEnumerable<ServerEventResponse>> StreamLoop(string url, CancellationToken cancellationToken, IEnumerable<KeyValuePair<string, string>> extraHeaders)
{
    HttpClient client = HttpHelper.CreateHttpClient(this.clientFactory, extraHeaders);

    using CancellationTokenSource delayCTS = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
    delayCTS.CancelAfter(TimeSpan.FromSeconds(10));

    try
    {
        Stream webStream = await client.GetStreamAsync(url, delayCTS.Token).ConfigureAwait(false);
        return Loop(webStream);
    }
    catch (Exception ex)
    {
        Log.Warning(ex, "Error");
        throw;
    }

    async IAsyncEnumerable<ServerEventResponse> Loop(Stream webStream)
    {
        StreamReader streamReader = new(webStream);
        while (!cancellationToken.IsCancellationRequested)
        {
            ServerEventResponse res = await this.ParseStream(streamReader, url, cancellationToken).ConfigureAwait(false);

            if (res.Exception != null)
            {
                throw res.Exception;
            }
            else
            {
                if (res.EventType != ServerEventType.NONE)
                {
                    yield return res;
                }
            }
        }

        streamReader.Dispose();
    }
}

private async Task<ServerEventResponse> ParseStream(StreamReader streamReader, string url, CancellationToken cancellationToken)
{
    ServerEventResponse streamResponse = new ServerEventResponse { StreamUrl = url };
    try
    {
        using CancellationTokenSource delayCTS = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);

        delayCTS.CancelAfter(TimeSpan.FromSeconds(10));

        string? text = await streamReader.ReadLineAsync(delayCTS.Token).ConfigureAwait(false);

        // can be an empty line from stream reader buffer.
        if (string.IsNullOrEmpty(text))
        {
            streamResponse.EventType = ServerEventType.NONE;
        }

        // data event starts with "data:" as a key, other event messages do not.
        if (text!.StartsWith("data:"))
        {
            // remove first "data:" from string
            text = text.Substring(5);

            // remove initial { and end }
            text = text.TrimStart('{').TrimEnd('}');

            Log.Verbose(text);
            streamResponse.OriginalData = text;

            Match dataText = DataRegex().Match(text);
            streamResponse.Message = dataText.Value;

            Match createdText = CreatedRegex().Match(text);
            streamResponse.Created = DateTime.Parse(createdText.Value);

            Match typeText = TypeRegex().Match(text);

            bool canParse = Enum.TryParse(typeText.Value, out ServerEventType result);
            if (canParse)
            {
                streamResponse.EventType = result;

                if (result == ServerEventType.HEARTBEAT)
                {
                    streamResponse.EventType = ServerEventType.NONE;
                }
            }
            else
            {
                streamResponse.EventType = ServerEventType.NONE;
            }
        }
        else
        {
            streamResponse.EventType = ServerEventType.NONE;
        }
    }
    catch (Exception ex) when (ex is OperationCanceledException || ex is ObjectDisposedException || ex is NotDataLineException)
    {
        streamResponse.EventType = ServerEventType.NONE;
    }
    catch (TimeoutException ex)
    {
        streamResponse.Exception = ex;
        Log.Warning(ex, "Timeout reading line in stream");
    }
    catch (Exception ex)
    {
        streamResponse.Exception = ex;
        streamResponse.EventType = ServerEventType.NONE;
        Log.Warning(ex, "Other exception");
    }

    return streamResponse;
}

Other useful information:

This works fine on Xamarin.Forms (for iOS and macOS)
This works better on macOS (on MAUI), but it does still timeout after around 10 mins, not 3 or so with iOS
It works fine with no issues on Android or WinUI (on MAUI)

I have tried reading from Task.Run and a Thread and the same issue occurs.

It produces the following exception on timeout:

System.TimeoutException: The request timed out.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.NSUrlSessionHandler.NSUrlSessionDataTaskStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) in /Users/builder/azdo/_work/1/s/xamarin-macios/src/Foundation/NSUrlSessionHandler.cs:line 1350
   --- End of inner exception stack trace ---
   at System.Net.Http.NSUrlSessionHandler.NSUrlSessionDataTaskStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) in /Users/builder/azdo/_work/1/s/xamarin-macios/src/Foundation/NSUrlSessionHandler.cs:line 1354
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)
   at NGT.NGPlatform.Services.ServerEventService.ParseStream(StreamReader streamReader, String url, CancellationToken cancellationToken) in /Users/dom/Code/work/RelayUKMAUI/NGT.NGPlatform/Services/ServerEventService.cs:line 168

rolfbjarne commented 6 months ago

Can you try adding this to your csproj, to see if it makes a difference?

<PropertyGroup>
    <UseNativeHttpHandler>false</UseNativeHttpHandler>
</PropertyGroup>

This will use .NET's built-in HttpClientHandler, instead of our NSUrlSessionHandler (which goes through Apple's networking APIs).

domneedham commented 6 months ago

UseNativeHttpHandler

I can confirm this has worked for iOS. I have ran the event stream for ~20 minutes, received 0 timeouts on iOS. I left macOS unchanged and received 4 in that time. Will apply the same change to macOS and report back.

Further update:

Made the change on macOS and no timeouts in another ~20 minute stream. Both iOS and macOS are therefore working with this change.

rolfbjarne commented 6 months ago

I tried to re-create this the scenario, but I wasn't able to make it fail.

This unfortunately means that we'll need a complete test project we can use to reproduce it in order to track it down, because something non-obvious is happening.

microsoft-github-policy-service[bot] commented 6 months ago

Hi @domneedham. We have added the "need-repro" label to this issue, which indicates that we require steps and sample code to reproduce the issue before we can take further action. Please try to create a minimal sample project/solution or code samples which reproduce the issue, ideally as a GitHub repo that we can clone. See more details about creating repros here: https://github.com/xamarin/xamarin-macios/blob/main/docs/bug-repro.md This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

microsoft-github-policy-service[bot] commented 6 months ago

Hi @domneedham. Due to inactivity, we will be closing this issue. Please feel free to re-open this issue if the issue persists. For enhanced visibility, if over 7 days have passed, please open a new issue and link this issue there. Thank you.

xamarin / xamarin-macios