Question: Named pipes: await JsonSerializer.DeserializeAsync<T> blocks, when reading from Named pipe

[Legends]

I have created a named pipe server (console app, .NET 7):

using var server = new NamedPipeServerStream(
                                    "hostpipe",
                                    PipeDirection.InOut,
                                    NamedPipeServerStream.MaxAllowedServerInstances,
                                    PipeTransmissionMode.Message,
                                    PipeOptions.Asynchronous);

that spawns a child process (client console app, .NET 7) which creates a NamedPipeClientStream :

using var client = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut, PipeOptions.Asynchronous);

Problem: For example I want to message an object from the server to the client like:

Server writes:

public static async Task WriteMessageAsyncAsObject<T>(PipeStream pipe, T obj)
{
    await JsonSerializer.SerializeAsync<T>(pipe, obj); // works!
 }

and the client should read the incoming "object" (following code does not work, DeserializeAsync is blocking):

public static async Task<T?> ReadMessageAsyncAsObject<T>(PipeStream pipe)
{
    // var bt = await ReadMessageAsync(pipe);
    // var obj = JsonSerializer.Deserialize<T>(bt);
    // return await Task.FromResult(obj);
    return await JsonSerializer.DeserializeAsync<T>(pipe);  // doesn't work, the program stops here, it is blocked!
}

But if I replace JsonSerializer.DeserializeAsync<T> on the clientside with the following (this code here works, does not block):

public static async Task<T?> ReadMessageAsyncAsObject<T>(PipeStream pipe)
{
    var bt = await ReadMessageAsync(pipe);  // reads the stream result asynchronously into a MemoryStream, which returns a byte[]
    var obj = JsonSerializer.Deserialize<T>(bt);  // works now, but synchronous!
    return await Task.FromResult(obj);
    //return await JsonSerializer.DeserializeAsync<T>(pipe);  
}

Why is await JsonSerializer.DeserializeAsync<T> blocking on the clientside?

[ghost]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have created a named pipe server (console app, .NET 7): ``` using var server = new NamedPipeServerStream( "hostpipe", PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous); ``` that spawns a child process which creates a `NamedPipeClientStream? : `using var client = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut, PipeOptions.Asynchronous);` **Problem:** For example I want to message an object from the server to the client like: Server writes: public static async Task WriteMessageAsyncAsObject(PipeStream pipe, T obj) { await JsonSerializer.SerializeAsync(pipe, obj); // works! } and the client should read: public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { // var bt = await ReadMessageAsync(pipe); // var obj = JsonSerializer.Deserialize(bt); // return await Task.FromResult(obj); return await JsonSerializer.DeserializeAsync(pipe); // doesn't work, the program stops here, it is blocked! } But if I replace `JsonSerializer.DeserializeAsync` on the clientside with the following: ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { var bt = await ReadMessageAsync(pipe); // reads the stream result asynchronously into a MemoryStream, which returns a byte[] var obj = JsonSerializer.Deserialize(bt); // works now, but synchronous! return await Task.FromResult(obj); //return await JsonSerializer.DeserializeAsync(pipe); } ``` **Why is `JsonSerializer.DeserializeAsync` blocking on the clientside?**

Author:	Legends
Assignees:	-
Labels:	`area-System.Net.Http`, `untriaged`
Milestone:	-

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have created a named pipe server (console app, .NET 7): ``` using var server = new NamedPipeServerStream( "hostpipe", PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous); ``` that spawns a child process (client console app, .NET 7) which creates a `NamedPipeClientStream` : ``` using var client = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut, PipeOptions.Asynchronous);``` **Problem:** For example I want to message an object from the server to the client like: Server writes: ``` public static async Task WriteMessageAsyncAsObject(PipeStream pipe, T obj) { await JsonSerializer.SerializeAsync(pipe, obj); // works! } ``` and the client should read the incoming "object" (following code does not work, DeserializeAsync is blocking): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { // var bt = await ReadMessageAsync(pipe); // var obj = JsonSerializer.Deserialize(bt); // return await Task.FromResult(obj); return await JsonSerializer.DeserializeAsync(pipe); // doesn't work, the program stops here, it is blocked! } ``` But if I replace `JsonSerializer.DeserializeAsync` on the clientside with the following (this code here works, does not block): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { var bt = await ReadMessageAsync(pipe); // reads the stream result asynchronously into a MemoryStream, which returns a byte[] var obj = JsonSerializer.Deserialize(bt); // works now, but synchronous! return await Task.FromResult(obj); //return await JsonSerializer.DeserializeAsync(pipe); } ``` Why is `await JsonSerializer.DeserializeAsync` blocking on the clientside?

Author:	Legends
Assignees:	-
Labels:	`area-System.IO`, `untriaged`
Milestone:	-

[wfurt]

I don't think this is IO problem since ReadMessageAsync works fine according to description. This is similar to https://github.com/dotnet/runtime/issues/73097 and we end up fixing it in HTTP.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have created a named pipe server (console app, .NET 7): ``` using var server = new NamedPipeServerStream( "hostpipe", PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous); ``` that spawns a child process (client console app, .NET 7) which creates a `NamedPipeClientStream` : ``` using var client = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut, PipeOptions.Asynchronous);``` **Problem:** For example I want to message an object from the server to the client like: Server writes: ``` public static async Task WriteMessageAsyncAsObject(PipeStream pipe, T obj) { await JsonSerializer.SerializeAsync(pipe, obj); // works! } ``` and the client should read the incoming "object" (following code does not work, DeserializeAsync is blocking): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { // var bt = await ReadMessageAsync(pipe); // var obj = JsonSerializer.Deserialize(bt); // return await Task.FromResult(obj); return await JsonSerializer.DeserializeAsync(pipe); // doesn't work, the program stops here, it is blocked! } ``` But if I replace `JsonSerializer.DeserializeAsync` on the clientside with the following (this code here works, does not block): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { var bt = await ReadMessageAsync(pipe); // reads the stream result asynchronously into a MemoryStream, which returns a byte[] var obj = JsonSerializer.Deserialize(bt); // works now, but synchronous! return await Task.FromResult(obj); //return await JsonSerializer.DeserializeAsync(pipe); } ``` Why is `await JsonSerializer.DeserializeAsync` blocking on the clientside?

Author:	Legends
Assignees:	-
Labels:	`area-System.Text.Json`, `untriaged`
Milestone:	-

[Jozkee]

I wasn't able to repro this. Perhaps, there's something special about the type you are serializing.

Program 1:

using System.Diagnostics;
using System.IO.Pipes;
using System.Text.Json;

namespace ConsoleApp4
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("Hello, World!");

            using var server = new NamedPipeServerStream(
                                    "hostpipe",
                                    PipeDirection.InOut,
                                    NamedPipeServerStream.MaxAllowedServerInstances,
                                    PipeTransmissionMode.Message,
                                    PipeOptions.Asynchronous);

            using Process p = Process.Start(@"C:\consoleapps\ConsoleApp5\ConsoleApp5\bin\Debug\net7.0\ConsoleApp5.exe");
            server.WaitForConnection();

            var person = new Person
            {
                Age = 42,
                Name = "Foo",
            };
            await WriteMessageAsyncAsObject(server, person);
        }

        public static async Task WriteMessageAsyncAsObject<T>(PipeStream pipe, T obj)
        {
            await JsonSerializer.SerializeAsync<T>(pipe, obj); // works!
        }
    }

    internal class Person
    {
        public int Age { get; set; }
        public string Name { get; set; }
    }
}

Program2:

using System.Diagnostics;
using System.IO.Pipes;
using System.Text.Json;

namespace ConsoleApp5
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("Hello, World!");

            using var client = new NamedPipeClientStream(".", "hostpipe", PipeDirection.InOut, PipeOptions.Asynchronous);
            client.Connect();

            var r = await ReadMessageAsyncAsObject<Person>(client);

            Console.WriteLine(r.Age);
            Console.WriteLine(r.Name);
        }

        public static async Task<T?> ReadMessageAsyncAsObject<T>(PipeStream pipe)
        {
            // var bt = await ReadMessageAsync(pipe);
            // var obj = JsonSerializer.Deserialize<T>(bt);
            // return await Task.FromResult(obj);
            return await JsonSerializer.DeserializeAsync<T>(pipe);  // doesn't work, the program stops here, it is blocked!
        }
    }

    internal class Person
    {
        public int Age { get; set; }
        public string Name { get; set; }
    }
}

Output:

Hello, World!
Hello, World!
42
Foo

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[wfurt]

It may be the size @Jozkee. If you are on it, I would try something that would perhaps fill the pipe buffers - maybe 100k ish.

[Jozkee]

@wfurt gave it a quick try with new NamedPipeServerStream(..., inBufferSize: 1, outBufferSize: 1), no error. Also tried serializing a new Person { Age = 42, Name = new string('a', 100_000) }, still no error :/.

Would be nice to hear back from OP and see if he/she can provide a repro.

[Legends]

I have created a reproduction repo here.

At first glance I can only see the following differences between your code and the repro:

• On the client I additionally have: client.ReadMode = PipeTransmissionMode.Message;
• and async connect waits: => ConnectAsync / WaitForConnectionAsync.

[eiriktsarpalis]

I can reproduce the issue, the application hangs when the serializer tries to read the underlying stream for a second time here:

https://github.com/dotnet/runtime/blob/f4ad730ac2c3f06fe68a3041c21590ef9de1b8c2/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/ReadBufferState.cs#L51-L57

The underlying implementation in Windows is this method:

https://github.com/dotnet/runtime/blob/f4ad730ac2c3f06fe68a3041c21590ef9de1b8c2/src/libraries/System.IO.Pipes/src/System/IO/Pipes/PipeStream.Windows.cs#L314-L366

Without looking much into the code, I suspect there might be an issue with the ReadWriteValueTaskSource implementation that prevents the STJ callback from being scheduled appropriately.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have created a named pipe server (console app, .NET 7): ``` using var server = new NamedPipeServerStream( "hostpipe", PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous); ``` that spawns a child process (client console app, .NET 7) which creates a `NamedPipeClientStream` : ``` using var client = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut, PipeOptions.Asynchronous);``` **Problem:** For example I want to message an object from the server to the client like: Server writes: ``` public static async Task WriteMessageAsyncAsObject(PipeStream pipe, T obj) { await JsonSerializer.SerializeAsync(pipe, obj); // works! } ``` and the client should read the incoming "object" (following code does not work, DeserializeAsync is blocking): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { // var bt = await ReadMessageAsync(pipe); // var obj = JsonSerializer.Deserialize(bt); // return await Task.FromResult(obj); return await JsonSerializer.DeserializeAsync(pipe); // doesn't work, the program stops here, it is blocked! } ``` But if I replace `JsonSerializer.DeserializeAsync` on the clientside with the following (this code here works, does not block): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { var bt = await ReadMessageAsync(pipe); // reads the stream result asynchronously into a MemoryStream, which returns a byte[] var obj = JsonSerializer.Deserialize(bt); // works now, but synchronous! return await Task.FromResult(obj); //return await JsonSerializer.DeserializeAsync(pipe); } ``` Why is `await JsonSerializer.DeserializeAsync` blocking on the clientside?

Author:	Legends
Assignees:	-
Labels:	`area-System.IO`, `needs-further-triage`
Milestone:	-

[stephentoub]

@eiriktsarpalis, I'm not sure why this would be considered a System.IO.Pipes problem. The JsonSerializer.DeserializeAsync enters this loop: https://github.com/dotnet/runtime/blob/ac7afb9ccb88b895eeb3264e38ab22d0c5d726ec/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/ReadBufferState.cs#L49-L67 It reads the 50 bytes the server sent (which is the entire message sent by the server) and checks the while loop condition. The fillBuffer argument is defaulting to true, so since it only read 50 bytes and was given a 16384 byte buffer, it loops around again to perform another read, which hangs, because there's nothing more to read (but it's not EOF because the Stream is still open for continued duplex communication).

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have created a named pipe server (console app, .NET 7): ``` using var server = new NamedPipeServerStream( "hostpipe", PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous); ``` that spawns a child process (client console app, .NET 7) which creates a `NamedPipeClientStream` : ``` using var client = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut, PipeOptions.Asynchronous);``` **Problem:** For example I want to message an object from the server to the client like: Server writes: ``` public static async Task WriteMessageAsyncAsObject(PipeStream pipe, T obj) { await JsonSerializer.SerializeAsync(pipe, obj); // works! } ``` and the client should read the incoming "object" (following code does not work, DeserializeAsync is blocking): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { // var bt = await ReadMessageAsync(pipe); // var obj = JsonSerializer.Deserialize(bt); // return await Task.FromResult(obj); return await JsonSerializer.DeserializeAsync(pipe); // doesn't work, the program stops here, it is blocked! } ``` But if I replace `JsonSerializer.DeserializeAsync` on the clientside with the following (this code here works, does not block): ``` public static async Task ReadMessageAsyncAsObject(PipeStream pipe) { var bt = await ReadMessageAsync(pipe); // reads the stream result asynchronously into a MemoryStream, which returns a byte[] var obj = JsonSerializer.Deserialize(bt); // works now, but synchronous! return await Task.FromResult(obj); //return await JsonSerializer.DeserializeAsync(pipe); } ``` Why is `await JsonSerializer.DeserializeAsync` blocking on the clientside?

Author:	Legends
Assignees:	-
Labels:	`area-System.Text.Json`, `needs-further-triage`
Milestone:	-

[eiriktsarpalis]

Ah yes, the async deserializer generally assumes that the stream only contains a single JSON document and will always try to fill its internal buffer up to capacity or EOF before it starts deserializing. This is due to how async converters are designed and was meant to improve deserialization performance.

Even though the internal buffer size can be controlled via the JsonSerializerOptions.DefaultBufferSize property, setting this to a smaller number would still make deserialization susceptible to the same hangs under certain circumstances. Deserialization could also fail if it detects trailing data after the first complete JSON document.

TL;DR the DeserializeAsync methods have not been designed to read from streams that represent channels and which could contain multiple values. The DeserializeAsyncEnumerable method does support this (and does not impose the same "fillBuffer" semantics) although it imposes certain constraints (the stream must be one JSON array) and it would make your application much less straightforward to implement. Alternatively you might want to consider filling a buffer manually and passing that to one of the sync Deserialize methods, although some work

In the future it might be possible to support your scenario in the regular DeserializeAsync methods by adding:

1. A JsonSerializerOptions flag that disables "fillBuffer" semantics.
2. A JsonSerializerOptions flag that disables failures if the stream contains trailing data.

Related to #36750, #33030.