How to get a Multisegment stream backed ReadOnlySequence

[Wraith2]

I'm writing something to parse a utf8 based network protocol and I'm trying to setup multiple segments so I can ensure the parser handles the boundaries correctly without having to take the data off the wire each time. It may well be the case that I've misunderstood the pipelines api but I can't figure out how to get a multi segment sequence setup using a stream.

The code I'm using so far is this:

public static async Task TestParser()
{
    byte[] msg = new byte[] { 239, 187, 191, 76, 111, 103, 111, 110, 58, 58, 68, 73, 68, 35, 95, 123, 101, 97, 101, 102, 50, 50, 57, 102, 45, 52, 51, 56, 97, 45, 52, 52, 52, 52, 45, 97, 102, 55, 56, 45, 98, 56, 51, 101, 101, 49, 100, 52, 97, 56, 49, 101, 125, 126, 65, 112, 112, 78, 97, 109, 101, 35, 80, 68, 65, 126, 65, 112, 112, 86, 101, 114, 115, 105, 111, 110, 35, 49, 46, 51, 51, 126, 69, 110, 97, 98, 108, 101, 67, 111, 109, 112, 114, 101, 115, 115, 105, 111, 110, 35, 84, 114, 117, 101, 64, 64 };

    using (var memory = new MemoryStream())
    {
        int length = 5;
        var duplex = StreamConnection.GetDuplex(memory);
        for (int count = 0; count<msg.Length/length; count++)
        {
            duplex.Output.Write(msg.AsSpan(count*length, length));
        }
        if (msg.Length%length>0)
        {
            duplex.Output.Write(msg.AsSpan(msg.Length-(msg.Length%length)));
        }
        await duplex.Output.FlushAsync();

        duplex.Output.Complete();

        var reader = await duplex.Input.ReadAsync();

        ReadOnlySequence<byte> buffer = reader.Buffer; // always a zero length single segment, what am i getting wrong?
    }
}

[mgravell]

It sounds like what you actually want here is just a single Pipe. A duplex-pipe over a MemoryStream is a bad idea, as it won't rewind the stream. A single pipe naps to MemoryStream automatically: just use a single Pipe, via new Pipe, and access the reader/writer from there. No MemoryStream required.

On Thu, 2 Aug 2018, 08:37 Wraith2, notifications@github.com wrote:

I'm writing something to parse a utf8 based network protocol and I'm trying to setup multiple segments so I can ensure the parser handles the boundaries correctly without having to take the data off the wire each time. It may well be the case that I've misunderstood the pipelines api but I can't figure out how to get a multi segment sequence setup using a stream.

The code I'm using so far is this:

public static async Task TestParser() { byte[] msg = new byte[] { 239, 187, 191, 76, 111, 103, 111, 110, 58, 58, 68, 73, 68, 35, 95, 123, 101, 97, 101, 102, 50, 50, 57, 102, 45, 52, 51, 56, 97, 45, 52, 52, 52, 52, 45, 97, 102, 55, 56, 45, 98, 56, 51, 101, 101, 49, 100, 52, 97, 56, 49, 101, 125, 126, 65, 112, 112, 78, 97, 109, 101, 35, 80, 68, 65, 126, 65, 112, 112, 86, 101, 114, 115, 105, 111, 110, 35, 49, 46, 51, 51, 126, 69, 110, 97, 98, 108, 101, 67, 111, 109, 112, 114, 101, 115, 115, 105, 111, 110, 35, 84, 114, 117, 101, 64, 64 };

using (var memory = new MemoryStream()) { int length = 5; var duplex = StreamConnection.GetDuplex(memory); for (int count = 0; count<msg.Length/length; count++) { duplex.Output.Write(msg.AsSpan(count*length, length)); } if (msg.Length%length>0) { duplex.Output.Write(msg.AsSpan(msg.Length-(msg.Length%length))); } await duplex.Output.FlushAsync();

  duplex.Output.Complete();

  var reader = await duplex.Input.ReadAsync();

  ReadOnlySequence<byte> buffer = reader.Buffer; // always a zero length single segment, what am i getting wrong?

} }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mgravell/Pipelines.Sockets.Unofficial/issues/7, or mute the thread https://github.com/notifications/unsubscribe-auth/AABDsFBZhaGFEgZxSY5ZOyWk3kK0kKFqks5uMqwxgaJpZM4Vrx-a .

[Wraith2]

Using a Pipe directly works, though the change from Input/Output to Reader/Writer feels a bit strange. It looks like passing in a new PipeOptions(minimumSegmentSize: 1) gets me multiple segments. Not of the size I requested but of 16 bytes each, presumably a minimum size constraint somewhere which means I can't test tiny odd segments but it seems a sensible restriction for the perf of larger data.

[mgravell]

If you want multiple segments, you're going to have to play with the PipeOptions when creating the pipe, and/or write more data (meaning: go over the segment boundary)

On Thu, 2 Aug 2018, 10:02 Wraith2, notifications@github.com wrote:

That gets me the data but as a single segment and what I'm looking for is a way to test the segment handover so I really want to try and get multiple segments. Presumably whatever I do with MemoryStream as backing is going to end up with the single buffer from the stream exposed as a single segment so I need another way.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mgravell/Pipelines.Sockets.Unofficial/issues/7#issuecomment-409858058, or mute the thread https://github.com/notifications/unsubscribe-auth/AABDsGygGIM9WIJFy2Z7RXMLON-Komwjks5uMsAqgaJpZM4Vrx-a .

[Wraith2]

Yup, got that just after I posted my earlier reply. I edited to reflect that but the email won't have had that in. Looks like it's working and I can bias by odd bytes interleaving utf preambles of 3 bytes if I need to. Thanks for the help.

[Drawaes]

For testing multi segment handling I wrote my own segment implementation and. Factory that takes in multiple arrays and returns Readonlysequence. That way your unit test focuses on parsing not pipes

[Wraith2]

That sounds like a useful bit of code to have. Any chance of getting that integrated into this library as a tool?

[Drawaes]

If @mgravell is interested in it I can take a look at making it public sure.

[mgravell]

Yes, I think that would be a useful addition. Your code, though; your choice!

On Thu, 2 Aug 2018, 12:58 Tim Seaward, notifications@github.com wrote:

If @mgravell https://github.com/mgravell is interested in it I can take a look at making it public sure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mgravell/Pipelines.Sockets.Unofficial/issues/7#issuecomment-409901731, or mute the thread https://github.com/notifications/unsubscribe-auth/AABDsKHp7M-f4VsFJe0S8rqnsDuyK1PWks5uMulwgaJpZM4Vrx-a .

[mgravell]

re the original MemoryStream usage; from 0.2.1-alpha.79 the lib detects this and issues guidance