using webstreams on Net.Socket

seriousme commented 3 weeks ago

Node.js Version

v22.10.0

NPM Version

v10.9.0

Operating System

ChromeOS (Linux 6.6.46)

Subsystem

stream

Description

I wrote a MQTT server on Deno using webstreams with Bring Your Own Buffer (BYOB) so that I can just read buffsize bytes from the Deno socket. (Deno.conn) with zero copy of data e.g.

async function readFull(conn: Deno.conn
, buf: Uint8Array): Promise<void> {
  let bytesRead = 0;
  while (bytesRead < buf.length) {
    const read = await conn.read(buf.subarray(bytesRead));
    bytesRead += read;
  }
}

I'm trying to port this to NodeJS but I can not use the: const read = await conn.read(buf.subarray(bytesRead)); on a NodeJS Net.Socket because the ReadStream of the DuplexStream is not a WebStream.

What is the proper way to achieve the same result on NodeJS, preferably also with zero copy of data ?

Kind regards, Hans

Minimal Reproduction

No response

Output

No response

Before You Submit

[X] I have looked for issues that already exist before submitting this
[X] My issue follows the guidelines in the README file, and follows the 'How to ask a good question' guide at https://stackoverflow.com/help/how-to-ask

RedYetiDev commented 3 weeks ago

Can you please provide a Node.js sample snippet, not a Deno one?

seriousme commented 2 weeks ago

I don't have a working Nodejs sample snippet yet , hence the question.

But what I would like to be able to do is:

async function readFull(conn: Socket
, buf: Uint8Array): Promise<void> {
  let bytesRead = 0;
  while (bytesRead < buf.length) {
    const read = await conn.read(buf.subarray(bytesRead));
    bytesRead += read;
  }
}

Where conn:Socket is a node:net.Socket object. To generalize it a bit:

I would like to be able to create a webstandards Readable stream from a NodeJS Readable stream without having to manage the flow between them myself. (and preferably with zero copy of data).

Alternatively I would like my readFull() function to use an efficient mechanism (little overhead of copying) to read its data directly from a socket (without using the webstandards stream readable.read(buffer).

Hope this clarifies it a bit.

Any hints tips are much appreciated! Hans

seriousme commented 2 weeks ago

Ok, I think I found what I'm looking for:

stream.Duplex.toWeb(streamDuplex)

Seems to allow me to create a webstandards ReadableStream/WritableStream from a DuplexStream and since Socket is a DuplexStream this should work.

Would be nice if (over time) the Net library would offer a choice between classical streams and webstreams E.g. net.CreateServer({webStream:true}, connectionListener)

And personally I also like Deno's:

for await (const conn of Deno.listen(options)) {
      connectionListener(conn);
    }

as it seems to be more intuitive to me than:

const server = net.createServer(options, connectionListener);
server.listen(port);

Let's say that one can dream ;-) For now I think I solved my problem in an elegant way.

Thanks for any effort. Hans

nodejs / help