feature proposal: Deno builtin function & ops to create an anonymous pipe.

[PaulThompson]

I propose idea of an additional builtin function Deno.pipe to open a new anonymous pipe. The call result ultimately would provide two resource ids which could be used to assemble process pipelines.

The key benefit compared to alternatives:

• Compared to using Deno.run with stdout and stdin using the piped option & using Deno.copy to copy between one stdout and the next stdin: Using direct "anonymous" pipes between child processes means that the data would not be handled in the deno process. It therefore would use less CPU.
• Compared to making a pipe on the filesystem using mkfifo and opening the fifo for read and write using Deno.read and Deno.write: Using an anonymous pipe avoids having to expose the pipe to the filesystem (cleanup issues, security issues?, inconvenient), avoids platform dependent hacks.
• Compared to using Deno.run({cmd:'bash','-c','process1 | process2'}): Using an anonymous pipe in the deno process avoids the overhead and inconvenience of a shell subprocess.

Use case sketch:

// prepare a pipe: (proposed new function).
const anonpipe = await Deno.pipe();

// start a process (esp if it will produce a lot of stdout output)
const producer = Deno.run({cmd:['producer'], stdout: anonpipe[0].rid});

// start next process in a pipeline of processes:
const consumer = Deno.run({cmd:['consumer'], stdin: anonpipe[1].rid});

// await processes status' and close all etc

The proposed function could have types as follows:

function Deno.pipe(): Promise<[AnonPipe,AnonPipe]>;

/// (one end of) an anonymous pipe
class Deno.AnonPipe {
// properties and methods similar to Deno.File but omitting seek & seekSync
readonly rid: number;
write(p: Uint8Array): Promise<number>;
writeSync(p: Uint8Array): number;
read(p: Uint8Array): Promise<number | null>;
readSync(p: Uint8Array): number | null;
close(): void;
};

The implementation in rust would be to call one of the platform depdendent functions for create an anonymous pipe eg https://github.com/rust-lang/rust/blob/f4e4485a052857e5dd32ea29ceb7b1a8223e83cc/library/std/src/sys/unix/pipe.rs#L12

fn anon_pipe() -> io::Result<(AnonPipe, AnonPipe)>

or https://github.com/rust-lang/rust/blob/f4e4485a052857e5dd32ea29ceb7b1a8223e83cc/library/std/src/sys/windows/pipe.rs#L48 etc

Note:

• The existing piped io options on Deno.run already internally create anonymous pipes for reading/writing stdio. (In tokio::process::Command and std::process::Command). In order to pipe between two processes it needs an API to create the anonymous pipes separately from the processes.
• Assuming the existing Deno.run option for stdio to & from a resource id number works.

[solson]

I think this would solve the problem in https://github.com/denoland/deno/issues/2494 — using OS-level pipes between the child processes (with no copying through the parent Deno process) would work exactly like shell piping, and when the writing process quits and closes its stdout, the reading process will detect this on its stdin the usual way.