Add new API for annoymous pipe and named fifo

[NobodyXu]

Proposal

Problem statement

Annoymous and named pipe are widely used, there is crate os_pipe which just provides an abstraction for pipe on unix and has 15 million download.

While having a third-party crate for it is enough for many crates, I think it'd be better if it's in stdlib so that it can be used without having to another dependency for this.

It would also enable better integration with the std::process API, since users might want to pipe output of multiple processes to one pipe and read them.

Motivating examples or use cases

jobserver-rs, for example, is used by cc-rs and pulled in as build-dependencies quite often.

It internally implements all kinds of API for annoymous pipe and named fifo, contains a bunch of unsafe code for this and quite some code for just managing the pipe/fifo.

It'd be great if we could move them to stdlib and make jobserver-rs easier to maintain. It might also speedup jobserver-rs compilation since it could've drop the libc dependency.

tokio, the widely used async executor, already provide pipe support in mod tokio::net::unix::pipe.

Solution sketch

I suppose we can use os_pipe as basis and then add in more functions used by jobserver-rs, and functions provided by rustix:

The basic API for annoymous pipe:

mod pipe { // Put under std::io
    // Create annoymous pipe that is close-on-exec and blocking.
    pub fn pipe() -> io::Result<(PipeReader, PipeWriter)> {
        PipeBuilder::new().build()
    }

    #[derive(Debug, Default, Clone)]
    pub struct PipeBuilder { ... }
    impl PipeBuilder {
        pub fn new() -> Self;

        /// Enable user to share this pipe across execve.
        ///
        /// NOTE that the fd itself is shared between process, not the file descriptor,
        /// so if you change its non-blocking mode, it would affect every process using it.
        pub fn cloexec(&mut self, cloexec: bool) -> &mut Self;

        pub fn build() -> io::Result<(PipeReader, PipeWriter)>;
    }

    #[derive(Debug)]
    pub struct PipeReader(/* private fields */);

    impl PipeReader {
        pub fn try_clone(&self) -> Result<Self>;
    }

    #[cfg(unix)]
    impl AsFd for PipeReader { ... }
    #[cfg(unix)]
    impl AsRawFd for PipeReader { ... }

    #[cfg(windows)]
    impl AsHandle for PipeReader { ... }
    #[cfg(windows)]
    impl AsRawHandle for PipeReader { ... }

    // Use TryFrom here, because not every owned fd is a valid pipe
    #[cfg(unix)]
    impl TryFrom<OwnedFd> for PipeReader { ... }
    #[cfg(windows)]
    impl TryFrom<OwnedHandle> for PipeReader { ... }

    #[cfg(unix)]
    impl From<PipeReader> for OwnedFd { ... }
    #[cfg(windows)]
    impl From<PipeReader> for OwnedHandle { ... }

    impl From<PipeReader> for Stdio { ... }

    #[cfg(unix)]
    impl FromRawFd for PipeReader { ... }
    #[cfg(unix)]
    impl IntoRawFd for PipeReader { ... }

    #[cfg(windows)]
    impl FromRawHandle for PipeReader { ... }
    #[cfg(windows)]
    impl IntoRawHandle for PipeReader { ... }

    impl<'a> Read for &'a PipeReader { ...}
    impl Read for PipeReader { ... }

    #[derive(Debug)]
    pub struct PipeWriter(/* private fields */);

    impl PipeWriter {
        pub fn try_clone(&self) -> Result<Self>;
    }

    #[cfg(unix)]
    impl AsFd for PipeWriter { ... }
    #[cfg(unix)]
    impl AsRawFd for PipeWriter { ... }

    #[cfg(windows)]
    impl AsHandle for PipeWriter { ... }
    #[cfg(windows)]
    impl AsRawHandle for PipeWriter { ... }

    // Use TryFrom here, because not every owned fd is a valid pipe
    #[cfg(unix)]
    impl TryFrom<OwnedFd> for PipeWriter { ... }
    #[cfg(windows)]
    impl TryFrom<OwnedHandle> for PipeWriter { ... }

    #[cfg(unix)]
    impl From<PipeWriter> for OwnedFd { ... }
    #[cfg(windows)]
    impl From<PipeWriter> for OwnedHandle { ... }

    impl From<PipeWriter> for Stdio { ... }

    #[cfg(unix)]
    impl FromRawFd for PipeWriter { ... }
    #[cfg(unix)]
    impl IntoRawFd for PipeWriter { ... }

    #[cfg(windows)]
    impl FromRawHandle for PipeWriter { ... }
    #[cfg(windows)]
    impl IntoRawHandle for PipeWriter { ... }

    impl<'a> Write for &'a PipeWriter { ...}
    impl Write for PipeWriter { ... }
}

The basic API for named fifo:

// Under std::fs
mod fifo {
    #[derive(Debug, Default, Clone)]
    pub struct FifoOpenOptions { ... }
    impl FifoOpenOptions {
        pub fn new() -> Self;

        pub fn create(&mut self, create: bool) -> &mut Self;
        pub fn create_new(&mut self, create_new: bool) -> &mut Self;

        pub fn write(&mut self, write: bool) -> &mut Self;
        pub fn read(&mut self, read: bool) -> &mut Self;

        pub fn open<P: AsRef<Path>>(&self, path: P) -> io::Result<Fifo>;
    }

    #[derive(Debug)]
    pub struct Fifo(/* private fields */);

    impl Fifo {
        pub fn try_clone(&self) -> Result<Self>;
    }

    #[cfg(unix)]
    impl AsFd for Fifo { ... }
    #[cfg(unix)]
    impl AsRawFd for Fifo { ... }

    #[cfg(windows)]
    impl AsHandle for Fifo { ... }
    #[cfg(windows)]
    impl AsRawHandle for Fifo { ... }

    // Use TryFrom here, because not every owned fd is a valid pipe
    #[cfg(unix)]
    impl TryFrom<OwnedFd> for Fifo { ... }
    #[cfg(windows)]
    impl TryFrom<OwnedHandle> for Fifo { ... }

    #[cfg(unix)]
    impl From<Fifo> for OwnedFd { ... }
    #[cfg(windows)]
    impl From<Fifo> for OwnedHandle { ... }

    impl From<Fifo> for Stdio { ... }

    #[cfg(unix)]
    impl FromRawFd for Fifo { ... }
    #[cfg(unix)]
    impl IntoRawFd for Fifo { ... }

    #[cfg(windows)]
    impl FromRawHandle for Fifo { ... }
    #[cfg(windows)]
    impl IntoRawHandle for Fifo { ... }

    impl<'a> Read for &'a Fifo { ...}
    impl Read for Fifo { ... }

    impl<'a> Write for &'a Fifo { ...}
    impl Write for Fifo { ... }
}

Extension methods for unix:

// Under std::os::unix
pub const PIPE_BUF: usize = c::PIPE_BUF; // 4_096usize

trait PipeBuildExt: Sealed {
    fn non_blocking(&mut self, non_blocking: bool) -> &mut Self;
}
impl PipeBuildExt for PipeBuild { ... }

trait PipeExt: Sealed {
    fn set_non_blocking(&mut self, non_blocking: bool) -> io::Result<()>;
}
impl PipeExt for PipeReader { ... }
impl PipeExt for PipeWriter { ... }
impl PipeExt for Fifo { ... }

trait FifoOpenOptionsExt: Sealed {
    fn non_blocking(&mut self, non_blocking: bool) -> &mut Self;

    fn mode(&mut self, mode: u32) -> &mut Self;
    fn custom_flags(&mut self, flags: i32) -> &mut Self;
}
impl FifoOpenOptionsExt for FifoOpenOptions { ... }

Extension methods for Linux:

// Under std::os::linux::pipe
#[derive(Debug, Clone, Copy)]
pub struct SpliceFlags { ... }
impl SpliceFlags {
    pub const MOVE: Self;
    pub const NONBLOCK: Self;
    pub const MORE: Self;
    pub const GIFT: Self;

    pub const fn combine(self, other: Self) -> Self;
}
impl BitOr for SpliceFlags { ... }
impl BitOrAssign for SpliceFlags { ... }

trait PipeReaderExt: Sealed {
    fn non_blocking_read(&self, bufs: &mut [io::IoSliceMut<'_>]) -> io::Result<()>;

    fn splice_to<FdOut: AsFd>(
        &self,
        fd_out: FdOut,
        off_out: Option<&mut u64>,
        len: usize,
        flags: SpliceFlags
    ) -> io::Result<usize>;

    /// convenient method for splice_to or splice_from, when both in/out
    /// are pipes/fifos.
    fn splice(
        &self,
        fd_out: &impl PipeWriterExt,
        len: usize,
        flags: SpliceFlags
    ) -> io::Result<usize>;

    fn tee(
        &self,
        fd_out: &impl PipeWriterExt,
        len: usize,
        flags: SpliceFlags
    ) -> io::Result<usize>;
}
impl PipeReaderExt for PipeReader { ... }
impl PipeReaderExt for Fifo { ... }

trait PipeWriterExt: Sealed {
    fn non_blocking_write(&self, bufs: &[io::IoSlice<'_>]) -> io::Result<()>;

    fn splice_from<FdIn: AsFd>(
        &self,
        fd_in: FdIn,
        off_in: Option<&mut u64>,
        len: usize,
        flags: SpliceFlags
    ) -> io::Result<usize>;
}
impl PipeWriterExt for PipeWriter { ... }
impl PipeWriterExt for Fifo { ... }

trait PipeExt: Sealed {
    pub fn get_pipe_size(&self) -> io::Result<usize>;
    pub fn set_pipe_size(&self, size: usize) -> io::Result<()>;
}
impl PipeExt for PipeWriter { ... }
impl PipeExt for PipeReader { ... }
impl PipeExt for Fifo { ... }

// Under std::os::linux::fs
trait FifoOpenOptionsExt: Sealed {
    fn open_at<P: AsRef<Path>>(&self, path: P, dirfd: BorrowedFd<'_>) -> io::Result<Fifo>;
}
impl FifoOpenOptionsExt for FifoOpenOptions { ... }

Alternatives

Alternatively, we could first implement a subset of the API I proposed here to reduce the scope. Then we could consider adding more methods that are needed.

Or we could just leave them up to third-party crates, which is the current status-quo, which is OK-ish but not good enough for users who need pipe/fifo, they would have to grab a third-party crate for this.

Links and related work

• mod tokio::net::unix::pipe
• os_pipe
• mod rustix::pipe

[NobodyXu]

Since the API surface I proposed is a bit large, I think it makes sense to implement them separately, under different feature flags.

[NobodyXu]

As per the discussion on zulip, I've updated the API to be platform-independent instead of unix-specific, since windows, WASI and most of the other OSes seem to have annoymous pipe and named fifo support.

[m-ou-se]

We discussed this in the libs-api meeting. As you said, the API surface is a bit large, and we think it can be split up. We're happy to accept a subset: the API for anonymous pipes, without the (Linux) extension methods. However, instead of the cloexec API, the close-on-exec flag should be set by default on creation (just like for any other file descriptor that the standard library opens). (Passing file descriptors to a child process should be the job of the Command API.)

The (Linux) extension methods and named pipes part of this ACP should be left for separate ACPs.

[NobodyXu]

Thanks, I've opened PR rust-lang/rust#127153 to add the anonymous pipe API

[NobodyXu]

Opened #416 for named fifo