Dart should support a cross-platform IPC system e.g. named pipes on Windows, Unix domain sockets on POSIX

[insinfo]

add named Pipe support in dart io

Pipes (inter-process communications) A pipe is a section of shared memory that handles communication usage. The process that creates a pipe is the pipe server. A process that connects to a pipe is a pipe client. One process writes the information to the pipe and then the other process reads the information from the pipe.

https://docs.microsoft.com/pt-br/dotnet/api/system.io.pipes.namedpipeserverstream?view=net-5.0 https://stackoverflow.com/questions/26561604/create-named-pipe-c-windows https://stackoverflow.com/questions/31513202/ipc-using-of-named-pipes-in-c-between-two-programs

[robert-ancell]

I am also interested in traditional unnamed pipes as well. Not sure if that should be tracked as a separate issue or not.

[cbracken]

/cc @a-siva for thoughts on the VM side.

[robert-ancell]

The use of pipes came up while implementing portals. They use D-Bus and some APIs use pipes to send data across that they either don't want exposed to anyone able to listen on D-Bus (i.e. secrets) or is large and want to avoid the overhead.

[robert-ancell]

I had a quick look at what sort of Dart API would make sense for this, I figured you would have something like this:

class Pipe {
  PipeEndpoint readEnd;
  PipeEndpoint writeEnd;
}
var pipe = Pipe();
await pipe.writeEnd.writeString('Hello world!');
var handle = ResourceHandle.fromPipe(pipe.readEnd);
// Send the handle somewhere, e.g. via D-Bus

Open questions were:

• Should PipeEndpoint just be RandomAccessFile - that would work practically and require less code to be written, but not sure if that matches the model that Dart wants to expose.
• Should there be PipeReadEndpoint and PipeWriteEndpoint - you'll just get errors if you try and write to the read end and vice versa.

[brianquinlan]

I think that we can break this up into a few use cases:

1. ability to create a new named pipe

We add new methods to File like:

• Future<void> createPipe({bool recursive = false})
• void createPipeSync({bool recursive = false})

These would be implemented using CreateNamedPipe on Windows and mkfifo everywhere else. For Windows, we'd only support the ability to create pipes in byte stream mode.

2. ability to create an new anonymous pipe

We add a new function to dart:io like:

class abstract PipePair {
  ResourceHandle get readPipe;
  ResourceHandle get writePipe;
}

Future<PipePair> createAnonymousPipe();

The idea is to return the pipes as ResourceHandle because at least one pipe will almost certainly be transmitted to another process for use.

createAnonymousPipe would be implemented using CreatePipe on Windows and pipe everywhere else.

3. ability to use (read/write) to an existing named pipe

This already works e.g.

f = await File('pipe name').open();
// Read/write as normal

4. ability to use (read/write) to an existing anonymous pipe

This would work using existing APIs - given a ResourceHandle from PipePair.readPipe or PipePair.writePipe, call the toFile method to get a usable file.

For example:

pipes = await createAnonymousPipe()
writePipe = pipes.writePipe.toFile();
socket.sendMessage([SocketControlMessage.fromHandles([pipes.readPipe]], ...);

Alternatives

The named pipe use case could be implemented in a non-core package using FFI e.g.

import 'dart:ffi';
import 'package:ffi/ffi.dart';

final lib = DynamicLibrary.process();
typedef MkFifoNative = Void Function(Pointer<Utf8> pathname, Int mode);
typedef MkFifo = void Function(Pointer<Utf8> pathname, int mode);

void main() {
  final MkFifo mkFifo =
      lib.lookup<NativeFunction<MkFifoNative>>('mkfifo').asFunction();
  mkFifo("/tmp/foo-fifo".toNativeUtf8(), 0);
}

The anonymous pipe use case could be handled similarly if we were willing to provide a way to get a file descriptor (int) from a file and to construct a ResourceHandle from an int.

Any thoughts?

[brianquinlan]

Do we have a use case for implementing socket servers on Windows i.e. do we need to support WaitNamedPipeA?

[cbracken]

@brianquinlan I'm not aware of any users of the Flutter side currently pushing for this on Windows.

[brianquinlan]

I actually started to look at the named pipe implementation (https://dart-review.googlesource.com/c/sdk/+/257282/) that uses mkfifo.

Unfortunately, there is a large semantic difference between named pipes on POSIX vs Windows because named pipes are disposed on Windows when the last reference disappears (see https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createnamedpipea).

@insinfo Could you explain your use case in detail?

[brianquinlan]

I fixed a separate issue for anonymous pipe support: #49917 - @robert-ancell please track your feature request there.

[insinfo]

@insinfo

Sorry for the delay in responding but I've been very busy. My use case is basically the communication between two processes, one process is the graphical interface of a backup application I made in Flutter, and the other process is the backup engine that performs the backup of a Linux server through SFTP.

[brianquinlan]

Hey @insinfo

Were you planning on using this approach on Windows? If so, did you consider using package:win32?

My concern with adding this feature is that it seems like named pipe semantics are very platform-specific and, if you want to use this on Windows, there might already be a solution.

[insinfo]

the backup app is cross platform, it is for MacOs, Windows and Linux

[insinfo]

the ideal is to have a multiplatform solution

[brianquinlan]

I think that Unix FIFO's and Windows named pipes are too semantically different to be used as a common IPC mechanism (in particular, Unix lacks an idea like ConnectNamedPipe).

Some applications (e.g. Docker) use Unix Domain Sockets on Unix operating systems and named pipes on Windows.

NodeJS also uses this sockets/named-pipes approach for their IPC server application.

I'm not sure if we want that in the core dart libraries though.

[jibbers42]

I haven't looked in-depth, but it looks like .net wrote "named pipes" on Linux in terms of Unix Domain Sockets (https://github.com/dotnet/runtime/blob/main/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs#L325).

Just mentioning in case that's a possibility here.

[brianquinlan]

I'm going to retitle this issue and unassign it - I don't think that we are looking into a cross-platform IPC mechanism at the moment.