Implements multiplexing for the gRPC broker so that additional gRPC connections are multiplexed onto the plugin's initial listener instead of creating a new Unix or TCP listener for each gRPC stream ID. The primary motivation is the simplification this gives containerised plugins which will only have to negotiate creating a single Unix socket that can be used by both sides, instead of one on each side - in particular this will greatly ease supporting rootless container engines running containers with a non-root user.
This will require both sides of the connection to update their go-plugin version, and as such is strictly opt-in via client config.
The implementation uses the same yamux.Session-based multiplexing that netrpc uses. However, given that we don't have control over when the gRPC library might redial a new connection, we introduce the concept of knocking (essentially a handshake, somewhat similar to TCP's SYN-ACK handshake) to synchronise the client and server on establishing a connection for a pre-agreed stream ID.
I briefly explored implementing gRPC's grpc.ClientConnInterface instead to give us full control over when a new connection is established, which would get rid of the need for knocking/pre-agreeing an ID, but I got put off by having to implement grpc.CallOptions, and also having to change the library's API, because GRPCBroker.Dial currently returns a concrete *grpc.ClientConn struct. However, I'm still open to exploring that further if people think those problems are preferable to the ones this PR currently solves.
tomhjp
commented
11 months ago
Implements multiplexing for the gRPC broker so that additional gRPC connections are multiplexed onto the plugin's initial listener instead of creating a new Unix or TCP listener for each gRPC stream ID. The primary motivation is the simplification this gives containerised plugins which will only have to negotiate creating a single Unix socket that can be used by both sides, instead of one on each side - in particular this will greatly ease supporting rootless container engines running containers with a non-root user.
This will require both sides of the connection to update their go-plugin version, and as such is strictly opt-in via client config.
The implementation uses the same yamux.Session-based multiplexing that netrpc uses. However, given that we don't have control over when the gRPC library might redial a new connection, we introduce the concept of knocking (essentially a handshake, somewhat similar to TCP's SYN-ACK handshake) to synchronise the client and server on establishing a connection for a pre-agreed stream ID.
I briefly explored implementing gRPC's
grpc.ClientConnInterfaceinstead to give us full control over when a new connection is established, which would get rid of the need for knocking/pre-agreeing an ID, but I got put off by having to implementgrpc.CallOptions, and also having to change the library's API, becauseGRPCBroker.Dialcurrently returns a concrete*grpc.ClientConnstruct. However, I'm still open to exploring that further if people think those problems are preferable to the ones this PR currently solves.