Readonly streams/partitions

This PR is a feature request and an API proposal to setting a readonly flag to streams and partitions.

Use-case

Our use-case involves the use of Liftbridge as part of the message processing pipeline in a future evolution of the Ably service. We currently implement elasticity of the service by allowing streams to be relocated - that is, the placement of a responsibility for a stream within an elastic cluster can move from time to time. We want to be able to implement a single, continuous, virtual stream, from multiple individual Liftbridge streams that each represent an epoch and are active at a given location for a period of time.

In order to be able to synchronise reliably from the ending of one epoch and the start of another, we would like to be able to put a stream into a read-only state, whereby subscribers can read from the stream, but no new messages can be published. Putting a stream into the read-only state needs to be idempotent, and a subscriber who reads to the end of a read-only stream is notified with an EOF-like error instead of blocking indefinitely.

This PR is a draft proposal for an API to set this state on a stream/partitions and also a possible implementation.

Further features

Other features we would like to have:

having paused and readonly statuses to the PartitionMetadata returned by a FetchMetaData call ;
expose the Raft commit index (or a timestamp) in FetchMetaData and in the activity stream, so that the client can ensure the validity of the data it is receiving ;
add a timestamp to publication ACK's.

Current issues/limitations

While implementing this feature we have encountered some issues that we have not been able to fix yet, but we would welcome any advice on this feature request nevertheless and/or suggestions for fixes.

no ErrEndOfReadonlyPartition error is sent when subscribing to a readonly partition that does not contain any message ;
there is no unit test to check that subscribers that have subscribed before a partition is made readonly receive the ErrEndOfReadonlyPartition error, but local testing has shown that this feature is working ;
in order to prevent existing subscribers that are already blocking (waiting for new data to arrive) to wait infinitely when a stream is set as read-only, some relatively extensive changes were needed (see the noBlocking channel). There might be a better solution to this.

Proposal

New RPCs to the API service:

// SetStreamReadonly sets a read-only flag to a partition. The latest
// message's offset is returned. Returns a NoSuchStream error code if the
// given stream or partition does not exist, and a PropertyAlreadySet error
// if this partition is already read-only.
rpc SetStreamReadonly(SetStreamReadonlyRequest) returns (SetStreamReadonlyResponse) {}

New messages:

// SetStreamReadonlyRequest is send to set a stream as read-only.
message SetStreamReadonlyRequest {
    string name               = 1; // Stream name
    repeated int32 partitions = 2; // Stream partitions
    bool readonly             = 3; // Should the readonly flag be set or removed
}

// SetStreamReadonlyResponse is sent by server after setting a stream's readonly
// flag.
message SetStreamReadonlyResponse {
    // Intentionally empty.
}

liftbridge-io / go-liftbridge