tylertreat
commented
4 years ago I have another use case for pausing (specifically auto-pausing) in Liftbridge itself as it relates to consumer groups. When consumer groups are implemented, they will rely on an internal offset stream to checkpoint consumer stream positions. This would be a partitioned stream where the partition used to checkpoint is determined by the consumer group name.
As a result, it's possible that a large number (or all) of the partitions are mostly idle. Therefore, I think pausing should occur at the partition level rather than the stream level. Ultimately, a stream is just a collection of partitions anyway. The stream itself doesn't really consume any resources. In the consumer group use case, we wouldn't want a message published on partition n to cause all other partitions to wake up.
To do this, PauseStreamRequest could, optionally, take a list of partition ids to pause. If none are provided, then all of the stream's partitions are paused. If we do want all partitions to wake up when one partition is published to, perhaps this could be an additional flag on PauseStreamRequest? Otherwise, I think the default behavior should be to operate at the partition level.
As an aside, I think this should eventually be generalized to allow for auto stream creation along with auto-pausing, but like auto-pausing that can come at a future time.
Jmgr
This PR proposes an API to “pause” a stream, that is, deactivate its partitions. Publication to one of these partitions using the Liftbridge API “resumes” the stream, restarting its partitions.
Context
Our use-case will include having a significant number of streams, but only a small fraction of those will be active at any given point in time (sparse streams).
In case of a server shutdown we would like to restart only active streams, not paused ones. The activity would record stream pausing and resuming, so that on server startup only the active streams would be restored.
The aims of this PR are twofold:
inactive streams are paused so as to avoid the memory and CPU cost of maintaining stream activity and replication when there are no messages;
by passing pause and resume events to a meta-event stream, we can schedule activity for a consumer pool more effectively.
This PR is therefore linked to the activity stream PR (https://github.com/liftbridge-io/liftbridge/pull/169).
Implementation
The implementation proposed in this PR provides a new
PauseStreamAPI that allows clients to request that a stream be paused. Pausing a steam is very similar to closing a stream; that is, closing all of its partitions. Closing a partition, in turn, means that the server is no longer a leader or a follower for it and that its commit log is also closed. A “paused” flag is also set on the stream’s partitions. This allows us to spare CPU, memory usage and close unused file handles. One could also imagine an auto-pausing feature for each stream, where after a certain duration without any message publication a stream and its partition would be automatically paused. This would have to be added in a future PR however.Resuming a stream occurs when a message is published to one of its partitions. All partition are then re-created using the same parameters, including the same data directory. The message is then published as usual.