Pipeless 1.x does not yet include distributed processing as its predecessor (0.x) which had it enabled by default.
Implementing distributed processing should be simple. It just requires to provide an option (--distributed) that replaces the local bus by a message broker like Kafka, Redis or RabbitMQ. Supporting different brokers would be ideal.
Also, the Pipeless node in distributed mode should behave as a worker node, simply taking events from the broker and processing them. Only of frame change events should be moved to the broker and ideally only when the node receiving the stream has not enough capacity.
This includes the implementation of a distributed KV store. See #71
Pipeless 1.x does not yet include distributed processing as its predecessor (0.x) which had it enabled by default.
Implementing distributed processing should be simple. It just requires to provide an option (
--distributed) that replaces the local bus by a message broker like Kafka, Redis or RabbitMQ. Supporting different brokers would be ideal.Also, the Pipeless node in distributed mode should behave as a worker node, simply taking events from the broker and processing them. Only of frame change events should be moved to the broker and ideally only when the node receiving the stream has not enough capacity.
This includes the implementation of a distributed KV store. See #71