The current implementation has a Backend shared between different local node instances. (NodeManager)[Backend](Node1, Node2, Node3, ..). The initial idea with this approach was to be able to easily scale up/down local instances within a KeyRange.
While it sounds nice, it doesn't really work out as not only is the epoch alignment blocking per Node. We are actually blocking until all nodes have reached the new epoch and the NodeManager sends back a confirmation after checkpointing :sweat_smile:
Set up state directories per Node instance /arcon/checkpoints/node_id/{node1, node2, node3}
The current implementation has a Backend shared between different local node instances. (NodeManager)[Backend](Node1, Node2, Node3, ..). The initial idea with this approach was to be able to easily scale up/down local instances within a KeyRange.
While it sounds nice, it doesn't really work out as not only is the epoch alignment blocking per Node. We are actually blocking until all nodes have reached the new epoch and the NodeManager sends back a confirmation after checkpointing :sweat_smile:
Set up state directories per Node instance
/arcon/checkpoints/node_id/{node1, node2, node3}