xanderdunn / kube-state-rs

A stateless Kubernetes service in Rust to preserve node state between removing and adding nodes in a cluster
0 stars 0 forks source link

Replace `StatefulSet` with `Deployment` #12

Closed xanderdunn closed 11 months ago

xanderdunn commented 11 months ago

Currently the service is deployed as a StatefulSet so that each node has a name with an incrementing identifier: kube-state-rs-0, kube-state-rs-1, ...

This is so that we can choose which partition of nodes belongs to this particular replica:

        let my_hostname = std::env::var("HOSTNAME").unwrap();
        let my_replica_id: usize = my_hostname.split('-').last().unwrap().parse::<usize>()?;
        let my_nodes = nodes_list
            .items
            .chunks(num_ready_replicas)
            .nth(my_replica_id)
            .unwrap_or(&[]);

However, the service itself is stateless and makes use of no other feature of a StatefulSet. It would therefore be desirable to use a different mechanism for knowing the replica ID so that we can use a stateless Deployment.

xanderdunn commented 11 months ago

The Redis INCR operation would work well here. On each iteration of the nodes, each replica can increment a replica_id value in the DB and INCR will atomically return the current value and increment it by 1.

This would require #3.

xanderdunn commented 11 months ago

The current implementation with separate Watcher and Processor pods does not rely on any pod naming for its partitioning of work across nodes. Hence, we're using Deployments rather than StatefulSets. Each Processor randomly chooses a node to process and then performs leader election with a lease on that node's name. If some other Processor is already operating on that node, then it fails leader election and randomly chooses a different node to work on.