Handle housekeeping with sharded cluster

[hackerwins]

What would you like to be added:

We recently introduced cluster mode based on the shard key. This shard key is used to assign APIs requested by clients to servers in the cluster based on project or document.

Housekeeping operations should also apply this sharded cluster structure. We can think of two options:

1. Only the master server executes housekeeping in the cluster
2. Split housekeeping tasks per project and execute them on servers mapped by project

Why is this needed:

• Bugs may occur in housekeeping of sharded cluster.

[krapie]

The options that you've suggested look good to me. I have listed some of the considerations for these options below.

1. Only the master server executes housekeeping in the cluster.
   • Implementation: Using etcd with the raft leader election algorithm will ease this implementation. We can use K8s go client leader election package with etcd for this operation:
   • Package: https://pkg.go.dev/k8s.io/client-go/tools/leaderelection
   • Example: https://github.com/travisjeffery/kube-leaderelection.
   • Pros: It looks easy to implement.
   • Cons: As workload for housekeeping increases, housekeeping overhead on the master server will increase, which may lead to server panic or failure. 
2. Split housekeeping tasks per project and execute them on servers mapped by project.
   • Implementation: We need to map the server with the project range to housekeeping in etcd and pull this information before housekeeping. Maybe we can reference Yorkie's etcd implementation in the sync package for this task
   • Yorkie sync package's Membership: https://github.com/yorkie-team/yorkie/blob/3022eecebad070db32d06ba3c762d54875743dc4/server/backend/sync/etcd/membermap.go
   • Pros: We can distribute housekeeping workload just like document workload sharding, which will ease the overhead of servers to handle housekeeping.
   • Cons: Implementation seems to be more difficult than the first option. 

These options seem like opposite options to me, but I think they both can be implemented by using etcd. I'll keep searching for more information on these options.

[krapie]

I have implemented leader election with K8s lease object, but this will make yorkie k8s dependent.

Therefore, I will change this implementation with MongoDB, where yorkie already have dependency on.

To be specific about the implementation, I will use the way of nodes try to update on the same document(same _id) by checking lease_expire_at field.

Also, to ensure that only one node becomes the leader when a lease has expired, I will use TTL indexes to remove document on expiry, preventing nodes to update the same document simultaneously.

[krapie]

@hackerwins Are we proceeding this issue?

[hackerwins]

No, it's still on hold. During housekeeping, it seems that Client Deactivation competing across nodes doesn't cause significant issues, so we haven't proceeded with this.