Deleting stale silos that are marked active but the pod has already been removed

OrleansContrib / Orleans.Clustering.Kubernetes

Orleans Membership provider for Kubernetes

MIT License

188 stars 47 forks source link

Deleting stale silos that are marked active but the pod has already been removed #48

Open tomodutch opened 3 years ago

tomodutch commented 3 years ago

After a pod exits prematurely it will create a silo entry with the status "Active". When this pod restarts it scans all silos from etcd and attempts to connect with its predecessor. I've noticed this behaviour several times when running in production on orleans 3 in a single silo cluster.

ReubenBond commented 3 years ago

The Orleans.Hosting.Kubernetes package does this: https://dotnet.github.io/orleans/docs/deployment/kubernetes.html It's still in "beta", but if you can try it and provide feedback, that will be helpful

tomodutch commented 3 years ago

Thanks! I will try it out in the next few days 😄

turowicz commented 3 years ago

@ReubenBond the package you directed @TFarla to doesn't do clustering. Could you please tell me what is the benefit of using it at all?

turowicz commented 3 years ago

Or should it be used in tandem?

ReubenBond commented 3 years ago

@turowicz the hosting package does a few things:

Configure silos based on the pod's environment (IP, name, ClusterId/ServiceId)
Monitor Kubernetes for changes in active pods, so that deleted pods can be removed immediately, without the need for health probes. This doesn't remove the need for health probes altogether, but provides a short-cut in case an administrative action was taken.
Kill pods in Kubernetes when they are declared dead by the cluster

It doesn't replace the need for a clustering provider. The sample here shows how to use it, and uses Redis for clustering (but something else could be used, of course) https://github.com/ReubenBond/hanbaobao-web

turowicz commented 3 years ago

@ReubenBond thanks! I'm testing Orleans.Clustering.Kubernetes with Orleans.Hosting.Kubernetes now. It seems like this combo will work well.