dotnet / orleans

Cloud Native application framework for .NET
https://docs.microsoft.com/dotnet/orleans
MIT License
9.89k stars 2.01k forks source link

Orleans membership table is not cleared properly from obsolete silos #8968

Open UladzimirVyaziginTR opened 1 month ago

UladzimirVyaziginTR commented 1 month ago

Context:

  1. I am using Orleans 8.1.0 and PostgreSQL clustering in Kubernetes containers. These containers are recreated enough frequently.
  2. Near the half of all requests fails due to incorrect IP address of a defunct silo. The error message may vary but the most frequent of them is on screenshot below.
  3. Cluster membership options are configured according to the screenshot below. The value of 'MembershipTableCleanupInMinutes' is 5 minutes.

Questions:

  1. Is it the bug of Orleans 8.1.0 or misconfiguration of cluster membership options?
  2. What exactly should I do to overcome the issue with defunct silos?

image image