matthewbrumpton-oddsbods
commented
2 years ago Another observation is OrleansClusterVersion is deleted after a period of time and then being recreated after we restart the app.
OrleansClusterVersion after we restart the app:
apiVersion: v1 items: apiVersion: orleans.dot.net/v1 clusterId: clusterId clusterVersion: 4 kind: OrleansClusterVersion metadata: creationTimestamp: "2021-11-15T14:43:13Z" generation: 5 name: name namespace: namspace resourceVersion: "148545790" uid: d4a66cf5-18a9-4474-bf88-351d496ac61d kind: List metadata: resourceVersion: "" selfLink: ""
OrleansClusterVersion when deleted:
kubectl get clusterversions --namespace namespace -o yaml apiVersion: v1 items: [] kind: List metadata: resourceVersion: "" selfLink: ""
We have a large number of KubeMembershipTable exceptions coming from our orleans application, the errors start after the application has been running for 5-9 hours.
Message: Unable to find silo entry 172.19.16.62-50000-374433332.
CategoryName: Orleans.Clustering.Kubernetes.KubeMembershipTable
Exception type: System.InvalidOperationException
Failed method: Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext
Problem Id: System.InvalidOperationException at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext
Assembly: Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null
Call Stack: System.InvalidOperationException: at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext (Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)