search

OrleansContrib / Orleans.Clustering.Kubernetes

Orleans Membership provider for Kubernetes
MIT License
188 stars 47 forks source link

Silo's fail in sequence if I go from 2 to 3 instances #50

Closed worldspawn closed 2 years ago

worldspawn commented 3 years ago

Hi, we've been using this library for a while in a dev context and we just tried to scale beyond 2 instances for the first time.

What happens is almost immediately one silo will fail, it then start up and another will fail. If we swap to azure storage membership this doesn't happen.

Here's a silo in its last moments

[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6fdaeb9d InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #457[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6fdaeb9d to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4b5895209cfe93c5086f48ac81befabc0000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @486061fd394345e11cdc7fc99abde0990000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  warn: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4a48f9d36f860ff5fe58c944db71f98e0000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4c9d690e9939a1bd086666daa0a232860000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  warn: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @44c2de53fd88c5212b34325eaecb5a9b0000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4dc092a0f1b53586476b33b7f41c3bb90000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  warn: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @49715c7e2bf900a0148980974659d4a00000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0 to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4d60b6f2dee90cc52fb699646b714a960000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  warn: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4860c1ca5d2dcb5de91afc3119e561a80000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8 to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4253032b40b2e8be1b667e48c75d25a40000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  info: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  warn: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  Process terminated. FATAL EXCEPTION from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! I should be Dead according to membership table (in CleanupTableEntries): entry = [SiloAddress=S10.244.6.207:10025:359102100 SiloName=projects-silo-695f465bb5-n7s6m Status=Dead HostName=projects-silo-695f465bb5-n7s6m ProxyPort=20025 RoleName= UpdateZone=0 FaultZone=0 StartTime = 2021-05-19 06:35:00.796 GMT IAmAliveTime = 2021-05-19 06:35:04.823 GMT Suspecters = [S10.244.6.210:10025:359102156] SuspectTimes = [2021-05-19 06:35:57.959 GMT]].. Exception: null.
[projects-silo-695f465bb5-n7s6m silo]  Current stack:    at System.Environment.get_StackTrace()
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.FatalErrorHandler.OnFatalException(Object sender, String context, Exception exception)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.KillMyselfLocally(String reason)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.CleanupMyTableEntries(MembershipTableData table)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.CleanupMyTableEntries(MembershipTableData table)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.RefreshInternal(Boolean requireCleanup)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable.ReadAll()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable.GetSilos()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.KubernetesExtensions.ListNamespacedCustomObjectAsync(IKubernetes operations, String group, String version, String namespaceParameter, String plural, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.Kubernetes.ListNamespacedCustomObjectWithHttpMessagesAsync(String group, String version, String namespaceParameter, String plural, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.WatcherDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ThreadPoolWorkQueue.Dispatch()
[projects-silo-695f465bb5-n7s6m silo]     at System.Environment.FailFast(System.String, System.Exception)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.FatalErrorHandler.OnFatalException(System.Object, System.String, System.Exception)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.KillMyselfLocally(System.String)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager+<CleanupMyTableEntries>d__47.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Orleans.Runtime.MembershipService.MembershipTableManager+<CleanupMyTableEntries>d__47, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]](<CleanupMyTableEntries>d__47 ByRef)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Orleans.Runtime.MembershipService.MembershipTableManager+<CleanupMyTableEntries>d__47, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]](<CleanupMyTableEntries>d__47 ByRef)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.CleanupMyTableEntries(Orleans.MembershipTableData)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__70.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__70, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.WatcherDelegatingHandler+<SendAsync>d__0.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.WatcherDelegatingHandler+<SendAsync>d__0, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.WatcherDelegatingHandler+<SendAsync>d__0, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.WatcherDelegatingHandler+<SendAsync>d__0, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.DiagnosticsHandler+<SendAsync>d__2, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.RedirectHandler+<SendAsync>d__4, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool+<SendWithNtConnectionAuthAsync>d__48.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<SendWithNtConnectionAuthAsync>d__48, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnection+<SendAsyncCore>d__53.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnection+<SendAsyncCore>d__53, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream+<ReadAsyncInternal>d__214`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReadAsyncInternal>d__214`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream+<<FillBufferAsync>g__InternalFillBufferAsync|215_0>d`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<<FillBufferAsync>g__InternalFillBufferAsync|215_0>d`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ThreadPoolWorkQueue.Dispatch()