Failed to connect (after upgrade?) to cluster: Collection was modified

[AndreKfm]

Describe the bug We had a running environment with BrideToKubernetes (BTK) but out of the blue it doesn't operate anymore. It's always prompting "Collection was modified, enumeration operation may not execute" directly after start of debugging.

To Reproduce We simply (try to) start Visual Studio (Enterprise) - Bridge To Kubernetes - Local Config Konfiguration debugging of any project.

**Expected behavior** Project to start debugging without any issues. **Logs** 2021-05-03T11:24:07.1794456Z | MindaroCli | TRACE | Event: Command.Start {"properties":{"arguments":"prep-connect --namespace default --output json","isRoutingEnabled":"false"},"metrics":null}\nOperation context: {"clientRequestId":null,"correlationId":"6f323113-d5b4-41c2-9786-bb201f5e2c71:3ceed11b9933","requestId":null,"userSubscriptionId":null,"startTime":"2021-05-03T11:24:07.0086921Z","userAgent":"LPK/2.1.20210411.1-5dda536c VisualStudio/16.9.31205.134","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210408.1","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Microsoft Windows 10.0.19041","framework":".NET Core 3.1.14","macAddressHash":"ce547ee5c924e8aa3f4db9c3e39d42ada0bfb153129400a223091a98df55637e","processId":20120,"targetEnvironment":"Production","commandId":"3ceed11b9933"}} 2021-05-03T11:24:07.2393127Z | MindaroCli | TRACE | Running Microsoft.BridgeToKubernetes.Exe.Commands.Connect.PrepConnectCommand... 2021-05-03T11:24:07.2414693Z | MindaroCli | TRACE | Initializing progress reporter... 2021-05-03T11:24:08.2864778Z | MindaroCli | ERROR | Dependency: Prep Connect {"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}} 2021-05-03T11:24:09.2827713Z | MindaroCli | ERROR | An unexpected error occurred: 'Collection was modified; enumeration operation may not execute.'\n 2021-05-03T11:24:09.2830895Z | MindaroCli | ERROR | To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report.\n 2021-05-03T11:24:09.2833245Z | MindaroCli | ERROR | For diagnostic information, see logs at 'C:\Users\Dev\AppData\Local\Temp\Bridge To Kubernetes'.\n 2021-05-03T11:24:09.2865923Z | MindaroCli | ERROR | Logging handled exception: System.InvalidOperationException: {"ClassName":"System.InvalidOperationException","Message":"Collection was modified; enumeration operation may not execute.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Collections.Generic.List`1.Enumerator.MoveNextRare()\r\n at System.Collections.Generic.List`1.Enumerator.MoveNext()\r\n at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.<>c__DisplayClass31_1.<_CollectServicesToRouteAsync>b__1(V1Endpoints e)\r\n at System.Collections.Generic.IEnumerableExtensions.ExecuteForEach[T](IEnumerable`1 items, Action`1 action, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.<>c__DisplayClass31_0.<<_CollectServicesToRouteAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.GetReachableServicesAsync(String namespaceName, ILocalProcessConfig localProcessConfig, IProgress`1 progress, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func`1 func, FailureConfig failureConfig)\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetElevationRequestsAsync(IProgress`1 progress, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.PrepConnectCommand.ExecuteAsync()\r\n at Microsoft.BridgeToKubernetes.Exe.CliApp.RunCommandAsync(String[] args, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Exe.CliApp.ExecuteAsync(String[] args, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233079,"Source":"System.Private.CoreLib","WatsonBuckets":null} 2021-05-03T11:24:09.2879364Z | MindaroCli | TRACE | Event: Command.End {"properties":{"arguments":"prep-connect --namespace default --output json","result":"Failed","failureReason":"Collection was modified; enumeration operation may not execute."},"metrics":{"duration":2157.0}} **Environment Details** Client used: Visual Studio, 2019 Enterprise, 16.9.4, Bridge To Kubernetes 0.1 Operating System: Windows 10, 2004 **Additional context**

[build3GB]

Additional information:

Seems to be related to our deployment, another fresh (nearly empty) Kubernetes deployment doesn't seem to have that issues

[rakeshvanga]

@build3GB, Thanks for sharing the update. Do you have any headless services deployed in the first instance and then they are removed in the second fresh deployment? From my investigation it seems that this issue is caused when there are headless services defined as part of your deployment.

FYI, we have fixed this issue and released it for VS Code. In the meantime until a release is done for VS, if possible you could work using VS Code.

[AndreKfm]

@rakeshvanga, yes exactly we have a lot of headless services.

Thank you very much for the hint.
Though we didn't change anything, so it must be an issue introduced in the latest version.

VS Code, we can give it a try, though it's currently not a complete replacement for our developers, since they all use VStudio and are not customized with VSCode. Thx also for this hint.