search

microsoft / mindaro

Bridge to Kubernetes - for Visual Studio and Visual Studio Code
MIT License
307 stars 106 forks source link

Erorr when stopping debugging #201

Open Vaccano opened 3 years ago

Vaccano commented 3 years ago

Describe the bug When I stop debugging, an error message appears saying that something went wrong. It does not seem to cause any issues.

To Reproduce

  1. Connect my Web API ASP.Net Core 3.1 service to my Kubernetes Cluster via Bridge to Kubernetes
  2. Start debugging.
  3. Debug for a while
  4. Press the stop button.
  5. A few seconds after the stop button is pressed, an banner like this appears in Visual Studio: image (It says "Visual Studio ran into an unexpected problem, and may affect some project system functions." An it has a "Open log file" button.)

When I open the log file, this is what is in it:

Click to Expand Log File Details 
===================
7/9/2021 2:26:15 PM
Recoverable
System.AggregateException: One or more errors occurred. ---> StreamJsonRpc.RemoteInvocationException: Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. System.InvalidOperationException: The 'InvokeCoreAsync' method cannot be called if the connection is not active
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReconnectingConnectionState.WaitForActiveConnectionAsync(String methodName, String memberName, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsyncCore(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsync(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.BridgeToKubernetes.Common.DevHostAgent.DevHostAgentExecutorClient.<>c__DisplayClass31_1.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.InvalidOperationException: The 'InvokeCoreAsync' method cannot be called if the connection is not active
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReconnectingConnectionState.WaitForActiveConnectionAsync(String methodName, String memberName, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsyncCore(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsync(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.BridgeToKubernetes.Common.DevHostAgent.DevHostAgentExecutorClient.<>c__DisplayClass31_1.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

   at StreamJsonRpc.JsonRpc.d__139`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ConnectionManager.ConnectionManagerService.d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ConnectionManager.ConnectionManagerService.<>c__DisplayClass27_0.<b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ConnectionManager.ConnectionManagerService.<>c__DisplayClass27_0.<b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ServiceHub.ConnectServiceBrokerClient.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.ProjectSystem.CommonProjectSystemTools.Rethrow(Exception ex)
   at Microsoft.VisualStudio.ProjectSystem.ProjectErrorReporting.<>c__DisplayClass6_0.b__0()
   at Microsoft.VisualStudio.ProjectSystem.ExceptionFilter.<>c__DisplayClass2_0.g__Action|0()
   at GuardMethodClass.GuardMethod(Func`1 , Func`2 , Func`2 )
   --- End of inner exception stack trace ---
---> (Inner Exception #0) StreamJsonRpc.RemoteInvocationException: Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. System.InvalidOperationException: The 'InvokeCoreAsync' method cannot be called if the connection is not active
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReconnectingConnectionState.WaitForActiveConnectionAsync(String methodName, String memberName, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsyncCore(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsync(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.BridgeToKubernetes.Common.DevHostAgent.DevHostAgentExecutorClient.<>c__DisplayClass31_1.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.InvalidOperationException: The 'InvokeCoreAsync' method cannot be called if the connection is not active
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReconnectingConnectionState.WaitForActiveConnectionAsync(String methodName, String memberName, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsyncCore(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsync(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.BridgeToKubernetes.Common.DevHostAgent.DevHostAgentExecutorClient.<>c__DisplayClass31_1.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

   at StreamJsonRpc.JsonRpc.d__139`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ConnectionManager.ConnectionManagerService.d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ConnectionManager.ConnectionManagerService.<>c__DisplayClass27_0.<b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ConnectionManager.ConnectionManagerService.<>c__DisplayClass27_0.<b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.ServiceHub.ConnectServiceBrokerClient.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.ProjectSystem.CommonProjectSystemTools.Rethrow(Exception ex)
   at Microsoft.VisualStudio.ProjectSystem.ProjectErrorReporting.<>c__DisplayClass6_0.b__0()
   at Microsoft.VisualStudio.ProjectSystem.ExceptionFilter.<>c__DisplayClass2_0.g__Action|0()
   at GuardMethodClass.GuardMethod(Func`1 , Func`2 , Func`2 )
RPC server exception:
Microsoft.VisualStudio.Kubernetes.Debugging.RpcContracts.DscException: Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. System.InvalidOperationException: The 'InvokeCoreAsync' method cannot be called if the connection is not active
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReconnectingConnectionState.WaitForActiveConnectionAsync(String methodName, String memberName, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsyncCore(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsync(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.BridgeToKubernetes.Common.DevHostAgent.DevHostAgentExecutorClient.<>c__DisplayClass31_1.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.InvalidOperationException: The 'InvokeCoreAsync' method cannot be called if the connection is not active
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReconnectingConnectionState.WaitForActiveConnectionAsync(String methodName, String memberName, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsyncCore(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsync(String methodName, Type returnType, Object[] args, CancellationToken cancellationToken)
   at Microsoft.BridgeToKubernetes.Common.DevHostAgent.DevHostAgentExecutorClient.<>c__DisplayClass31_1.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

      at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.DscClient.d__16.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.DscClient.d__18.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.Connect.ConnectService.<>c__DisplayClass11_0.<b__0>d.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.VisualStudio.Kubernetes.Debugging.RpcContracts.Telemetry.KubeConnectScopeBase`1.d__26.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at Microsoft.VisualStudio.Kubernetes.Debugging.RpcContracts.Telemetry.KubeConnectScopeBase`1.d__26.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.Connect.ConnectService.d__11.MoveNext()
<---

===================
**Expected behavior** This does not seem to cause any issues, so the only expected behavior is that the error banner would not show. **Logs** Logs had my company info in them. I would be happy to provide logs privately if desired. **Environment Details** Client used: **Visual Studio** Visual Studio version: **Version 16.9.2** Client's version: **Windows 10 Version - 10.0.19042 Build 19042** Operating System: **Windows 10 Version - 10.0.19042 Build 19042** **Additional context** I am not sure if it is related, but I have noticed that have have to wait about a minute or so before I can successfully startup a Bridge to Kubernetes debug session again.
daniv-msft commented 3 years ago

Thanks @Vaccano for logging this issue! This InvalidOperationException: The 'InvokeCoreAsync' method cannot be called if the connection is not active is an error we've seen in our telemetry but that we could hardly reproduce. Adding @lolodi for FYI, as he's looking at this bug. One of our hypotheses is that it was related to very long connections (let's say staying connected using Bridge for more than an hour). @Vaccano Could you please confirm if it is the case for you? Or does it reproduce as well for small ones?

Regarding the banner you see/the fact that you cannot debug again right after, that's something we should improve indeed. @danegsta Could you please let us know if you have an idea or where these might come from? Do we handle errors differently when stopping the connection (shouldn't we swallow this error and just log it on our side)?

Vaccano commented 3 years ago

It reproduces with much shorter debugging times. I just had it happen with a less than 5 minute debugging session. The time it happened that had me log this issue was less than 10 minutes tops.

daniv-msft commented 3 years ago

OK thanks, it's definitely good to know! @lolodi on our side is looking into it, and we might ask you for more information if required.

Rob623 commented 3 years ago

I'm also getting the same error and after it happens VS has to be restarted because its... not working right anymore... text is missing where hints should be - getting empty spaces in text editor... and most importantly, its getting unresponsive or freezes completely. I'm guessing that the next VS extension is being run on the same thread and once BTK throws the exception, the extensions run loop (guessing) is broken or something similar.

@Vaccano did you see such behavior as well? if not, are you using additional extension? I suspect that resharper is the one getting broken in my case. The reason is that it influences the text editor in such a way that can result in the unusable state I`m getting.

Vaccano commented 3 years ago

@Rob623 - I have not seen any such issues. I do use ReSharper, but it has not interacted with Bridge to Kubernetes in any way I can see. Nor have I seen any issues with with my text editing in Visual Studio.

As I said in my bug report, starting up debugging right after stopping does not work out well. I got issues where my Kubernetes application service is put in an unstable state. I got error messages complaining about replicas and I had to re-deploy the application to Kubernetes to work around it. But these symptoms are not as what you are reporting. (And this is easily avoided by waiting a minute or two before starting to debug again.)

Rob623 commented 3 years ago

Hmm I see. since the stack trace looks the same, Im guessing that the issues are related however, since the symptoms appear to be different, @daniv-msft should I open a separate issue for it? btw I forgot to mention that Ive also seen the problem @Vaccano mentioned. Anyway, if I find more info or better reproduction scenario I`ll update.

daniv-msft commented 3 years ago

@Rob623 Thanks, I think it's fine to keep the same issue for now as even though the symptoms are slightly different, I believe they have the same root cause. @lolodi made good progress on a tentative fix for this, and we will work to integrate them in Visual Studio to get the changes to you.

tigerquoll commented 3 years ago

@daniv-msft @lolodi I've seeing the same issues here as well. mindaro-vscode-2021-07-14T05-47-42.911Z.txt Very easy to reproduce, if I stop at a breakpoint, continue, and then stop debugging I get this issue. I have a busy cluster which has previously caused timeouts in your commands in the past if that helps you track down the issue.

tigerquoll commented 3 years ago

@daniv-msft @lolodi is there any update on the issue?

daniv-msft commented 3 years ago

@tigerquoll Thanks for following up! We're releasing our new version (including the fix from @lolodi) in VS Code this week. It's already available to 10% of the customers, and we will gradually increase this if we don't detect issues. Visual Studio will come right after, so likely in 1/1.5 weeks.