Bridge to kubernetes error: "Failed to get routing manager deployment status"

[OTimmy]

Describe the bug When starting a microservice in isolation, the bridge to kubernetes extension gives the following error "Failed to get routing manager deployments status". This does not occur when starting without isolation.

To Reproduce

1. Edit profile for bridge to kubernetes
2. Select service, namespace, cluster
3. Set routing isolation as enabled
4. Run the extension

Logs bridge-library

...
2021-03-26T06:44:36.4139030Z | Library | WARNG | Routing manager's GetStatus: Routing wasn't setup properly on time: ''
...
2021-03-26T06:44:38.4300345Z | Library | ERROR | Failed to get routing manager deployment status: ' : '

bridge-mindarocli

...
2021-03-26T06:42:17.0414010Z | MindaroCli | TRACE | Preparing to run Bridge To Kubernetes configured as pod dev/seto20972-frontend-admin-7fbcbc4c7f-tfv4f ...\n
2021-03-26T06:44:38.4401203Z | MindaroCli | WARNG | Failed to get routing manager deployment status
2021-03-26T06:44:38.4425690Z | MindaroCli | ERROR | Failed to get routing manager deployment status\n

rouingmanager-deployment pod

 ...
2021-03-26T06:41:08.5405476Z | RoutingManager | ERROR | ListServicesInNamespaceAsync threw HttpOperationException: StatusCode='Forbidden', ReasonPhrase='Forbidden', Content='{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \"system:serviceaccount:dev:routingmanager-sa\" cannot list resource \"services\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"services"},"code":403}\n'
2021-03-26T06:41:08.5415755Z | RoutingManager | WARNG | Error when retrieving all services : Forbidden: services is forbidden: User "system:serviceaccount:dev:routingmanager-sa" cannot list resource "services" in API group "" at the cluster scope
...
2021-03-26T06:41:08.9718988Z | RoutingManager | ERROR | Logging handled exception: System.ArgumentNullException: {"ClassName":"System.ArgumentNullException","Message":"Value cannot be null.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at System.Text.RegularExpressions.Regex.IsMatch(String input)\n   at Microsoft.BridgeToKubernetes.RoutingManager.RoutingStateEstablisher.<>c.<RunAsync>b__9_17(Networkingv1beta1IngressRule rule) in /src/routingmanager/RoutingStateEstablisher.cs:line 162\n   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)\n   at Microsoft.BridgeToKubernetes.RoutingManager.RoutingStateEstablisher.<>c.<RunAsync>b__9_13(Networkingv1beta1Ingress userIngress) in /src/routingmanager/RoutingStateEstablisher.cs:line 162\n   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source, Func`2 predicate)\n   at Microsoft.BridgeToKubernetes.RoutingManager.RoutingStateEstablisher.RunAsync(IDictionary`2 inputs, CancellationToken cancellationToken) in /src/routingmanager/RoutingStateEstablisher.cs:line 162\n   at Microsoft.BridgeToKubernetes.RoutingManager.RoutingManagerApp.RefreshAsync(CancellationToken cancellationToken) in /src/routingmanager/RoutingManagerApp.cs:line 247\n   at Microsoft.BridgeToKubernetes.RoutingManager.RoutingManagerApp.RefreshLoopAsync(CancellationToken cancellationToken) in /src/routingmanager/RoutingManagerApp.cs:line 184","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"System.Text.RegularExpressions","WatsonBuckets":null,"ParamName":"input"}

Temp\Microsoft.VisualStudio.Kubernetes.Debugging

{
  "timestamp": "2021-03-26 07-21-25.356",
  "message": "Value cannot be null.\r\nParameter name: token",
  "exception": {
    "ClassName": "System.ArgumentNullException",
    "Message": "Value cannot be null.",
    "Data": null,
    "InnerException": null,
    "HelpURL": null,
    "StackTraceString": "   at Microsoft.Rest.TokenCredentials..ctor(String token, String tokenType)\r\n   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.AccountManagement.AccountManagerExtensions.<AcquireAzureCredentialsAsync>d__1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.AccountManagement.AccountManagerExtensions.<AcquireAzureCredentialsAsync>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.VisualStudio.Kubernetes.Debugging.Core.Services.Configuration.ConfigurationCallbacks.<GetAzureClusterSourcesAsync>d__31.MoveNext()",
    "RemoteStackTraceString": null,
    "RemoteStackIndex": 0,
    "ExceptionMethod": "1\n.ctor\nMicrosoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35\nMicrosoft.Rest.TokenCredentials\nVoid .ctor(System.String, System.String)",
    "HResult": -2147467261,
    "Source": "Microsoft.Rest.ClientRuntime",
    "WatsonBuckets": null,
    "ParamName": "token"
  }
}

Environment Details Client's version: Microsoft Visual Studio Enterprise 2019 4.804084 Operating System: Windows 10 Enterprise 10.0.19042 Bridge to Kubernetes 2.1.20210317.1

[rakeshvanga]

@OTimmy Thanks for sharing the logs and reporting the issue. We are looking into this issue right away and provide you with a fix soon. Thanks.

[daniv-msft]

@OTimmy We believe we have the fix, and we'll release it with our next release. As a mitigation, would it be possible for you to make sure you declare the host field in your ingress' rules? It should be sufficient to bypass that problem while we release the fix.

[OTimmy]

@daniv-msft @rakeshvanga Thanks. The host field is already defined in our ingress rules.

[daniv-msft]

Thanks @OTimmy for your reply. It seems indeed that you encountered another issue as well: User "system:serviceaccount:dev:routingmanager-sa" cannot list resource "services" in API group "" at the cluster scope

This seems like an RBAC issue. We're having a look at it on our side, but at first sight I expect that you will likely need to provide more rights to the RoutingManager service account (routingmanager-sa) to allow it to work on your cluster.

[OTimmy]

@daniv-msft Thanks for taking your time to troubleshoot this with me. It will be greatly apperciated when we will get this working again.

From running the following command, it looks like it should have the right to list services.

kubectl auth can-i --list --as=system:serviceaccount:dev:routingmanager-sa

Resources                                       Non-Resource URLs   Resource Names   Verbs
...
services                                        []                  []               [list create update delete]

[OTimmy]

@daniv-msft @rakeshvanga Fantastic work! The latest release fixed my problems :)

[daniv-msft]

Thanks @OTimmy for giving it a try, and glad that our fix unblocked you! :)