search

microsoft / mindaro

Bridge to Kubernetes - for Visual Studio and Visual Studio Code
MIT License
307 stars 106 forks source link

Failing to forward traffic to local macOS #224

Open Sher-Chowdhury opened 3 years ago

Sher-Chowdhury commented 3 years ago

Describe the bug Hello, I'm working on an operator that was created using the operator-sdk. I'm trying to get my openshift cluster traffic to forwarded to my local macbook. When I tried following the instructions, it managed create the restore pod, but it's log shows that it keeps dying:

$ oc get pods my-dummy-operator-restore-9b74e-tvbmt
NAME                                          READY   STATUS    RESTARTS   AGE
my-dummy-operator-restore-9b74e-tvbmt   1/1     Running   4          6m22s

oc logs -f my-dummy-operator-restore-9b74e-tvbmt
2021-09-02T22:03:18.1153787Z | RestorationJob | TRACE | Waiting to restore previous state on Deployment test-namespace/my-dummy-operator...\nOperation context: <json>{"clientRequestId":"7cc60257-3152-4678-ab65-7fba88c9792e","correlationId":"64f41c1b-b0e6-471d-9eff-8399446689091630619987829:4ff690733350:365338ea3b21","requestId":null,"userSubscriptionId":null,"startTime":"2021-09-02T22:03:17.5412239Z","userAgent":"RestorationJob/1.0.20210723.2","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210723.2","requestHeaders":{},"loggingProperties":{"applicationName":"RestorationJob","deviceOperatingSystem":"Linux 4.18.0-240.22.1.el8_3.x86_64 #1 SMP Thu Mar 25 14:36:04 EDT 2021","framework":".NET Core 3.1.16","processId":1,"targetEnvironment":"Production"}}</json>
2021-09-02T22:03:28.6478301Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:28.7381267Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:28.7393179Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:28.7423988Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:03:28.7658849Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"0","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":598.0}}</json>
2021-09-02T22:03:33.7896167Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:33.8021296Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:33.8037203Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:33.8050670Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:03:33.8071637Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"1","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":31.0}}</json>
2021-09-02T22:03:38.8220937Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:38.8322549Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:38.8335120Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:38.8345520Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:03:38.8362568Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"2","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":26.0}}</json>
2021-09-02T22:03:43.8505308Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:43.8584799Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:43.8588221Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:43.8592443Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:03:43.8600589Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"3","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":21.0}}</json>
2021-09-02T22:03:48.8743129Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:48.8848887Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:48.8852749Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:48.8857232Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:03:48.8865187Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"4","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":24.0}}</json>
2021-09-02T22:03:53.8983950Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:53.9064767Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:53.9072252Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:53.9081559Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:03:53.9092644Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"5","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":20.0}}</json>
2021-09-02T22:03:58.9195637Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:58.9267684Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:58.9273276Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:03:58.9277928Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:03:58.9286471Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"6","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":17.0}}</json>
2021-09-02T22:04:03.9421855Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:03.9540946Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:03.9545852Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:03.9551197Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:04:03.9561196Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"7","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":25.0}}</json>
2021-09-02T22:04:08.9659802Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:08.9718036Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:08.9721962Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:08.9725751Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:04:08.9732605Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"8","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":15.0}}</json>
2021-09-02T22:04:13.9964174Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:14.0043220Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:14.0047979Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:14.0052446Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:04:14.0061035Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"9","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":31.0}}</json>
2021-09-02T22:04:19.0156033Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:19.0232453Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:19.0236410Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:19.0241907Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:04:19.0258266Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"10","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":17.0}}</json>
2021-09-02T22:04:24.0363254Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"GetV1DeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:24.0431701Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:24.0435601Z | RestorationJob | TRACE | Dependency: Kubernetes <json>{"name":"Kubernetes","target":"ListPodsForDeploymentAsync","success":true,"duration":null,"properties":{}}</json>
2021-09-02T22:04:24.0441057Z | RestorationJob | WARNG | Deployment test-namespace/my-dummy-operator has already been restored
2021-09-02T22:04:24.0449175Z | RestorationJob | TRACE | Event: RestorationJob-AgentPing <json>{"eventName":"RestorationJob-AgentPing","properties":{"restorePerformed":"false","numFailedPings":"11","hasConnectedClients":"","result":"Failed"},"metrics":{"durationInMs":17.0}}</json>
2021-09-02T22:04:24.0458385Z | RestorationJob | ERROR | Failed to ping agent 12 times. Exiting...
2021-09-02T22:04:24.3169688Z | RestorationJob | ERROR | Logging handled exception: System.NullReferenceException: {"ClassName":"System.NullReferenceException","Message":"Object reference not set to an instance of an object.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.BridgeToKubernetes.Common.Logging.ThreadSafeFileWriter.Dispose() in /src/common/Logging/Writers/ThreadSafeFileWriter.cs:line 42\n   at Autofac.Core.Disposer.Dispose(Boolean disposing)\n   at Autofac.Util.Disposable.Dispose()\n   at Autofac.Core.Lifetime.LifetimeScope.Dispose(Boolean disposing)\n   at Autofac.Util.Disposable.Dispose()\n   at Autofac.Core.Container.Dispose(Boolean disposing)\n   at Autofac.Util.Disposable.Dispose()\n   at Microsoft.BridgeToKubernetes.Common.ConsoleRunner.RunApp[T](Func`1 containerFunc, String[] args, String userAgent, Int32 logFlushTimeoutMs, Boolean suppressConsoleErrorLogs) in /src/common/ConsoleRunner.cs:line 66\n   at Microsoft.BridgeToKubernetes.DevHostAgent.RestorationJob.Program.Main(String[] args) in /src/devhostagent.restorationjob/Program.cs:line 14","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"Microsoft.BridgeToKubernetes.Common","WatsonBuckets":null}
2021-09-02T22:04:24.3268180Z | RestorationJob | ERROR | Unhandled AppDomain exception! System.NullReferenceException: Object reference not set to an instance of an object.
2021-09-02T22:04:24.3271390Z | RestorationJob | ERROR | AppDomain base exception details: System.NullReferenceException: Object reference not set to an instance of an object.
2021-09-02T22:04:24.3274158Z | RestorationJob | ERROR | Is AppDomain terminating? 'True'
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.BridgeToKubernetes.Common.Logging.ThreadSafeFileWriter.Dispose() in /src/common/Logging/Writers/ThreadSafeFileWriter.cs:line 42
   at Autofac.Core.Disposer.Dispose(Boolean disposing)
   at Autofac.Util.Disposable.Dispose()
   at Autofac.Core.Lifetime.LifetimeScope.Dispose(Boolean disposing)
   at Autofac.Util.Disposable.Dispose()
   at Autofac.Core.Container.Dispose(Boolean disposing)
   at Autofac.Util.Disposable.Dispose()
   at Microsoft.BridgeToKubernetes.Common.ConsoleRunner.RunApp[T](Func`1 containerFunc, String[] args, String userAgent, Int32 logFlushTimeoutMs, Boolean suppressConsoleErrorLogs) in /src/common/ConsoleRunner.cs:line 66
   at Microsoft.BridgeToKubernetes.DevHostAgent.RestorationJob.Program.Main(String[] args) in /src/devhostagent.restorationjob/Program.cs:line 14

Here's what my launch.json looks like:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Operator with Kubernetes",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/stable/operators/my-dummy-operator/main.go",
            "env": {
                "OPERATOR_NAME": "my-dummy-operator",
                "WATCH_NAMESPACE": "test-namespace",
                "SKIP_WEBHOOK": false
            },
            "args": [],
            "preLaunchTask": "bridge-to-kubernetes.resource"
        }
    ]
}

And here's what my tasks.json looks like:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "bridge-to-kubernetes.resource",
            "type": "bridge-to-kubernetes.resource",
            "resource": "my-dummy-webhook",
            "resourceType": "service",
            "ports": [
                8443
            ],
            "targetCluster": "my-test-cluster-com:6443",
            "targetNamespace": "test-namespace",
            "useKubernetesServiceEnvironmentVariables": false
        }
    ]
}

Since the preLaunchTask fails, I think it means that the go code doesn't even start running, so nothing on my macbook ends up listening on port 8443.

Cluster version:

$ oc version
Client Version: 4.6.8
Server Version: 4.7.13
Kubernetes Version: v1.20.0+df9c838

To Reproduce Steps to follow to reproduce this issue.

  1. install an operator on the cluster
  2. Follow instructions - https://docs.microsoft.com/en-gb/visualstudio/bridge/bridge-to-kubernetes-vs-code to try to forward service's traffic to local workstation

Logs See attachments bridge-endpointmanager-2021-09-02-22-02-59-57570.txt bridge-mindarocli-2021-09-02-22-02-51-57561.txt bridge-library-2021-09-02-22-02-51-57561.txt

Environment Details Client used (VS Code/Visual Studio): VS Code

Client's version: 1.60.0 Operating System: macOS Big Sur 11.5.2
Sher-Chowdhury commented 3 years ago

I tried getting the bridge to work using the sample todo-app, but the pods for that app failed to start up, probably because that app was meant to work on kubernetes cluster rather than an openshift cluster.

I then tried this demo - https://github.com/ThorstenHans/bridge-to-kubernetes-demo, and this time the bridging worked as expected. So not sure where I'm going wrong when running this vs code extension against my operator repo.

amsoedal commented 3 years ago

Hi @Sher-Chowdhury, thanks so much for your question! A few thoughts from me:

  1. Can you include the logs from the extension so that I can see the output from the preLaunchTask when it exits? To do this, open the command palette (Command+shift+P) and select "Developer: Open Extension Logs Folder". The log should be found in the "mindaro.mindaro" directory. Have you also verified that the operator runs successfully w/o Bridge?

  2. Can you give more details about the operator that you're trying to create? Unfortunately Bridge doesn't support debugging CRDs, so I'm wondering if that could be the issue.

  3. Regarding your latest comment, you're correct in that the images from our sample todo-app haven't been updated to meet the Openshift requirements. I'm glad you were able to find another sample that works for you -- we should probably update our sample app.