search

vmware-archive / octant

Highly extensible platform for developers to better understand the complexity of Kubernetes clusters.
https://octant.dev
Apache License 2.0
6.28k stars 488 forks source link

octant crashes #2073

Open ReggieCarey opened 3 years ago

ReggieCarey commented 3 years ago

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.)

Using octant in a terminal window. Accessing from a browser. octant process crashes at some point

What did you expect to happen:

Application should stay up and running until I terminate it.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

LOGS: Forwarding from 127.0.0.1:63559 -> 8888 Forwarding from [::1]:63559 -> 8888 Handling connection for 63559 Handling connection for 63559 Handling connection for 63559 E0224 18:31:41.655296 74909 runtime.go:78] Observed a panic: "close of closed channel" (close of closed channel) goroutine 3389277 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x2796c60, 0x847f950) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86 panic(0x2796c60, 0x847f950) /usr/local/Cellar/go/1.16/libexec/src/runtime/panic.go:965 +0x1b9 github.com/vmware-tanzu/octant/internal/objectstore.(informerFactory).watchErrorHandler.func1(0xc00913c8f0, 0x84a1400, 0xc03aebcfa0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/internal/objectstore/informer.go:57 +0xf8 k8s.io/client-go/tools/cache.(Reflector).Run.func1() /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/client-go/tools/cache/reflector.go:210 +0x73 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0417796e0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc03d64bee0, 0x84a1660, 0xc0214fd680, 0x1, 0xc0455747e0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b k8s.io/client-go/tools/cache.(Reflector).Run(0xc00913c8f0, 0xc0455747e0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/client-go/tools/cache/reflector.go:208 +0x196 k8s.io/apimachinery/pkg/util/wait.(Group).StartWithChannel.func1() /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x2e k8s.io/apimachinery/pkg/util/wait.(Group).Start.func1(0xc0457587c0, 0xc00f18d410) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51 created by k8s.io/apimachinery/pkg/util/wait.(Group).Start /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65 panic: close of closed channel [recovered] panic: close of closed channel

goroutine 3389277 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x109 panic(0x2796c60, 0x847f950) /usr/local/Cellar/go/1.16/libexec/src/runtime/panic.go:965 +0x1b9 github.com/vmware-tanzu/octant/internal/objectstore.(informerFactory).watchErrorHandler.func1(0xc00913c8f0, 0x84a1400, 0xc03aebcfa0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/internal/objectstore/informer.go:57 +0xf8 k8s.io/client-go/tools/cache.(Reflector).Run.func1() /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/client-go/tools/cache/reflector.go:210 +0x73 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0417796e0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/uti(0xc03d64bee0, 0x84a1660, 0xc0214fd680, 0x1, 0xc0455747e0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b k8s.io/client-go/tools/cache.(Reflector).Run(0xc00913c8f0, 0xc0455747e0) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/client-go/tools/cache/reflector.go:208 +0x196 k8s.io/apimachinery/pkg/util/wait.(Group).StartWithChannel.func1() /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x2e k8s.io/apimachinery/pkg/util/wait.(Group).Start.func1(0xc0457587c0, 0xc00f18d410) /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51 created by k8s.io/apimachinery/pkg/util/wait.(Group).Start /private/tmp/octant-20210224-87142-1ovpw0z/src/github.com/vmware-tanzu/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

[Process completed]

Environment:

wwitzel3 commented 3 years ago

Thank you, this is very helpful.

The stopCh is already closed when the error handler here https://github.com/vmware-tanzu/octant/blob/master/internal/objectstore/informer.go#L57 attempts to close it again.

chkohner commented 2 years ago

Fwiw, I have a slightly different looking stack trace but the same results (though now it won't start, previously it ran for at least a while):

2022-04-12T09:06:46.579-0700    WARN    plugin/manager.go:405   Unable to add C:\Users\chkohner\AppData\Local\octant\plugins to the plugin watcher. Error: GetFileAttributes: The system cannot find the path specified.

github.com/vmware-tanzu/octant/pkg/plugin.(*Manager).watchPluginFiles
        /__w/octant/octant/pkg/plugin/manager.go:405
2022-04-12T09:06:47.743-0700    INFO    cluster/cluster.go:126  removing cluster client temporary directory     {"component": "cluster client"}
2022-04-12T09:06:48.699-0700    INFO    config/dash.go:182      updated kube config context     {"new-kube-context": ""}
2022-04-12T09:06:48.699-0700    INFO    cluster/cluster.go:126  removing cluster client temporary directory     {"component": "cluster client"}
E0412 09:06:48.701068  103664 runtime.go:78] Observed a panic: "close of closed channel" (close of closed channel)
goroutine 147 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1c3b120, 0x8e7dc60})
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x33ac72})
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x1c3b120, 0x8e7dc60})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0004d5380)
        /__w/octant/octant/vendor/k8s.io/client-go/tools/cache/shared_informer.go:752 +0x287
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88
panic: close of closed channel [recovered]
        panic: close of closed channel

goroutine 147 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x33ac72})
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x1c3b120, 0x8e7dc60})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0004d5380)
        /__w/octant/octant/vendor/k8s.io/client-go/tools/cache/shared_informer.go:752 +0x287
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88