search

vmware-archive / kubewatch

Watch k8s events and trigger Handlers
Apache License 2.0
2.44k stars 442 forks source link

panic, if any resource is enabled #93

Open kidiyoor opened 6 years ago

kidiyoor commented 6 years ago

Anybody seen this error before ?

~/.kube/config is configured correctly.

$ cat ~/.kubewatch.yaml
handler:
  slack:
    token: ""
    channel: ""
  hipchat:
    token: ""
    room: ""
    url: ""
  mattermost:
    channel: ""
    url: ""
    username: ""
  flock:
    url: http://localhost
resource:
  deployment: false
  replicationcontroller: false
  replicaset: false
  daemonset: false
  services: false
  pod: false
  job: false
  persistentvolume: false
  namespace: true
$ ./kubewatch
INFO[0000] Starting kubewatch controller                 pkg=kubewatch-namespace
ERROR: logging before flag.Parse: E0406 23:03:00.743823   27106 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/lib/google-golang/src/runtime/asm_amd64.s:573
/usr/lib/google-golang/src/runtime/panic.go:505
/usr/lib/google-golang/src/runtime/panic.go:63
/usr/lib/google-golang/src/runtime/signal_unix.go:388
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/kubernetes/clientset.go:271
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/pkg/controller/controller.go:172
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/pager/pager.go:39
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/pager/pager.go:77
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/listwatch.go:97
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/reflector.go:249
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/reflector.go:204
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/reflector.go:203
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/controller.go:122
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54
/usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
/usr/lib/google-golang/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x80 pc=0xfa0f65]

goroutine 56 [running]:
github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x10e8660, 0x1a4be40)
    /usr/lib/google-golang/src/runtime/panic.go:505 +0x229
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/kubernetes.(*Clientset).CoreV1(0x0, 0x0, 0x0)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/kubernetes/clientset.go:271 +0x5
github.com/bitnami-labs/kubewatch/pkg/controller.Start.func11(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1222447, ...)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/pkg/controller/controller.go:172 +0x44
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1(0x1323940, 0xc4200ce018, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/pager/pager.go:39 +0x64
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/pager.(*ListPager).List(0xc4201c39f8, 0x1323940, 0xc4200ce018, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/pager/pager.go:77 +0x105
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List(0xc42023ef80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/listwatch.go:97 +0x147
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc42032e140, 0xc4200ba420, 0x0, 0x0)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/reflector.go:249 +0x208
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/reflector.go:204 +0x33
github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420074f18)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4201c3f18, 0x3b9aca00, 0x0, 0xc400000001, 0xc4200ba420)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420074f18, 0x3b9aca00, 0xc4200ba420)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run(0xc42032e140, 0xc4200ba420)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/reflector.go:203 +0x157
github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm(0xc4200ba420)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/client-go/tools/cache/controller.go:122 +0x34
github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x31
github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc4203140a0, 0xc4202c8140)
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /usr/local/google/home/gauthamvk/work/git/go/src/github.com/bitnami-labs/kubewatch/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62
nabadger commented 6 years ago

Yes - getting this same issue.

nabadger commented 6 years ago

@kidiyoor did you manage to get around this issue?

This almost looks like it can't load the kubeconfig. I'm surprised there aren't more people with this issue

snebel29 commented 6 years ago

I'm getting the same issue, however it feels to me that kubewatch is reading kubeconfig correctly

$ sudo sysdig proc.name=kubewatch
...
...
170545 15:15:03.776075489 2 kubewatch (32067) < stat res=0 path=/home/snebel/.kube/config 
170546 15:15:03.776077937 2 kubewatch (32067) > openat 
170547 15:15:03.776082957 2 kubewatch (32067) < openat fd=5(<f>/home/snebel/.kube/config) dirfd=-100(AT_FDCWD) name=/home/snebel/.kube/config flags=4097(O_RDONLY|O_CLOEXEC) mode=0 
170548 15:15:03.776085231 2 kubewatch (32067) > epoll_ctl 
170549 15:15:03.776085691 2 kubewatch (32067) < epoll_ctl 
170550 15:15:03.776086377 2 kubewatch (32067) > epoll_ctl 
170551 15:15:03.776086677 2 kubewatch (32067) < epoll_ctl 
170552 15:15:03.776089973 2 kubewatch (32067) > fstat fd=5(<f>/home/snebel/.kube/config) 
170553 15:15:03.776090740 2 kubewatch (32067) < fstat res=0 
170554 15:15:03.776096612 2 kubewatch (32067) > read fd=5(<f>/home/snebel/.kube/config) size=60580 
170555 15:15:03.776121373 2 kubewatch (32067) < read res=60068 data=apiVersion: v1.clusters:.- cluster:.    certificate-authority-data: LS0tLS1CRUdJ 
170556 15:15:03.776123490 2 kubewatch (32067) > read fd=5(<f>/home/snebel/.kube/config) size=512 
170557 15:15:03.776124238 2 kubewatch (32067) < read res=0 data= 
170558 15:15:03.776127353 2 kubewatch (32067) > close fd=5(<f>/home/snebel/.kube/config) 
170559 15:15:03.776128478 2 kubewatch (32067) < close res=0 
...
...
snebel29 commented 6 years ago

I've tried into two of the kubernetes clusters that I have with very different versions (1.6.6 vs 1.9.7) and the panic only shows in the 1.9.7 version, feels like if some of the vendor dependencies (apimachinery?) needs to be upgraded for newer API versions

This kubernetes version panic

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"1dc5c66f5dd61da08412a74221ecc79208c2165b", GitTreeState:"clean", BuildDate:"2017-07-14T02:00:46Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.6", GitCommit:"9b635efce81582e1da13b35a7aa539c0ccb32987", GitTreeState:"clean", BuildDate:"2018-08-16T21:33:47Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

This kubernetes version don't panic

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"1dc5c66f5dd61da08412a74221ecc79208c2165b", GitTreeState:"clean", BuildDate:"2017-07-14T02:00:46Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
snebel29 commented 6 years ago

After some digging found that the panic was caused by https://github.com/bitnami-labs/kubewatch/blob/5772afdd620a4b2bfe26f71e2dfb6d6a96b076cb/pkg/utils/k8sutil.go#L47 returning an un-handled error

panic: No Auth Provider found for name "gcp"

Which was ultimately related to client-go auth plugins not being explicitly imported to be available, and making kubewatch to fail when trying to use through https://github.com/bitnami-labs/kubewatch/blob/e8eec939953748c415a40e0dee4c5123eb98679a/pkg/utils/k8sutil.go#L32 And using managed cloud vendors clusters contexts such as gcp, I've provided a proof of concept fix including all available auth plugins in https://github.com/bitnami-labs/kubewatch/pull/140