search

tczekajlo / kube-consul-register

a tool to register Kubernetes PODs as Consul Services
Apache License 2.0
105 stars 50 forks source link

Observed a panic "index out of range" on first event. #17

Open webbrandon opened 6 years ago

webbrandon commented 6 years ago

Problem

It seems on the first event emitted when Kubernetes is adding or removing (could be pods, service, deployment configs, not sure yet) when used as a in-cluster Replicaset is crashing kube-consul-register.

Stack Trace:

Observed a panic: "index out of range" (runtime error: index out of range)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:72
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:65
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/panic.go:28
/root/gopath/src/github.com/tczekajlo/kube-consul-register/utils/utils.go:54
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:266
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:180
<autogenerated>:1
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:246
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:279
/root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:141
/usr/local/go/src/runtime/asm_amd64.s:2337
E0127 01:05:59.699263       1 runtime.go:66] Observed a panic: "index out of range" (runtime error: index out of range)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:72
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:65
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/panic.go:28
/root/gopath/src/github.com/tczekajlo/kube-consul-register/utils/utils.go:54
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:266
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:180
<autogenerated>:1
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:246
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:279
/root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:141
/usr/local/go/src/runtime/asm_amd64.s:2337
panic: runtime error: index out of range [recovered]
    panic: runtime error: index out of range [recovered]
    panic: runtime error: index out of range

goroutine 27 [running]:
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58 +0x111
panic(0x138cb80, 0x1e558c0)
    /usr/local/go/src/runtime/panic.go:491 +0x283
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58 +0x111
panic(0x138cb80, 0x1e558c0)
    /usr/local/go/src/runtime/panic.go:491 +0x283
github.com/tczekajlo/kube-consul-register/utils.HasLabel(0xc420373800, 0xc4201d0337, 0x5, 0x7fdcc782f2d8)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/utils/utils.go:54 +0x1f8
github.com/tczekajlo/kube-consul-register/controller/pods.(*Controller).Watch.func3(0x14e5c60, 0xc4205e2000, 0x14e5c60, 0xc4205e22c0)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:266 +0x136
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(0x156f428, 0xc4203ae4c0, 0xc4203ae4b0, 0x14e5c60, 0xc4205e2000, 0x14e5c60, 0xc4205e22c0)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:180 +0x5d
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate(0xc42021ab60, 0x14e5c60, 0xc4205e2000, 0x14e5c60, 0xc4205e22c0)
    <autogenerated>:1 +0x76
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.NewInformer.func1(0x13a7660, 0xc420170560, 0x13a7660, 0xc420170560)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:246 +0x290
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc4200c4630, 0xc4203b5050, 0x0, 0x0, 0x0, 0x0)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420 +0x202
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).processLoop(0xc4202ec230)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x3c
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).(github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x2a
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil.func1(0xc42060eec0)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 +0x5e
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil(0xc420805ec0, 0x3b9aca00, 0x0, 0x1303b01, 0xc420088480)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 +0xa1
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.Until(0xc42060eec0, 0x3b9aca00, 0xc420088480)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 +0x4d
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).Run(0xc4202ec230, 0xc420088480)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x1be
github.com/tczekajlo/kube-consul-register/controller/pods.(*Controller).Watch(0xc420064080)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:279 +0x2e1
main.main.func3(0x7fdcc7a34c28, 0xc420064080)
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:141 +0x31
created by main.main
    /root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:140 +0x580

Environment

Kubernetes v1.7.4 Consul 1.0.3 kube-consul-register v0.1.6

My Configurations

kube-consul-register configurations.
configmap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-consul-register
data:
  consul_address: "consul"
  consul_port: "8500"
  consul_scheme: "http"
  consul_ca_file: ""
  consul_cert_file: ""
  consul_key_file: ""
  consul_insecure_skip_verify: "false"
  consul_token: ""
  consul_timeout: "2s"
  consul_container_name: "consul"
  consul_node_selector: "consul=enabled"
  pod_label_selector: "stage"
  k8s_tag: ""
  register_mode: "single"
  register_source: "endpoints"

replicaset:

apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
  name: kube-consul-register
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-consul-register
    spec:
      containers:
      - name: kube-consul-register
        image: consul-register:0.1.6
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 1
            memory: 300Mi
        args:
        - -logtostderr=true
        - -configmap=default/kube-consul-register

Sample application configurations.
deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: render-stg-test
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 2
  template:
    metadata:
      labels:
        app: render-stg-test
        stage: tag
      annotations: 
        consul.regiser/enabled: "true"
    spec:
      containers:
      - name: render-stg-test
        image: render:0.1.6
        imagePullPolicy: Always
        command:
          - "render"
        env:
        -   name: RENDER_SECRET_ENV
            valueFrom:
                secretKeyRef:
                    name: render-secret-test
                    key: RENDER_SECRET_ENV
        -   name: RENDER_ENV
            value: 'foobar'
      dnsPolicy: ClusterFirst

service:

apiVersion: v1
kind: Service
metadata:
 name: render-test
spec:
 ports:
   - protocol: TCP
     port: 3000
     targetPort: 3000
 selector:
   app: render-stg-test
 type: NodePort
webbrandon commented 6 years ago

Not sure if it matters but this stopped when i added either the label to the service or matchLabel to deployment.

But still not seeing services being added.

tczekajlo commented 6 years ago

@webbrandon thanks for created the issue. I'll take a look at it as soon as I found some spare time.

webbrandon commented 6 years ago

I am trying to get down how to configure it to listen for new service pools and can't seem to get things working. I just updated my configs to run register_mode as single andregister_source to pods and got this error again.

I also tried running several instances side by side with different configs and that "seemed" to have stopped all event watching across each instance.

tczekajlo commented 6 years ago

Indeed, with your configuration, I can reproduce your issue.

@webbrandon let's take a look at the first post. There are a few mistakes in configuration.

First off, register_source: "endpoints" should be register_source: "endpoint" (without "s"). When you use endpoint as a source you should add the annotation to endpoint (kubectl annotate endpoints my-nginx consul.register/enabled=true).

Second off, in such configuration also pod_label_selector: "stage" and the annotation consul.regiser/enabled: "true" in deployment template is useless.

When you fix your configuration everything should be fine.