search

cbeneke / hcloud-fip-controller

Kubernetes controller to (re-)assign floating IPs on hetzner cloud instances
Apache License 2.0
128 stars 14 forks source link

fip-controller version v0.3.2 not start #38

Closed marcocini closed 4 years ago

marcocini commented 4 years ago

I have tried to install in kubernetes 1.18.2 (the latest). I have used the daemonset.yaml and rbac.yaml that I have found in repository version 0.3.2. I have 3 master nodes with etcd and 5 worker nodes. I have configured 2 floating ip. My config.yml file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fip-controller-config
  namespace: fip-controller
data:
  config.json: |
    {
      "hcloud_floating_ips": [ "xxx.xxx.xxx.158", "xxx.xxx.xxx.121"],
      "node_address_type": "external"
    }
---
apiVersion: v1
kind: Secret
metadata:
  name: fip-controller-secrets
  namespace: fip-controller
stringData:
  HCLOUD_API_TOKEN: <token>

In the log file of all pods I find:

I0419 15:27:29.436033       1 leaderelection.go:235] attempting to acquire leader lease  fip-controller/fip...
E0419 15:27:29.458649       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x11359c0, 0x1c80c70)
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/runtime/runtime.go:48 +0x82
panic(0x11359c0, 0x1c80c70)
    /usr/local/go/src/runtime/panic.go:967 +0x15d
k8s.io/client-go/tools/leaderelection/resourcelock.LeaseSpecToLeaderElectionRecord(...)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/resourcelock/leaselock.go:108
k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).Get(0xc0002f1540, 0xc00003e069, 0x14, 0x43c21c)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/resourcelock/leaselock.go:45 +0x18a
k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryAcquireOrRenew(0xc0002ac900, 0xb9)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:321 +0x13e
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:237 +0x65
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000195cb8)
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000195cb8, 0x12a05f200, 0x3ff3333333333333, 0x1, 0xc0000945a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153 +0xf8
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire(0xc0002ac900, 0x1428d00, 0xc0002a3a00, 0x142f700)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:236 +0x1ee
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc0002ac900, 0x1428d00, 0xc0002a39c0)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:195 +0x6b
k8s.io/client-go/tools/leaderelection.RunOrDie(0x1428d00, 0xc0002a39c0, 0x142f3c0, 0xc0002f1540, 0x6fc23ac00, 0x37e11d600, 0x12a05f200, 0xc0002fc310, 0xc0002fc320, 0x0, ...)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:214 +0x96
github.com/cbeneke/hcloud-fip-controller/internal/app/fipcontroller.(*Controller).RunWithLeaderElection(0xc0002eefa0, 0x1428d00, 0xc0002a39c0)
    /app/internal/app/fipcontroller/leaderelection.go:43 +0x9e
main.main()
    /app/cmd/fip-controller/main.go:52 +0x5d0
time="2020-04-19T15:27:29Z" level=info msg="Stopped leading" func="github.com/sirupsen/logrus.(*Logger).Log" file="/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/logger.go:192"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x10496aa]

goroutine 1 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/runtime/runtime.go:55 +0x105
panic(0x11359c0, 0x1c80c70)
    /usr/local/go/src/runtime/panic.go:967 +0x15d
k8s.io/client-go/tools/leaderelection/resourcelock.LeaseSpecToLeaderElectionRecord(...)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/resourcelock/leaselock.go:108
k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).Get(0xc0002f1540, 0xc00003e069, 0x14, 0x43c21c)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/resourcelock/leaselock.go:45 +0x18a
k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryAcquireOrRenew(0xc0002ac900, 0xb9)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:321 +0x13e
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:237 +0x65
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000195cb8)
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000195cb8, 0x12a05f200, 0x3ff3333333333333, 0x1, 0xc0000945a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153 +0xf8
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire(0xc0002ac900, 0x1428d00, 0xc0002a3a00, 0x142f700)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:236 +0x1ee
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc0002ac900, 0x1428d00, 0xc0002a39c0)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:195 +0x6b
k8s.io/client-go/tools/leaderelection.RunOrDie(0x1428d00, 0xc0002a39c0, 0x142f3c0, 0xc0002f1540, 0x6fc23ac00, 0x37e11d600, 0x12a05f200, 0xc0002fc310, 0xc0002fc320, 0x0, ...)
    /go/pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/leaderelection/leaderelection.go:214 +0x96
github.com/cbeneke/hcloud-fip-controller/internal/app/fipcontroller.(*Controller).RunWithLeaderElection(0xc0002eefa0, 0x1428d00, 0xc0002a39c0)
    /app/internal/app/fipcontroller/leaderelection.go:43 +0x9e
main.main()
    /app/cmd/fip-controller/main.go:52 +0x5d0
phiphi282 commented 4 years ago

Pretty hard to say where the issue is exactly. The leaderelection feature got a few updates in the last few months, including some nil panic fixes.

I have created an update to client-go version v0.18.2. You can test it out by replacing the image in the daemonset with phiphi282/hcloud-fip-controller:v0.3.2

If the update will not fix the error we should at least see better error messages.

I created a pull request for these changes as well #40

cbeneke commented 4 years ago

The issue seems to be resolved by the PR. closing here