search

observatorium / thanos-receive-controller

Kubernetes controller to automatically configure Thanos receive hashrings
Apache License 2.0
95 stars 45 forks source link

Data race in Controller, sending on closed channel #10

Closed metalmatze closed 5 years ago

metalmatze commented 5 years ago

Running go test -v -race ./... throws an error:

=== RUN   TestController/OneHashringNoStatefulSet
==================
WARNING: DATA RACE
Write at 0x00c0001ec730 by goroutine 20:
  runtime.closechan()
      /usr/lib/go/src/runtime/chan.go:334 +0x0
  github.com/observatorium/thanos-receive-controller.(*controller).run()
      /home/metalmatze/src/github.com/observatorium/thanos-receive-controller/main.go:186 +0x550
  github.com/observatorium/thanos-receive-controller.TestController.func1.1()
      /home/metalmatze/src/github.com/observatorium/thanos-receive-controller/main_test.go:154 +0x46

Previous read at 0x00c0001ec730 by goroutine 42:
  runtime.chansend()
      /usr/lib/go/src/runtime/chan.go:142 +0x0
  github.com/observatorium/thanos-receive-controller.(*controller).addWorkItem()
      /home/metalmatze/src/github.com/observatorium/thanos-receive-controller/main.go:194 +0x5a
  github.com/observatorium/thanos-receive-controller.(*controller).run.func3()
      /home/metalmatze/src/github.com/observatorium/thanos-receive-controller/main.go:177 +0x41
  k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate()
      /home/metalmatze/pkg/mod/k8s.io/client-go@v0.0.0-20190615125933-7de88b14dcc8/tools/cache/controller.go:202 +0xa6
  k8s.io/client-go/tools/cache.(*processorListener).run.func1.1()
      /home/metalmatze/pkg/mod/k8s.io/client-go@v0.0.0-20190615125933-7de88b14dcc8/tools/cache/shared_informer.go:605 +0x31d
  k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff()
      /home/metalmatze/pkg/mod/k8s.io/apimachinery@v0.0.0-20190612125636-6a5db36e93ad/pkg/util/wait/wait.go:284 +0x5e
  k8s.io/client-go/tools/cache.(*processorListener).run.func1()
      /home/metalmatze/pkg/mod/k8s.io/client-go@v0.0.0-20190615125933-7de88b14dcc8/tools/cache/shared_informer.go:601 +0xdb
  k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
      /home/metalmatze/pkg/mod/k8s.io/apimachinery@v0.0.0-20190612125636-6a5db36e93ad/pkg/util/wait/wait.go:152 +0x61
  k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /home/metalmatze/pkg/mod/k8s.io/apimachinery@v0.0.0-20190612125636-6a5db36e93ad/pkg/util/wait/wait.go:153 +0x108
  k8s.io/client-go/tools/cache.(*processorListener).run()
      /home/metalmatze/pkg/mod/k8s.io/apimachinery@v0.0.0-20190612125636-6a5db36e93ad/pkg/util/wait/wait.go:88 +0xa9
  k8s.io/client-go/tools/cache.(*processorListener).run-fm()
      /home/metalmatze/pkg/mod/k8s.io/client-go@v0.0.0-20190615125933-7de88b14dcc8/tools/cache/shared_informer.go:593 +0x41
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      /home/metalmatze/pkg/mod/k8s.io/apimachinery@v0.0.0-20190612125636-6a5db36e93ad/pkg/util/wait/wait.go:71 +0x5c

Goroutine 20 (running) created at:
  github.com/observatorium/thanos-receive-controller.TestController.func1()
      /home/metalmatze/src/github.com/observatorium/thanos-receive-controller/main_test.go:153 +0x39a
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:865 +0x163

Goroutine 42 (running) created at:
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start()
      /home/metalmatze/pkg/mod/k8s.io/apimachinery@v0.0.0-20190612125636-6a5db36e93ad/pkg/util/wait/wait.go:69 +0x6f
  k8s.io/client-go/tools/cache.(*sharedProcessor).addListener()
      /home/metalmatze/pkg/mod/k8s.io/client-go@v0.0.0-20190615125933-7de88b14dcc8/tools/cache/shared_informer.go:443 +0x2dd
  k8s.io/client-go/tools/cache.(*sharedIndexInformer).AddEventHandlerWithResyncPeriod()
      /home/metalmatze/pkg/mod/k8s.io/client-go@v0.0.0-20190615125933-7de88b14dcc8/tools/cache/shared_informer.go:391 +0x280
  k8s.io/client-go/tools/cache.(*sharedIndexInformer).AddEventHandler()
      /home/metalmatze/pkg/mod/k8s.io/client-go@v0.0.0-20190615125933-7de88b14dcc8/tools/cache/shared_informer.go:327 +0x69
  github.com/observatorium/thanos-receive-controller.(*controller).run()
      /home/metalmatze/src/github.com/observatorium/thanos-receive-controller/main.go:174 +0x365
  github.com/observatorium/thanos-receive-controller.TestController.func1.1()
      /home/metalmatze/src/github.com/observatorium/thanos-receive-controller/main_test.go:154 +0x46

Something we should fix and then always run go test with -race. :blush:

/cc @brancz @squat