search

Griesbacher / nagflux

A connector which copies performancedata from Nagios / Icinga(2) / Naemon to InfluxDB
GNU General Public License v2.0
65 stars 30 forks source link

fix race condition when pausing gearman worker #50

Closed sni closed 4 years ago

sni commented 4 years ago

this PR fixes the issue

fatal error: sync: unlock of unlocked mutex

in collector/modGearman/gearmanWorker.go:133

Seems like nagflux tried to unlock the Lock, when it never had Locked the worker in first place. Checking the global state fixes the issue.

Before:

  1. pause = false
  2. globalPause := config.IsAnyTargetOnPause -> true
  3. try to unlock
  4. fatal

So if nagflux checks the globalpause, it would had run the Lock, which is probably the thing we want here.

the relevant log entries are:

2019-11-14 12:44:43 Warn: Post https://127.0.0.1:8086/write?precision=ms&db=nagflux&u=omdadmin&p=omd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019-11-14 12:44:43 Info: Is InfluxDB(nagflux) running: true
2019-11-14 12:44:44 Warn: Post https://127.0.0.1:8086/write?precision=ms&db=nagflux&u=omdadmin&p=omd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019-11-14 12:44:50 Info: Is InfluxDB(nagflux) running: false
fatal error: sync: unlock of unlocked mutex

goroutine 12 [running]:
runtime.throw(0x800cd3, 0x1e)
    /opt/projects/omd/rpm.topdir/BUILD/omd/packages/go-1.12/go-1.12.3/src/runtime/panic.go:617 +0x72 fp=0xc000203eb8 sp=0xc000203e88 pc=0x42daa2
sync.throw(0x800cd3, 0x1e)
    /opt/projects/omd/rpm.topdir/BUILD/omd/packages/go-1.12/go-1.12.3/src/runtime/panic.go:603 +0x35 fp=0xc000203ed8 sp=0xc000203eb8 pc=0x42da25
sync.(*Mutex).Unlock(0xc00005ef60)
    /opt/projects/omd/rpm.topdir/BUILD/omd/packages/go-1.12/go-1.12.3/src/sync/mutex.go:184 +0xc1 fp=0xc000203f00 sp=0xc000203ed8 pc=0x46f001
github.com/griesbacher/nagflux/collector/modGearman.GearmanWorker.handlePause(0xc00001c300, 0xc000134450, 0xc00005ab80, 0xc00000ee40, 0xc00005ef60, 0xc000134420, 0xc000123890, 0x7)
    /opt/projects/omd/rpm.topdir/BUILD/omd/packages/nagflux/go/src/github.com/griesbacher/nagflux/collector/modGearman/gearmanWorker.go:133 +0x120 fp=0xc000203fa0 sp=0xc000203f00 pc=0x717550
runtime.goexit()
    /opt/projects/omd/rpm.topdir/BUILD/omd/packages/go-1.12/go-1.12.3/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000203fa8 sp=0xc000203fa0 pc=0x459ff1
created by github.com/griesbacher/nagflux/collector/modGearman.NewGearmanWorker
    /opt/projects/omd/rpm.topdir/BUILD/omd/packages/nagflux/go/src/github.com/griesbacher/nagflux/collector/modGearman/gearmanWorker.go:52 +0x510

Signed-off-by: Sven Nierlein sven@nierlein.de

Griesbacher commented 4 years ago

Great, thank you!