CastawayLabs / cachet-monitor

Distributed monitoring plugin for CachetHQ
https://castawaylabs.github.io/cachet-monitor/
MIT License
439 stars 126 forks source link

Panic when creating DNS failed event #76

Open ericfri opened 6 years ago

ericfri commented 6 years ago

I've got a DNS check setup and when testing it out the service crashes when creating a new event for the failed DNS check.

Aug 06 16:43:05 cachet cachet[3214]: time="2017-08-06T16:43:05Z" level=warning msg="dns is now saturated" Aug 06 16:43:05 cachet cachet[3214]: time="2017-08-06T16:43:05Z" level=info msg="monitor down 100.00%/80.00%" monitor=dns time="06/08/2017 16:43:05 PST" Aug 06 16:43:05 cachet cachet[3214]: panic: runtime error: invalid memory address or nil pointer dereference [recovered] Aug 06 16:43:05 cachet cachet[3214]: panic: runtime error: invalid memory address or nil pointer dereference Aug 06 16:43:05 cachet cachet[3214]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x6a6839] Aug 06 16:43:05 cachet cachet[3214]: goroutine 21 [running]: Aug 06 16:43:05 cachet cachet[3214]: panic(0x7a3340, 0xc42000e050) Aug 06 16:43:05 cachet cachet[3214]: /usr/local/Cellar/go/1.7.5/libexec/src/runtime/panic.go:500 +0x1a1 Aug 06 16:43:05 cachet cachet[3214]: text/template.errRecover(0xc42014bb90) Aug 06 16:43:05 cachet cachet[3214]: /usr/local/Cellar/go/1.7.5/libexec/src/text/template/exec.go:140 +0x2ad Aug 06 16:43:05 cachet cachet[3214]: panic(0x7a3340, 0xc42000e050) Aug 06 16:43:05 cachet cachet[3214]: /usr/local/Cellar/go/1.7.5/libexec/src/runtime/panic.go:458 +0x243 Aug 06 16:43:05 cachet cachet[3214]: text/template.(Template).execute(0x0, 0x99a820, 0xc42015ff10, 0x7a1560, 0xc4201e50b0, 0x0, 0x0) Aug 06 16:43:05 cachet cachet[3214]: /usr/local/Cellar/go/1.7.5/libexec/src/text/template/exec.go:186 +0x1b9 Aug 06 16:43:05 cachet cachet[3214]: text/template.(Template).Execute(0x0, 0x99a820, 0xc42015ff10, 0x7a1560, 0xc4201e50b0, 0xc4201e50b0, 0xc42014bc18) Aug 06 16:43:05 cachet cachet[3214]: /usr/local/Cellar/go/1.7.5/libexec/src/text/template/exec.go:175 +0x53 Aug 06 16:43:05 cachet cachet[3214]: github.com/castawaylabs/cachet-monitor.(MessageTemplate).exec(0xc420075bd8, 0x0, 0x7a1560, 0xc4201e50b0, 0x40d99b, 0x9b9f70) Aug 06 16:43:05 cachet cachet[3214]: /Users/m/p/go/src/github.com/castawaylabs/cachet-monitor/template.go:47 +0x6e Aug 06 16:43:05 cachet cachet[3214]: github.com/castawaylabs/cachet-monitor.(MessageTemplate).Exec(0xc420075bd8, 0x7a1560, 0xc4201e50b0, 0xc42014bd98, 0xc4201d5720, 0x2, 0x17) Aug 06 16:43:05 cachet cachet[3214]: /Users/m/p/go/src/github.com/castawaylabs/cachet-monitor/template.go:41 +0x4c Aug 06 16:43:05 cachet cachet[3214]: github.com/castawaylabs/cachet-monitor.(AbstractMonitor).AnalyseData(0xc420075b80) Aug 06 16:43:05 cachet cachet[3214]: /Users/m/p/go/src/github.com/castawaylabs/cachet-monitor/monitor.go:210 +0x828 Aug 06 16:43:05 cachet cachet[3214]: github.com/castawaylabs/cachet-monitor.(AbstractMonitor).tick(0xc420075b80, 0x9a29e0, 0xc420075b80) Aug 06 16:43:05 cachet cachet[3214]: /Users/m/p/go/src/github.com/castawaylabs/cachet-monitor/monitor.go:161 +0x170 Aug 06 16:43:05 cachet cachet[3214]: github.com/castawaylabs/cachet-monitor.(*AbstractMonitor).ClockStart(0xc420075b80, 0xc42000c780, 0x9a29e0, 0xc420075b80, 0xc42015a5b0) Aug 06 16:43:05 cachet cachet[3214]: /Users/m/p/go/src/github.com/castawaylabs/cachet-monitor/monitor.go:125 +0x196 Aug 06 16:43:05 cachet cachet[3214]: created by main.main Aug 06 16:43:05 cachet cachet[3214]: /Users/m/p/go/src/github.com/castawaylabs/cachet-monitor/cli/main.go:96 +0x873 Aug 06 16:43:05 cachet systemd[1]: cachet-monitor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Aug 06 16:43:05 cachet systemd[1]: cachet-monitor.service: Unit entered failed state. Aug 06 16:43:05 cachet systemd[1]: cachet-monitor.service: Failed with result 'exit-code'. Aug 06 16:43:05 cachet systemd[1]: cachet-monitor.service: Service hold-off time over, scheduling restart. Aug 06 16:43:05 cachet systemd[1]: Stopped Cachet Monitor. -- Subject: Unit cachet-monitor.service has finished shutting down -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit cachet-monitor.service has finished shutting down. Aug 06 16:43:05 cachet systemd[1]: cachet-monitor.service: Failed to reset devices.list: Operation not permitted Aug 06 16:43:05 cachet systemd[1]: Started Cachet Monitor. -- Subject: Unit cachet-monitor.service has finished start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit cachet-monitor.service has finished starting up. -- -- The start-up result is done.

The config file:

  • name: dns target: targetdomain.ca question: mx type: dns component_id: 4 metric_id: 2 interval: 5 timeout: 1 threshold: 80 dns: ns1.nameserver.ca:53 answers:
  • exact: 3 targetdomain.ca.

Both the component and metric are valid. If I get rid of the threshold line, the metric works just fine, but as soon as I added the threshold to the component gets updated the crash happens.

axnsan12 commented 6 years ago

Same problem as #79?