greenpau / gobgp_exporter

Prometheus Exporter for GoBGP
Apache License 2.0
26 stars 18 forks source link

runtime error: invalid memory address or nil pointer dereference #4

Closed mrueg closed 5 years ago

mrueg commented 5 years ago

Unfortunately gobgp_exporter sometimes seems to segfault.

time="2019-05-16T12:54:26Z" level=error msg="Can't query GoBGP: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: <nil>" source="collect.go:49"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa43bad]

goroutine 1128 [running]:
github.com/ovnworks/gobgp_exporter/pkg/gobgp_exporter.(*RouterNode).GatherMetrics(0x3cce41f0100)
       /var/tmp/portage/app-metrics/gobgp_exporter-1.1.0/work/gobgp_exporter-1.1.0/src/github.com/ovnworks/gobgp_exporter/pkg/gobgp_exporter/collect.go:61 +0x1dd
github.com/ovnworks/gobgp_exporter/pkg/gobgp_exporter.(*RouterNode).Collect(0x3cce41f0100, 0x3cce4095560)
       /var/tmp/portage/app-metrics/gobgp_exporter-1.1.0/work/gobgp_exporter-1.1.0/src/github.com/ovnworks/gobgp_exporter/pkg/gobgp_exporter/router_node.go:122 +0x8c
github.com/ovnworks/gobgp_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
       github.com/prometheus/client_golang/prometheus/registry.go:434 +0x19d
created by github.com/ovnworks/gobgp_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
       github.com/prometheus/client_golang/prometheus/registry.go:445 +0x571
greenpau commented 5 years ago

time="2019-05-16T12:54:26Z" level=error msg="Can't query GoBGP: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: " source="collect.go:49"

@mrueg , does it bring down the exporter's process?

greenpau commented 5 years ago

@mrueg, ... 🤔 ...

It goes to gather metrics https://github.com/ovnworks/gobgp_exporter/blob/master/pkg/gobgp_exporter/router_node.go#L122

Hits "Unavailable" https://github.com/ovnworks/gobgp_exporter/blob/master/pkg/gobgp_exporter/collect.go#L49

Then, it looks we need to amend this code ... to account for "connection error" in the message ...

https://github.com/ovnworks/gobgp_exporter/blob/baaf7fa35f9a4c548bbf8bcf799a61a9c3681a93/pkg/gobgp_exporter/router_node.go#L182-L188

greenpau commented 5 years ago

something like:

    if strings.Contains(err.Error(), "connection error") { 
        return true 
    }

@mrueg , how often does this happen?

greenpau commented 5 years ago

reading https://github.com/grpc/grpc-go/issues/2636

mrueg commented 5 years ago

Yep it brought down the exporter process. Happened once I am aware of.

greenpau commented 5 years ago

@mrueg , it is a quick fix.

mrueg commented 5 years ago

Awesome, can you add what you mentioned in https://github.com/ovnworks/gobgp_exporter/issues/4#issuecomment-493555328 and tag a new release? :)

greenpau commented 5 years ago

@mrueg , done!