search

blakelead / couchbase_exporter

Export metrics from Couchbase Server for Prometheus consumption
Other
35 stars 18 forks source link

[xdcr] crash when scraping #12

Closed lconnell closed 5 years ago

lconnell commented 5 years ago 
go version go1.8.3 linux/amd64
-sh-4.2$ go run *go
INFO[0000] No configuration file was found in the working directory /tmp/go-build797253098/command-line-arguments/_obj/exe 
INFO[0000] Couchbase version: 3.0.1-1444-rel-community  
INFO[0000] Community version: true                      
WARN[0000] Version 3.0.1-1444-rel-community may not be supported by this exporter 
ERRO[0000] Could not read file /tmp/go-build797253098/command-line-arguments/_obj/exe/metrics/cluster-default.json 
ERRO[0000] Error during creation of cluster exporter. Cluster metrics won't be scraped 
INFO[0000] Listening at :9191
lconnell commented 5 years ago

Was not executing the binary properly. Closed.

blakelead commented 5 years ago

Thanks for trying the exporter, do not hesitate if you have any questions. And I'm curious to know if the exporter works properly with Couchbase 3 so I would be glad to have your feedback!

lconnell commented 5 years ago

Unfortunately it is crashing with version 3.0 :-(

On Wed, Nov 14, 2018, 4:12 PM Adel Abdelhak <notifications@github.com wrote:

Thanks for trying the exporter, do not hesitate if you have any questions. And I'm curious to know if the exporter works properly with Couchbase 3 so I would be glad to have your feedback!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/blakelead/couchbase_exporter/issues/12#issuecomment-438820543, or mute the thread https://github.com/notifications/unsubscribe-auth/AADsnQYIoOg1JBhTUejKwxB7fOFLKcbLks5uvIdTgaJpZM4Yekwd .

blakelead commented 5 years ago

Could you tell me what you did to run the exporter ? Did you use the latest release 0.5.0 or did you build it from sources ?

LordJeffrey commented 5 years ago

Hi, I work "lconnell" above. We did use the latest version (downloaded today) and then got/built the go code. Here's the last revision of your repo: 896ac95, 13 days ago. When we execute the binary, we get this at first: -sh-4.2$ ./couchbase_exporter INFO[0000] Couchbase version: 3.0.1-1444-rel-community
INFO[0000] Community version: true
WARN[0000] Version 3.0.1-1444-rel-community may not be supported by this exporter INFO[0000] Listening at :9191

If, and only if, I go to port 9191 on that server to see the metrics, etc., we get this: panic: interface conversion: interface {} is string, not float64

goroutine 34 [running]: github.com/blakelead/couchbase_exporter/collector.(XDCRExporter).Collect(0xc420104840, 0xc420061440) /home/jharr200/go/src/github.com/blakelead/couchbase_exporter/collector/xdcr.go:108 +0xe7a github.com/prometheus/client_golang/prometheus.(Registry).Gather.func1() /home/jharr200/go/src/github.com/prometheus/client_golang/prometheus/registry.go:434 +0x113 created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather /home/jharr200/go/src/github.com/prometheus/client_golang/prometheus/registry.go:445 +0x637

Note: we are using couchbase 3.0.1.

Hope that helps? And thanks for considering this.

blakelead commented 5 years ago

Hello @LordJeffrey,

Unfortunately I was not able to run the exporter against Couchbase 3.0.1 (I could not get the version running on my computer).

Could you please try again with launching the exporter the following way:

./couchbase_exporter -scrape.xdcr=false

It will disable XDCR metrics. If this works, it means that Couchbase 3 XDCR API endpoints are not compatible with Couchbase 4 and 5.

LordJeffrey commented 5 years ago

Hey there -- it works! :) not crashing, and i'm seeing metrics. You rock, thanks.

blakelead commented 5 years ago

Glad to hear it :) I'll work on better handling of Couchbase 3 when I'll get the time. I'm keeping this issue alive so I can remember it!

lconnell commented 5 years ago

Thank you very much for your prompt support!

blakelead commented 5 years ago

Reproduced the error in Couchbase 4.5 : 

Nov 22 18:54:49 ******* systemd: Started Couchbase Exporter.
Nov 22 18:54:49 ******* systemd: Starting Couchbase Exporter...
Nov 22 18:54:49 ******* couchbase_exporter: {"level":"info","msg":"Couchbase version: 4.5.1-2844-community","time":"2018-11-22T18:54:49+01:00"}
Nov 22 18:54:49 ******* couchbase_exporter: {"level":"info","msg":"Community version: true","time":"2018-11-22T18:54:49+01:00"}
Nov 22 18:54:49 ******* couchbase_exporter: {"level":"info","msg":"Listening at :9191","time":"2018-11-22T18:54:49+01:00"}
Nov 22 18:54:53 ******* couchbase_exporter: panic: interface conversion: interface {} is string, not float64
***********************************************************************************
Nov 22 18:54:53 ******* systemd: couchbase-exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 22 18:54:53 ******* systemd: Unit couchbase-exporter.service entered failed state.
Nov 22 18:54:53 ******* systemd: couchbase-exporter.service failed.
Nov 22 18:54:53 ******* systemd: couchbase-exporter.service holdoff time over, scheduling restart.

For me it happened when rebalancing, and I found that float metrics could be valued as "undefined"... : 

{"samplesCount":60,"isPersistent":true,"lastTStamp":1542909872440,"interval":1000,"timestamp":[1542909814479,...],"nodeStats":{"************":["undefined","..."]}}
blakelead commented 5 years ago

This fix will allow you to scrape XDCR metrics, do you still have the possibility to test it ?

LordJeffrey commented 5 years ago

I will try to do so on Monday -- and thank you!

On Thursday, November 22, 2018, 6:12:05 PM EST, Adel Abdelhak <notifications@github.com> wrote:

This fix will allow you to scrape XDCR metrics, do you still have the possibility to test it ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

LordJeffrey commented 5 years ago

Hi there. The new version seems to also crash with couch 3.0.1. -sh-4.2$ ./couchbase_exporter INFO[0000] Couchbase version: 3.0.1-1444-rel-community
INFO[0000] Community version: true
WARN[0000] Version 3.0.1-1444-rel-community may not be supported by this exporter INFO[0000] Listening at :9420
panic: interface conversion: interface {} is string, not float64

goroutine 42 [running]: github.com/blakelead/couchbase_exporter/collector.(XDCRExporter).Collect(0xc420061ce0, 0xc42019c060) /home/jharr200/go/src/github.com/blakelead/couchbase_exporter/collector/xdcr.go:108 +0xe7a github.com/prometheus/client_golang/prometheus.(Registry).Gather.func1() /home/jharr200/go/src/github.com/prometheus/client_golang/prometheus/registry.go:434 +0x113 created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather /home/jharr200/go/src/github.com/prometheus/client_golang/prometheus/registry.go:526 +0xf85

I did pull down the new version with the hotfix, as confirmed by "git log". Thanks so much for all the work you're doing on this by the way.

LordJeffrey commented 5 years ago

Sometimes before it crashes, we see any number of ERRO[0021] messages saying " request canceled (Client.Timeout exceeded while awaiting headers) "

blakelead commented 5 years ago

Hi @LordJeffrey,

Thanks to you for using the exporter, feedback like yours keeps me motivated :)

Seeing the logs you posted, it seems that you still execute an earlier version. I don't know how that is, since you have the latest git logs, but the incriminated line:

/home/jharr200/go/src/github.com/blakelead/couchbase_exporter/collector/xdcr.go:108 +0xe7a

corresponds to the old code before the hotfix. I tested again with bad XDCR destination and could not reproduce the error with the latest version of the exporter.

I would suggest you to use the docker image if you can, or to clone the repo anew from tag 0.5.2 or from master.

I hope this will help!

LordJeffrey commented 5 years ago

Finally got time to work on this today :) Ok, seems that when I run "go build" it isn't pulling in the collector/xdcr.go file. It is just pulling it from somewhere, somehow, who knows? I move that xdcr.go file out of the way and it still runs/crashes on 108. So I have to figure out how to make it look in collector and grab that file. I need to do this not using docker because that's the use case -- we're building an rpm for centos 6.0 systems, where docker isn't supported. I dont' expect you to solve my include/caching issues or whatever, but if you reply, I'll gladly look -- as I look for the answer on my side. cheers!

LordJeffrey commented 5 years ago

Actually, I just solved it. I had to delete go/src/* from my "gopath" and then rerun "go get" then "go build". I'm new to go. Anyway, it works now! You can close this. Big thanks.

blakelead commented 5 years ago

If there's one thing I dislike with go is their package management (I don't know if 1.11 is better). I won't forget how you solved this particular problem as I'm sure it will happen to someone some day :)

Anyway I'm glad you got it working!