search

go-graphite / go-carbon

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister
MIT License
804 stars 123 forks source link

[persister] fail to update metric #339

Open TomMizrachi opened 4 years ago

TomMizrachi commented 4 years ago

Has anyone encountered this error? ERROR [persister] fail to update metric

It happens quite a lot as can be seen in the go-carbon log.

Added full log example: [2020-04-06T08:59:33.813Z] ERROR [persister] fail to update metric {"path": "/var/lib/graphite/whisper/mymetric/count.wsp", "error": "Failed to read baseInterval: EOF\ngoroutine 31 [running]:\nruntime/debug.Stack(0xc000358708, 0xbf97e0, 0xc000217360)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper.(*Whisper).UpdateManyForArchive.func1(0xc000358bb8)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper/whisper.go:686 +0x60\npanic(0xbf97e0, 0xc000217360)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper.(*Whisper).getBaseInterval(0xc000d21730, 0xc000a1dd00, 0x402a000000000000)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper/whisper.go:854 +0x137\ngithub.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper.(*Whisper).getPointOffset(0xc000d21730, 0x5e8aef54, 0xc000a1dd00, 0xc)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper/whisper.go:840 +0x39\ngithub.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper.(*Whisper).propagate(0xc000d21730, 0x5e8aef54, 0xc000a1dc00, 0xc000a1dd00, 0x3, 0xc, 0x0)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper/whisper.go:906 +0x2f0\ngithub.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper.(*Whisper).archiveUpdateMany(0xc000d21730, 0xc000a1dc00, 0xc000833968, 0x1, 0x1, 0xc000833968, 0x1)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper/whisper.go:776 +0x695\ngithub.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper.(*Whisper).UpdateManyForArchive(0xc000d21730, 0xc000833968, 0x1, 0x1, 0xffffffffffffffff, 0x0, 0x0)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper/whisper.go:713 +0x40c\ngithub.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper.(*Whisper).UpdateMany(...)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/vendor/github.com/go-graphite/go-whisper/whisper.go:679\ngithub.com/lomik/go-carbon/persister.(*Whisper).updateMany(0xc000274000, 0xc000d21730, 0xc000568d80, 0x79, 0xc000833968, 0x1, 0x1)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/persister/whisper.go:172 +0xbb\ngithub.com/lomik/go-carbon/persister.(*Whisper).store(0xc000274000, 0xc000a40c60, 0x5b)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/persister/whisper.go:323 +0x3af\ngithub.com/lomik/go-carbon/persister.(*Whisper).worker(0xc000274000, 0xc0000a6720)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/persister/whisper.go:402 +0x13d\ngithub.com/lomik/go-carbon/helper.(*Stoppable).StartFunc.func1.1(0xc000217060, 0xc0000a6720, 0xc000274000)\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/helper/stoppable.go:42 +0x30\ncreated by github.com/lomik/go-carbon/helper.(*Stoppable).StartFunc.func1\n\t/home/admin/go-carbon/_vendor/src/github.com/lomik/go-carbon/helper/stoppable.go:41 +0x7d\n"

deniszh commented 4 years ago

Looks similar to https://github.com/grobian/carbonwriter/issues/1 I suspecting corrupted whisper file, can be empty or truncated, because of disk full or something else.

azhiltsov commented 4 years ago

If it's a normal uncompressed whisper file - check it for corruption with something like https://github.com/graphite-project/whisper/blob/master/bin/whisper-dump.py