search

veepee-oss / influxdb-relay

Service to replicate InfluxDB data for high availability.
MIT License
198 stars 47 forks source link

Http panic error while serving requests #34

Open kaugh opened 4 years ago

kaugh commented 4 years ago

~200 servers with telegraf-1.12.2-1.x86_64 with a fairly standard configuration Influxdb-relay (3.0.1 version) bulded on go-1.12.10 Influxdb-1.7.8 on the same host

Config

[[http]]
name = "influxdb01-http-relay"
bind-addr = "0.0.0.0:9096"

[[http.output]]
name = "influxdb01"
location = "http://server01:8086/"
endpoints = {write="/write", write_prom="/api/v1/prom/write", ping="/ping", query="/query"}
timeout = "10s"

Every few minutes, errors appear in the log

Nov 5 22:02:10 server01 influxdb-relay: 2019/11/05 22:02:10 http: panic serving serverXXX:37880: runtime error: slice bounds out of range Nov 5 22:02:10 server01 influxdb-relay: goroutine 28766 [running]: Nov 5 22:02:10 server01 influxdb-relay: net/http.(conn).serve.func1(0xc001411400) Nov 5 22:02:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:1769 +0x139 Nov 5 22:02:10 server01 influxdb-relay: panic(0x719160, 0xa10af0) Nov 5 22:02:10 server01 influxdb-relay: /usr/local/go/src/runtime/panic.go:522 +0x1b5 Nov 5 22:02:10 server01 influxdb-relay: bytes.Replace(0xc00032e71d, 0x18, 0x96e3, 0x9e5584, 0x2, 0x2, 0x9e5583, 0x1, 0x1, 0x1, ...) Nov 5 22:02:10 server01 influxdb-relay: /usr/local/go/src/bytes/bytes.go:788 +0x5bf Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.unescapeTag(0xc00032e71d, 0x18, 0x96e3, 0xc00032e718, 0x4, 0x96e8) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1256 +0x17e Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.walkTags(0xc00032e6f1, 0x44, 0x970f, 0xc000bbd778) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1517 +0x229 Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.parseTags(0xc00032e6f1, 0x44, 0x970f, 0xc00032e6f1, 0xa, 0x970f) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1560 +0x12b Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.(point).Tags(0xc001edcd20, 0xc00032e6f1, 0xa, 0x970f) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1469 +0x5c Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/relay.(httpBackend).validateRegexps(0xc00009a180, 0xc000429c00, 0xb0, 0xb1, 0x1a, 0x0) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/relay/http.go:338 +0x16c Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/relay.(HTTP).handleStandard(0xc00007b1e0, 0x7ee0e0, 0xc000101960, 0xc00119b400, 0xbf688fac846acaa4, 0x22fcf3e61efad, 0xa1d900) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/relay/http_handlers.go:304 +0x6c2 Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/relay.(HTTP).bodyMiddleWare.func1(0xc00007b1e0, 0x7ee0e0, 0xc000101960, 0xc00119b400, 0xbf688fac846acaa4, 0x22fcf3e61efad, 0xa1d900) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:42 +0xfa Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/relay.(HTTP).queryMiddleWare.func1(0xc00007b1e0, 0x7ee0e0, 0xc000101960, 0xc00119b400, 0xbf688fac846acaa4, 0x22fcf3e61efad, 0xa1d900) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:60 +0x152 Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/relay.(HTTP).logMiddleWare.func1(0xc00007b1e0, 0x7ee0e0, 0xc000101960, 0xc00119b400, 0xbf688fac846acaa4, 0x22fcf3e61efad, 0xa1d900) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:23 +0x81 Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/relay.(HTTP).rateMiddleware.func1(0xc00007b1e0, 0x7ee0e0, 0xc000101960, 0xc00119b400, 0xbf688fac846acaa4, 0x22fcf3e61efad, 0xa1d900) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:72 +0x10d Nov 5 22:02:10 server01 influxdb-relay: github.com/veepee-moc/influxdb-relay/relay.(HTTP).ServeHTTP(0xc00007b1e0, 0x7ee0e0, 0xc000101960, 0xc00119b400) Nov 5 22:02:10 server01 influxdb-relay: /root/go/src/github.com/veepee-moc/influxdb-relay/relay/http.go:197 +0xd7 Nov 5 22:02:10 server01 influxdb-relay: net/http.serverHandler.ServeHTTP(0xc000194000, 0x7ee0e0, 0xc000101960, 0xc00119b400) Nov 5 22:02:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:2774 +0xa8 Nov 5 22:02:10 server01 influxdb-relay: net/http.(conn).serve(0xc001411400, 0x7eea60, 0xc000aa7780) Nov 5 22:02:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:1878 +0x851 Nov 5 22:02:10 server01 influxdb-relay: created by net/http.(*Server).Serve Nov 5 22:02:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:2884 +0x2f4

What could be the reason for this?

damoun commented 4 years ago

Hi @kaugh,

Thanks for your report. Do you have the HTTP request that trigger this panic ? What do you use as telegraf plugins ?

If I follow the backtrace, there is an issue on the Tags.

damoun commented 4 years ago

Since our influxdb import was lock on v1.5.4, I upgraded it to v1.7.9. Could you try the branch upgrade-vendor and see if your issue is fixed ?

damoun commented 4 years ago

We upgraded influxdb dependency into master. Can you try our latest release/docker image ?

kaugh commented 4 years ago

I've tried on latest release. There are 2 types of errors now

[root@server01 ~]# cat /var/log/influxdb-relay/influxdb-relay.log | grep "panic serving" ... Dec 18 17:16:10 server01 influxdb-relay: 2019/12/18 17:16:10 http: panic serving serverXXX:59622: runtime error: index out of range Dec 18 17:24:10 server01 influxdb-relay: 2019/12/18 17:24:10 http: panic serving serverXXX:58848: runtime error: slice bounds out of range Dec 18 17:28:10 server01 influxdb-relay: 2019/12/18 17:28:10 http: panic serving serverXXX:35970: runtime error: slice bounds out of range Dec 18 17:41:10 server01 influxdb-relay: 2019/12/18 17:41:10 http: panic serving serverXXX:45220: runtime error: slice bounds out of range Dec 18 17:57:10 server01 influxdb-relay: 2019/12/18 17:57:10 http: panic serving serverXXX:37106: runtime error: index out of range Dec 18 18:05:10 server01 influxdb-relay: 2019/12/18 18:05:10 http: panic serving serverXXX:32862: runtime error: index out of range Dec 18 18:14:10 server01 influxdb-relay: 2019/12/18 18:14:10 http: panic serving serverXXX:33800: runtime error: index out of range Dec 18 18:33:10 server01 influxdb-relay: 2019/12/18 18:33:10 http: panic serving serverXXX:39294: runtime error: slice bounds out of range Dec 18 18:40:10 server01 influxdb-relay: 2019/12/18 18:40:10 http: panic serving serverXXX:34648: runtime error: slice bounds out of range Dec 18 19:12:10 server01 influxdb-relay: 2019/12/18 19:12:10 http: panic serving serverXXX:60252: runtime error: index out of range Dec 18 19:26:10 server01 influxdb-relay: 2019/12/18 19:26:10 http: panic serving serverXXX:51200: runtime error: slice bounds out of range Dec 18 19:41:10 server01 influxdb-relay: 2019/12/18 19:41:10 http: panic serving serverXXX:59512: runtime error: index out of range Dec 18 19:45:10 server01 influxdb-relay: 2019/12/18 19:45:10 http: panic serving serverXXX:53840: runtime error: slice bounds out of range Dec 18 20:15:10 server01 influxdb-relay: 2019/12/18 20:15:10 http: panic serving serverXXX:59572: runtime error: index out of range Dec 18 20:15:10 server01 influxdb-relay: 2019/12/18 20:15:10 http: panic serving serverXXX:36276: runtime error: slice bounds out of range Dec 18 20:28:10 server01 influxdb-relay: 2019/12/18 20:28:10 http: panic serving serverXXX:45478: runtime error: index out of range Dec 18 20:40:10 server01 influxdb-relay: 2019/12/18 20:40:10 http: panic serving serverXXX:35444: runtime error: index out of range Dec 18 20:55:10 server01 influxdb-relay: 2019/12/18 20:55:10 http: panic serving serverXXX:55268: runtime error: index out of range Dec 18 20:56:10 server01 influxdb-relay: 2019/12/18 20:56:10 http: panic serving serverXXX:48740: runtime error: index out of range

ServerXXX - different servers

index out of range

Dec 18 20:40:10 server01 influxdb-relay: 2019/12/18 20:40:10 http: panic serving serverXXX:35444: runtime error: index out of range Dec 18 20:40:10 server01 influxdb-relay: goroutine 277 [running]: Dec 18 20:40:10 server01 influxdb-relay: net/http.(conn).serve.func1(0xc000199400) Dec 18 20:40:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:1769 +0x139 Dec 18 20:40:10 server01 influxdb-relay: panic(0x71a2a0, 0xa11ae0) Dec 18 20:40:10 server01 influxdb-relay: /usr/local/go/src/runtime/panic.go:522 +0x1b5 Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.parseTags.func1(0xc0015b81e2, 0x5, 0x37c1e, 0xc0015b81e8, 0x2, 0x37c18, 0x2) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1613 +0xc6 Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.walkTags(0xc0015b81b1, 0x4e, 0x37c4f, 0xc0004af760) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1555 +0x2ba Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.parseTags(0xc0015b81b1, 0x4e, 0x37c4f, 0x0, 0x0, 0x4, 0xc0015b81b1, 0xa, 0x37c4f) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1612 +0x13a Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.(point).Tags(0xc001653e00, 0xc0015b81b1, 0xa, 0x37c4f) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1499 +0x71 Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(httpBackend).validateRegexps(0xc00009c180, 0xc000b68000, 0x146, 0x147, 0x1a, 0x0) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http.go:338 +0x16c Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).handleStandard(0xc00007ef70, 0x7ef2c0, 0xc000102460, 0xc000ef4600, 0xbf76b6fe8343a9f5, 0x598e7575dc66, 0xa1e900) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_handlers.go:304 +0x6c2 Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).bodyMiddleWare.func1(0xc00007ef70, 0x7ef2c0, 0xc000102460, 0xc000ef4600, 0xbf76b6fe8343a9f5, 0x598e7575dc66, 0xa1e900) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:42 +0xfa Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).queryMiddleWare.func1(0xc00007ef70, 0x7ef2c0, 0xc000102460, 0xc000ef4600, 0xbf76b6fe8343a9f5, 0x598e7575dc66, 0xa1e900) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:60 +0x152 Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).logMiddleWare.func1(0xc00007ef70, 0x7ef2c0, 0xc000102460, 0xc000ef4600, 0xbf76b6fe8343a9f5, 0x598e7575dc66, 0xa1e900) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:23 +0x81 Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).rateMiddleware.func1(0xc00007ef70, 0x7ef2c0, 0xc000102460, 0xc000ef4600, 0xbf76b6fe8343a9f5, 0x598e7575dc66, 0xa1e900) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:72 +0x10d Dec 18 20:40:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).ServeHTTP(0xc00007ef70, 0x7ef2c0, 0xc000102460, 0xc000ef4600) Dec 18 20:40:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http.go:197 +0xd7 Dec 18 20:40:10 server01 influxdb-relay: net/http.serverHandler.ServeHTTP(0xc000194000, 0x7ef2c0, 0xc000102460, 0xc000ef4600) Dec 18 20:40:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:2774 +0xa8 Dec 18 20:40:10 server01 influxdb-relay: net/http.(conn).serve(0xc000199400, 0x7efc40, 0xc00038f0c0) Dec 18 20:40:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:1878 +0x851 Dec 18 20:40:10 server01 influxdb-relay: created by net/http.(*Server).Serve Dec 18 20:40:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:2884 +0x2f4

slice bounds out of range

Dec 18 19:45:10 server01 influxdb-relay: 2019/12/18 19:45:10 http: panic serving serverXXX:53840: runtime error: slice bounds out of range Dec 18 19:45:10 server01 influxdb-relay: goroutine 2215927 [running]: Dec 18 19:45:10 server01 influxdb-relay: net/http.(conn).serve.func1(0xc00017b040) Dec 18 19:45:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:1769 +0x139 Dec 18 19:45:10 server01 influxdb-relay: panic(0x71a2a0, 0xa11af0) Dec 18 19:45:10 server01 influxdb-relay: /usr/local/go/src/runtime/panic.go:522 +0x1b5 Dec 18 19:45:10 server01 influxdb-relay: bytes.Replace(0xc000122e45, 0x13, 0x2044, 0x9e6589, 0x2, 0x2, 0x9e6588, 0x1, 0x1, 0x1, ...) Dec 18 19:45:10 server01 influxdb-relay: /usr/local/go/src/bytes/bytes.go:788 +0x5bf Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.unescapeTag(0xc000122e45, 0x13, 0x2044, 0xc000122e3e, 0x6, 0x204b) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1279 +0x17e Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.walkTags(0xc000122e33, 0x53, 0x2056, 0xc001ab3760) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1551 +0x229 Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.parseTags(0xc000122e33, 0x53, 0x2056, 0x0, 0x0, 0x5, 0xc000122e33, 0xa, 0x2056) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1612 +0x13a Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models.(point).Tags(0xc000a8ce10, 0xc000122e33, 0xa, 0x2056) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1499 +0x71 Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(httpBackend).validateRegexps(0xc00009c180, 0xc000902800, 0x71, 0x72, 0x1a, 0x0) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http.go:338 +0x16c Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).handleStandard(0xc00007ef70, 0x7ef2c0, 0xc0007bbea0, 0xc000f67100, 0xbf76b3c580b58405, 0x568e1b93ce94, 0xa1e900) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_handlers.go:304 +0x6c2 Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).bodyMiddleWare.func1(0xc00007ef70, 0x7ef2c0, 0xc0007bbea0, 0xc000f67100, 0xbf76b3c580b58405, 0x568e1b93ce94, 0xa1e900) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:42 +0xfa Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).queryMiddleWare.func1(0xc00007ef70, 0x7ef2c0, 0xc0007bbea0, 0xc000f67100, 0xbf76b3c580b58405, 0x568e1b93ce94, 0xa1e900) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:60 +0x152 Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).logMiddleWare.func1(0xc00007ef70, 0x7ef2c0, 0xc0007bbea0, 0xc000f67100, 0xbf76b3c580b58405, 0x568e1b93ce94, 0xa1e900) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:23 +0x81 Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).rateMiddleware.func1(0xc00007ef70, 0x7ef2c0, 0xc0007bbea0, 0xc000f67100, 0xbf76b3c580b58405, 0x568e1b93ce94, 0xa1e900) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http_middlewares.go:72 +0x10d Dec 18 19:45:10 server01 influxdb-relay: github.com/strike-team/influxdb-relay/relay.(HTTP).ServeHTTP(0xc00007ef70, 0x7ef2c0, 0xc0007bbea0, 0xc000f67100) Dec 18 19:45:10 server01 influxdb-relay: /root/work/src/github.com/strike-team/influxdb-relay/relay/http.go:197 +0xd7 Dec 18 19:45:10 server01 influxdb-relay: net/http.serverHandler.ServeHTTP(0xc000194000, 0x7ef2c0, 0xc0007bbea0, 0xc000f67100) Dec 18 19:45:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:2774 +0xa8 Dec 18 19:45:10 server01 influxdb-relay: net/http.(conn).serve(0xc00017b040, 0x7efc40, 0xc000056e80) Dec 18 19:45:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:1878 +0x851 Dec 18 19:45:10 server01 influxdb-relay: created by net/http.(*Server).Serve Dec 18 19:45:10 server01 influxdb-relay: /usr/local/go/src/net/http/server.go:2884 +0x2f4

[root@build-server ~]# go version
go version go1.12.10 linux/amd64
[root@server01 ~]# influxdb-relay --version
influxdb-relay version 3.0.2
phatad commented 4 years ago

Hi @damoun, I have the same problems

2019/12/20 10:59:00 http: panic serving 10.11.10.101:58646: runtime error: slice bounds out of range goroutine 30310341 [running]: net/http.(conn).serve.func1(0xc0039ce780) /usr/local/go/src/net/http/server.go:1769 +0x139 panic(0x719140, 0xa10af0) /usr/local/go/src/runtime/panic.go:522 +0x1b5 bytes.Replace(0xc005c263d0, 0x25, 0x1da30, 0x9e5584, 0x2, 0x2, 0x9e5583, 0x1, 0x1, 0x2, ...) /usr/local/go/src/bytes/bytes.go:788 +0x5bf github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.unescapeTag(0xc005c263d0, 0x25, 0x1da30, 0xc005c263cb, 0x4, 0x1da35) /root/go_project/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1256 +0x17e github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.walkTags(0xc005c26397, 0x5e, 0x1da69, 0xc00510b778) /root/go_project/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1517 +0x229 github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.parseTags(0xc005c26397, 0x5e, 0x1da69, 0xc005c26397, 0xa, 0x1da69) /root/go_project/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1560 +0x12b github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models.(point).Tags(0xc004bc1d10, 0xc005c26397, 0xa, 0x1da69) /root/go_project/src/github.com/veepee-moc/influxdb-relay/vendor/github.com/influxdata/influxdb/models/points.go:1469 +0x5c github.com/veepee-moc/influxdb-relay/relay.(httpBackend).validateRegexps(0xc0000ac180, 0xc005c44000, 0x111, 0x112, 0x0, 0x0) /root/go_project/src/github.com/veepee-moc/influxdb-relay/relay/http.go:338 +0x16c github.com/veepee-moc/influxdb-relay/relay.(HTTP).handleStandard(0xc00008eea0, 0x7edfe0, 0xc000112a80, 0xc005eb9f00, 0xbf772fa10da7feda, 0x47498dcc1886e, 0xa1d900) /root/go_project/src/github.com/veepee-moc/influxdb-relay/relay/http_handlers.go:304 +0x6c2 github.com/veepee-moc/influxdb-relay/relay.(HTTP).bodyMiddleWare.func1(0xc00008eea0, 0x7edfe0, 0xc000112a80, 0xc005eb9f00, 0xbf772fa10da7feda, 0x47498dcc1886e, 0xa1d900) /root/go_project/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:42 +0xfa github.com/veepee-moc/influxdb-relay/relay.(HTTP).queryMiddleWare.func1(0xc00008eea0, 0x7edfe0, 0xc000112a80, 0xc005eb9f00, 0xbf772fa10da7feda, 0x47498dcc1886e, 0xa1d900) /root/go_project/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:60 +0x152 github.com/veepee-moc/influxdb-relay/relay.(HTTP).logMiddleWare.func1(0xc00008eea0, 0x7edfe0, 0xc000112a80, 0xc005eb9f00, 0xbf772fa10da7feda, 0x47498dcc1886e, 0xa1d900) /root/go_project/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:23 +0x81 github.com/veepee-moc/influxdb-relay/relay.(HTTP).rateMiddleware.func1(0xc00008eea0, 0x7edfe0, 0xc000112a80, 0xc005eb9f00, 0xbf772fa10da7feda, 0x47498dcc1886e, 0xa1d900) /root/go_project/src/github.com/veepee-moc/influxdb-relay/relay/http_middlewares.go:72 +0x10d github.com/veepee-moc/influxdb-relay/relay.(HTTP).ServeHTTP(0xc00008eea0, 0x7edfe0, 0xc000112a80, 0xc005eb9f00) /root/go_project/src/github.com/veepee-moc/influxdb-relay/relay/http.go:197 +0xd7 net/http.serverHandler.ServeHTTP(0xc0001ac000, 0x7edfe0, 0xc000112a80, 0xc005eb9f00) /usr/local/go/src/net/http/server.go:2774 +0xa8 net/http.(conn).serve(0xc0039ce780, 0x7ee960, 0xc0023a7900) /usr/local/go/src/net/http/server.go:1878 +0x851 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2884 +0x2f4

And the nginx output error log like this: 2019/12/20 10:59:00 [error] 103861#103861: *22357974 upstream prematurely closed connection while reading response header from upstream, client: 10.11.10.113, server: , request: "POST /write?db=metric_database HTTP/1.1", upstream: "http://10.11.10.101:7086/write?db=metric_database", host: "10.11.10.101:8086" All the metrics will be dropped at 10:59:00 when I display on grafana

influxdb-relay -version
influxdb-relay version 3.0.1
go version go1.12.7 linux/amd64

And then I install influxdb-relay version 3.0.2 Output error: 2019/12/20 16:47:00 http: panic serving 10.11.10.101:49660: runtime error: index out of range

Do you have any advices for me? Thank you

rockyluke commented 4 years ago

Thanks for you report. Due to Christmas holidays we will be slower to answer but we will keep you informed soon.

🤶 🎅

kaugh commented 4 years ago

Hello. Is there any news about issue?

phatad commented 4 years ago

Hello @kaugh It still panic :'(

kaugh commented 4 years ago

all stalled (

damoun commented 4 years ago

I cannot reproduce your issues. Could you share your telegraf configuration ?

phatad commented 4 years ago

Hi @damoun, Here is my configuration :D

[global_tags] cluster = "tag-cluster"

Configuration for telegraf agent

[agent] interval = "60s" round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = "0s" flush_interval = "60s" flush_jitter = "0s" precision = "" debug = true quiet = false logfile = "/var/log/telegraf/telegraf.log" hostname = "" omit_hostname = false

Configuration for sending metrics to InfluxDB

[[outputs.influxdb]] urls = ["http://10.11.10.101:8086"] database = "db_new_telegraf"

###############################################################################

INPUT PLUGINS

###############################################################################

Read metrics about cpu usage

[[inputs.cpu]] percpu = true totalcpu = true collect_cpu_time = false report_active = false

Read metrics about disk usage by mount point

[[inputs.disk]] ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs"]

Read metrics about disk IO by device

[[inputs.diskio]]

Get kernel statistics from /proc/stat

[[inputs.kernel]]

Read metrics about memory usage

[[inputs.mem]]

Get the number of processes and group them by status

[[inputs.processes]]

Read metrics about swap memory usage

[[inputs.swap]]

Read metrics about system load & uptime

[[inputs.system]]

Read metrics about network interface usage

[[inputs.net]] [[inputs.netstat]] [[inputs.nstat]] [[inputs.interrupts]] [[inputs.linux_sysctl_fs]] [[inputs.exec]] commands = ["/root/script.sh"] json_name_key = "name" tag_keys = ["ring","cpu","hostname"] timeout = "60s" data_format = "json"

druchoo commented 4 years ago

Hello, is there any update on this issue? Still seeing it in v3.1.0 built with go version go1.14.4 linux/amd64.

Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: 2020/07/14 13:27:00 http: panic serving 127.0.0.1:18240: runtime error: index out of range [4] with length 4
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: goroutine 22655269 [running]:
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: net/http.(*conn).serve.func1(0xc0019497c0)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /usr/local/go/src/net/http/server.go:1772 +0x139
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: panic(0x8ca5e0, 0xc001c38100)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /usr/local/go/src/runtime/panic.go:975 +0x3e3
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/influxdata/influxdb/models.parseTags.func1(0xc000c1a052, 0x4, 0x3ee72, 0xc000c1a057, 0x3, 0x3ee6d, 0x3)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/pkg/mod/github.com/influxdata/influxdb@v1.7.9/models/points.go:1613 +0xcc
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/influxdata/influxdb/models.walkTags(0xc000c1a000, 0x61, 0x3eec4, 0xc00082d590)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/pkg/mod/github.com/influxdata/influxdb@v1.7.9/models/points.go:1555 +0x2ba
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/influxdata/influxdb/models.parseTags(0xc000c1a000, 0x61, 0x3eec4, 0x0, 0x0, 0x4, 0xc000c1a000, 0x6, 0x3eec4)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/pkg/mod/github.com/influxdata/influxdb@v1.7.9/models/points.go:1612 +0x151
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/influxdata/influxdb/models.(*point).Tags(0xc001e3c0f0, 0xc000c1a000, 0x6, 0x3eec4)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/pkg/mod/github.com/influxdata/influxdb@v1.7.9/models/points.go:1499 +0x71
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/strike-team/influxdb-relay/relay.(*httpBackend).validateRegexps(0xc0002ec000, 0xc000e522c0, 0x28, 0x29, 0x0, 0x0)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/src/github.com/veepee-oss/influxdb-relay/relay/http.go:339 +0x153
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/strike-team/influxdb-relay/relay.(*HTTP).handleStandard(0xc000060750, 0x9a8dc0, 0xc001991bd0, 0xc001e36e00, 0xbfbb8a8909c1247c, 0x27ac06a5e7dc, 0xcd3860)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/src/github.com/veepee-oss/influxdb-relay/relay/http_handlers.go:304 +0x6ae
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/strike-team/influxdb-relay/relay.(*HTTP).bodyMiddleWare.func1(0xc000060750, 0x9a8dc0, 0xc001991bd0, 0xc001e36e00, 0xbfbb8a8909c1247c, 0x27ac06a5e7dc, 0xcd3860)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/src/github.com/veepee-oss/influxdb-relay/relay/http_middlewares.go:42 +0x116
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/strike-team/influxdb-relay/relay.(*HTTP).queryMiddleWare.func1(0xc000060750, 0x9a8dc0, 0xc001991bd0, 0xc001e36e00, 0xbfbb8a8909c1247c, 0x27ac06a5e7dc, 0xcd3860)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/src/github.com/veepee-oss/influxdb-relay/relay/http_middlewares.go:60 +0x150
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/strike-team/influxdb-relay/relay.(*HTTP).logMiddleWare.func1(0xc000060750, 0x9a8dc0, 0xc001991bd0, 0xc001e36e00, 0xbfbb8a8909c1247c, 0x27ac06a5e7dc, 0xcd3860)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/src/github.com/veepee-oss/influxdb-relay/relay/http_middlewares.go:23 +0x81
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/strike-team/influxdb-relay/relay.(*HTTP).rateMiddleware.func1(0xc000060750, 0x9a8dc0, 0xc001991bd0, 0xc001e36e00, 0xbfbb8a8909c1247c, 0x27ac06a5e7dc, 0xcd3860)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/src/github.com/veepee-oss/influxdb-relay/relay/http_middlewares.go:72 +0x10c
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: github.com/strike-team/influxdb-relay/relay.(*HTTP).ServeHTTP(0xc000060750, 0x9a8dc0, 0xc001991bd0, 0xc001e36e00)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/src/github.com/veepee-oss/influxdb-relay/relay/http.go:199 +0xd7
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP(0xc000080370, 0x9a8dc0, 0xc001991bd0, 0xc001e36c00)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /go/pkg/mod/go.opencensus.io@v0.22.2/plugin/ochttp/server.go:92 +0x27e
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: net/http.serverHandler.ServeHTTP(0xc0000781c0, 0x9a80c0, 0xc000ad9260, 0xc001e36c00)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /usr/local/go/src/net/http/server.go:2807 +0xa3
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: net/http.(*conn).serve(0xc0019497c0, 0x9a9cc0, 0xc001c51f80)
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /usr/local/go/src/net/http/server.go:1895 +0x86c
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: created by net/http.(*Server).Serve
Jul 14 13:27:00 influxdb-relay-host influxdb-relay[884]: /usr/local/go/src/net/http/server.go:2933 +0x35c