search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.97k stars 3.56k forks source link

Scanner incorrectly handles fields after exactly 2^n*100 tags #21296

Closed davidby-influx closed 3 years ago

davidby-influx commented 3 years ago

We are seeing the panic below at a customer, which appears to be an array overrun in this line of code:

https://github.com/influxdata/influxdb/blob/e234aa7ff092d0c7cfa22259941fbe6368500f74/models/points.go#L651

The array is not correctly expanded if the first field value occurs after a 2^n*100 number of tags.

We need to formulate a repro data set to ensure that this is tested.

[httpd] 10.84.216.22,10.56.120.18 - vzwadmin [23/Apr/2021:20:03:29 +0000] "POST /write?db=vcpc_twb_metrics HTTP/1.1" 200 0 "-" "Telegraf/1.15.2 Go/1.14.5" f90cf8f5-a46e-11eb-b214-525400e85f2e 299774 [panic:runtime error: index out of range [100] with length 100] goroutine 95426493 [running]:
runtime/debug.Stack(0xc0b24bc480, 0xc063f7cc00, 0xc018e7e467bcc5e7)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1.1(0xc0b24bc480, 0xc063f7cc00, 0xc018e7e467bcc5e7, 0x617fc47ccff9, 0x38af9a0, 0xc117142500, 0x27249c0, 0xc085048fc0)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:1982 +0xc7
panic(0x1707c80, 0xc0b8f1f8a0)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models.scanTags(0xc00fd0ca2b, 0x1548, 0x773d5, 0x11, 0xc00c8428b8, 0x64, 0x64, 0x0, 0x7740d, 0xc00c8428b8, ...)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models/points.go:651 +0x29c
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models.scanKey(0xc00fd0ca2b, 0x1548, 0x773d5, 0x0, 0xed814fd6c, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models/points.go:512 +0xa2f
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models.parsePoint(0xc00fd0ca2b, 0x1548, 0x773d5, 0x39901b48, 0xed8151f11, 0x0, 0x0, 0x0, 0x2751520, 0xc00faabd10, ...)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models/points.go:385 +0x66
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models.ParsePointsWithPrecision(0xc00fd04000, 0x562f0, 0x7fe00, 0x39901b48, 0xed8151f11, 0x0, 0x0, 0x0, 0x1, 0xc00c842f10, ...)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/models/points.go:368 +0x30b
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.(*Handler).serveWrite(0xc117142500, 0xc0784fd74f, 0x10, 0x0, 0x0, 0x0, 0x0, 0x7f4f21cc10c0, 0xc0b24bc4c0, 0xc063f7cc00, ...)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:939 +0x83d
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.(*Handler).serveWriteV1(0xc117142500, 0x7f4f21cc10c0, 0xc0b24bc4c0, 0xc063f7cc00, 0x273aee0, 0xc021fca428)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:855 +0x14f
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.authenticate.func1(0x7f4f21cc10c0, 0xc0b24bc4c0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:1844 +0x94
net/http.HandlerFunc.ServeHTTP(0xc0c872b8a0, 0x7f4f21cc10c0, 0xc0b24bc4c0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.(*Handler).responseWriter.func1(0x271f700, 0xc072f857c0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:1959 +0xab
net/http.HandlerFunc.ServeHTTP(0xc0c872b8c0, 0x271f700, 0xc072f857c0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.gzipFilter.func1(0x271f780, 0xc0b24bc4a0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/gzip.go:39 +0x245
net/http.HandlerFunc.ServeHTTP(0xc0c872b8e0, 0x271f780, 0xc0b24bc4a0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
net/http.HandlerFunc.ServeHTTP(0xc0c872b900, 0x271f780, 0xc0b24bc4a0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.cors.func1(0x271f780, 0xc0b24bc4a0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:1883 +0xde
net/http.HandlerFunc.ServeHTTP(0xc0c872b960, 0x271f780, 0xc0b24bc4a0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.requestID.func1(0x271f780, 0xc0b24bc4a0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:1914 +0x34d
net/http.HandlerFunc.ServeHTTP(0xc0c872b9c0, 0x271f780, 0xc0b24bc4a0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1(0x271f780, 0xc0b24bc480, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:1940 +0xdf
net/http.HandlerFunc.ServeHTTP(0xc0c872b9e0, 0x271f780, 0xc0b24bc480, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1(0x27249c0, 0xc085048fc0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:1996 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc0c872ba40, 0x27249c0, 0xc085048fc0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/influxdata/plutonium/vendor/github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc0c872b160, 0x27249c0, 0xc085048fc0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/bmizerany/pat/mux.go:117 +0x153
github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP(0xc117142500, 0x27249c0, 0xc085048fc0, 0xc063f7cc00)
    /go/src/github.com/influxdata/plutonium/vendor/github.com/influxdata/influxdb/services/httpd/handler.go:459 +0x398
net/http.serverHandler.ServeHTTP(0xc0c81a10a0, 0x27249c0, 0xc085048fc0, 0xc063f7cc00)
    /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc153810c80, 0x272d140, 0xc1ac635d40)
    /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2928 +0x384
davidby-influx commented 3 years ago

An import file which will reproduce the problem: test.txt

davidby-influx commented 3 years ago

@rogpeppe - for your test cases.....

danxmoran commented 3 years ago

Agh, I think there's been a fix for this waiting in our PR queue. See #20138