search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.65k stars 3.54k forks source link

Panic with concurrent map writes error when running Flux scripts from Chronograf #19702

Open CheyiLin opened 3 years ago

CheyiLin commented 3 years ago

Steps to reproduce:

Unfortunately I can't reproduce this all the time, maybe it's also data related?

Simplified Flux scripts:

(calculating aggregated Apdex score from HTTP access logs, not yet completed)

ds = from(bucket:"my-http-access-log-database")
  |> range(start: dashboardTime, stop: upperDashboardTime)
  |> filter(fn: (r) => 
    r._measurement == "my-http-access-log-measurement" and 
    r._field == "v.response-time"
  )

apdexT = 0.3

satisfiedCount = ds
  |> filter(fn: (r) => 
    r._value <= apdexT and
    r["v.status"] !~ /5../
  )
  |> keep(columns: ["_time", "_value"])
  |> aggregateWindow(every: 5m, fn: count)

toleratingCount = ds
  |> filter(fn: (r) => 
    r._value <= apdexT * 4.0 and r._value > apdexT and
    r["v.status"] !~ /5../
  )
  |> keep(columns: ["_time", "_value"])
  |> aggregateWindow(every: 5m, fn: count)

join(
  tables: {sat: satisfiedCount, tol: toleratingCount},
  on: ["_time"]
) |> yield()

Expected behavior: Got result from InfluxDB.

Actual behavior: Got bad gateway error, InfluxDB been terminated due to panic. (logs attached below)

Environment info:

Config: Only with following additional environment variables:

INFLUXDB_REPORTING_DISABLED: "true"
INFLUXDB_HTTP_FLUX_ENABLED: "true"
INFLUXDB_COORDINATOR_QUERY_TIMEOUT: 10s
INFLUXDB_COORDINATOR_LOG_QUERIES_AFTER: 8s

Logs:

Panic goroutine

fatal error: concurrent map writes
goroutine 861 [running]:
runtime.throw(0x16a301b, 0x15)
    /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc002dffac8 sp=0xc002dffa98 pc=0x431272
runtime.mapassign_faststr(0x1421660, 0xc002a03cb0, 0x31721d2, 0x1, 0x25ab320)
    /usr/local/go/src/runtime/map_faststr.go:211 +0x417 fp=0xc002dffb30 sp=0xc002dffac8 pc=0x4150d7
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/values.(*object).Set(0xc002891e60, 0x31721d2, 0x1, 0x25ab320, 0xc002cbb4a0)
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/values/object.go:110 +0x650 fp=0xc002dffbb8 sp=0xc002dffb30 pc=0xc4f480
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*RowPredicateFn).Eval(0xc002891ec0, 0x258c820, 0xc002a038f0, 0x25ade20, 0xc002cbb4a0, 0xc003119f00, 0x0, 0x0)
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/row_fn.go:214 +0xa2 fp=0xc002dffc40 sp=0xc002dffbb8 pc=0xce9ef2
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filter(0xc002a03da0, 0x25a6d80, 0xc00036b720, 0x25ade20, 0xc002cbb4a0, 0xc003060d20, 0xc002a46000, 0x1f40, 0x1f40)
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:279 +0x24e fp=0xc002dffcf8 sp=0xc002dffc40 pc=0xe8760e
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x25a6d80, 0xc00036b720, 0x0, 0x0)
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:241 +0xa5 fp=0xc002dffe00 sp=0xc002dffcf8 pc=0xf04345
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*tableBuffer).Do(0xc00036b7c0, 0xc003084180, 0x0, 0x0)
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/table.go:64 +0xd6 fp=0xc002dffe78 sp=0xc002dffe00 pc=0xcee306
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x258c760, 0xc003084100, 0xc003084140, 0xc004058d20, 0xc0004f0730)
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:240 +0xe7 fp=0xc002dffed8 sp=0xc002dffe78 pc=0xf04817
github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc004001da0, 0xc004001d40, 0xc0030840c0, 0x258c760, 0xc003084100, 0xc003084140)
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:109 +0xd6 fp=0xc002dfffb0 sp=0xc002dffed8 pc=0xe625d6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc002dfffb8 sp=0xc002dfffb0 pc=0x460f61
created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext
    /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:106 +0x1a7

Other goroutines

Show logs ``` goroutine 1 [chan receive, 1 minutes]: main.(*Main).Run(0xc00038bf20, 0xc0000aa1b0, 0x0, 0x0, 0xc00038bf20, 0x43f0ca) /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:90 +0x2c9 main.main() /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45 +0x13d goroutine 19 [syscall, 1 minutes]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:147 +0x9c os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:29 +0x41 goroutine 8 [select]: github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view.(*worker).start(0xc00017ef50) /go/src/github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100 created by github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view.init.0 /go/src/github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57 goroutine 9 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7fc9b3795ec8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc000173518, 0x72, 0x0, 0x0, 0x168abea) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000173500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8 net.(*netFD).accept(0xc000173500, 0x0, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000427580, 0x0, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000427580, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/tcpsock.go:261 +0x47 github.com/influxdata/influxdb/tcp.(*Mux).Serve(0xc00045e120, 0x2584720, 0xc000427580, 0xc000427580, 0x0) /go/src/github.com/influxdata/influxdb/tcp/mux.go:75 +0x92 created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:393 +0x280 goroutine 189 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795c58, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc00011ff18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00011ff00, 0xc002e3f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc00011ff00, 0xc002e3f000, 0x1000, 0x1000, 0xc002e099e8, 0x4d33dd, 0xc00011ff00) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0000c2820, 0xc002e3f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc002e0da10, 0xc002e3f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002dd0ea0) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).Peek(0xc002dd0ea0, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc002e09ad0) /usr/local/go/src/bufio/bufio.go:138 +0x4f net/http.(*conn).readRequest(0xc000221e00, 0x258c760, 0xc00044cdc0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:962 +0xb3b net/http.(*conn).serve(0xc000221e00, 0x258c760, 0xc00044cdc0) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 205 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc0000d2140) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2027 +0x132 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc002c67ba0, 0xc0000d2140) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x5b created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x155 goroutine 853 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).UnsafeWriteBuffer(0xc002bf47c0, 0x25a7080, 0xc002c3c880, 0x0, 0xc002e00d78) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:67 +0x126 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).write(0xc002bf47c0, 0xc0029fd7c0, 0xa, 0xa, 0x25a3601, 0xc002ad97a0, 0xc0045d8140) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:54 +0xc7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).Write(...) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:32 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x25a6d80, 0xc00036b720, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:262 +0x341 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*tableBuffer).Do(0xc00036b770, 0xc002bf4800, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/table.go:64 +0xd6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x258c760, 0xc002bf4780, 0xc002bf47c0, 0xc002a26240, 0xc001d5e1e0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:240 +0xe7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc003188c00, 0xc003188ba0, 0xc002bf4740, 0x258c760, 0xc002bf4780, 0xc002bf47c0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:109 +0xd6 created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:106 +0x1a7 goroutine 196 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc00017d180, 0xc002c67b10) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2074 +0x26f github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc002c67b10, 0xc00017d180) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x65 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x131 goroutine 313 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795848, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0009cdf18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0009cdf00, 0xc002b50000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc0009cdf00, 0xc002b50000, 0x1000, 0x1000, 0xc00168f9e8, 0x4d33dd, 0xc0009cdf00) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000424908, 0xc002b50000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc002e0ef60, 0xc002b50000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002dd0a80) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).Peek(0xc002dd0a80, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc00168fad0) /usr/local/go/src/bufio/bufio.go:138 +0x4f net/http.(*conn).readRequest(0xc001a2e1e0, 0x258c760, 0xc0002d63c0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:962 +0xb3b net/http.(*conn).serve(0xc001a2e1e0, 0x258c760, 0xc0002d63c0) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 850 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*executionState).do.func3(0xc000b6d380, 0x258c820, 0xc002a038f0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/executor.go:292 +0x16a created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*executionState).do /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/executor.go:289 +0x1b4 goroutine 829 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*result).Do(0xc002a26240, 0xc002a0a3c0, 0xc002a26240, 0xc000c3b000) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/result.go:60 +0xd9 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/csv.(*ResultEncoder).Encode(0xc002a26270, 0x2556ec0, 0xc0029ff520, 0x2571100, 0xc002a26240, 0xc0012e4f48, 0x40be65, 0x1471880) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/csv/result.go:830 +0x326 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux.(*DelimitedMultiResultEncoder).Encode(0xc002a262a0, 0x7fc99eee7f68, 0xc004696520, 0x2591d60, 0xc002a262d0, 0x2592101, 0xc0001e0160, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/result.go:294 +0xd9 github.com/influxdata/influxdb/services/httpd.(*Handler).serveFluxQuery(0xc00022dd00, 0x7fc99ef183d0, 0xc004696520, 0xc00022d700, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1419 +0x570 github.com/influxdata/influxdb/services/httpd.authenticate.func1(0x7fc99ef183d0, 0xc004696520, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1756 +0x7e1 net/http.HandlerFunc.ServeHTTP(0xc0004744c0, 0x7fc99ef183d0, 0xc004696520, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 github.com/influxdata/influxdb/services/httpd.(*Handler).responseWriter.func1(0x257c8e0, 0xc00034a3c0, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1959 +0xab net/http.HandlerFunc.ServeHTTP(0xc0004744e0, 0x257c8e0, 0xc00034a3c0, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 github.com/influxdata/influxdb/services/httpd.gzipFilter.func1(0x257c960, 0xc004696500, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/gzip.go:39 +0x245 net/http.HandlerFunc.ServeHTTP(0xc000474500, 0x257c960, 0xc004696500, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 net/http.HandlerFunc.ServeHTTP(0xc000474520, 0x257c960, 0xc004696500, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 github.com/influxdata/influxdb/services/httpd.cors.func1(0x257c960, 0xc004696500, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1883 +0xde net/http.HandlerFunc.ServeHTTP(0xc000474540, 0x257c960, 0xc004696500, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 github.com/influxdata/influxdb/services/httpd.requestID.func1(0x257c960, 0xc004696500, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1914 +0x34d net/http.HandlerFunc.ServeHTTP(0xc000474560, 0x257c960, 0xc004696500, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1(0x257c960, 0xc0046964e0, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1940 +0xdf net/http.HandlerFunc.ServeHTTP(0xc000474580, 0x257c960, 0xc0046964e0, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1(0x25849e0, 0xc0003b9260, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1996 +0x17b net/http.HandlerFunc.ServeHTTP(0xc0004745a0, 0x25849e0, 0xc0003b9260, 0xc00022d700) /usr/local/go/src/net/http/server.go:2007 +0x44 github.com/influxdata/influxdb/vendor/github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc0004275c0, 0x25849e0, 0xc0003b9260, 0xc00022d700) /go/src/github.com/influxdata/influxdb/vendor/github.com/bmizerany/pat/mux.go:117 +0x153 github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP(0xc00022dd00, 0x25849e0, 0xc0003b9260, 0xc00022d700) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:459 +0x398 net/http.serverHandler.ServeHTTP(0xc0001260e0, 0x25849e0, 0xc0003b9260, 0xc00022d700) /usr/local/go/src/net/http/server.go:2802 +0xa4 net/http.(*conn).serve(0xc000458280, 0x258c760, 0xc0007add80) /usr/local/go/src/net/http/server.go:1890 +0x875 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 870 [runnable]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.Values(0x25a6d80, 0xc006264320, 0x0, 0xc001a2f400, 0xc8) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/utils.go:11 +0x322 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x25a6d80, 0xc006264320, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:255 +0x1fc github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*tableBuffer).Do(0xc0062644b0, 0xc0031ab300, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/table.go:64 +0xd6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x258c760, 0xc0031ab280, 0xc0031ab2c0, 0xc000275e50, 0xc0004f6718) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:240 +0xe7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc004251aa0, 0xc004251a40, 0xc0031ab240, 0x258c760, 0xc0031ab280, 0xc0031ab2c0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:109 +0xd6 created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:106 +0x1a7 goroutine 371 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b37956a8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc003717d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc003717d80, 0xc003c1a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc003717d80, 0xc003c1a000, 0x1000, 0x1000, 0x0, 0x3030320000000000, 0xed7107cd3) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000424f08, 0xc003c1a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc0037ab2c0, 0xc003c1a000, 0x1000, 0x1000, 0xc000822f80, 0x315e2fc, 0x38600000002) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002dd1680) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).ReadSlice(0xc002dd1680, 0xa, 0x7fc99f271b80, 0xc002f399a8, 0x40de86, 0xc006032b00, 0x100) /usr/local/go/src/bufio/bufio.go:359 +0x3d bufio.(*Reader).ReadLine(0xc002dd1680, 0xc002f399b0, 0x35f5a60, 0x7fc9b383e6d0, 0x0, 0x1, 0xc002f399f0) /usr/local/go/src/bufio/bufio.go:388 +0x34 net/textproto.(*Reader).readLineSlice(0xc002e0dda0, 0xc006032b00, 0xc003717d80, 0x0, 0x0, 0x43309c) /usr/local/go/src/net/textproto/reader.go:57 +0x6c net/textproto.(*Reader).ReadLine(...) /usr/local/go/src/net/textproto/reader.go:38 net/http.readRequest(0xc002dd1680, 0x0, 0xc006032b00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:1012 +0x92 net/http.(*conn).readRequest(0xc001a2e5a0, 0x258c760, 0xc000822f40, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:965 +0x15f net/http.(*conn).serve(0xc001a2e5a0, 0x258c760, 0xc000822f40) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 195 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc00017d180) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2027 +0x132 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc002c67b00, 0xc00017d180) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x5b created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x155 goroutine 198 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc00017d540, 0xc002c67b30) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2074 +0x26f github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc002c67b30, 0xc00017d540) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x65 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x131 goroutine 261 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b37959e8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc000280718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000280700, 0xc0014ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc000280700, 0xc0014ef000, 0x1000, 0x1000, 0xc002f3b9e8, 0x4d33dd, 0xc000280700) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0004248e0, 0xc0014ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc002e0dd70, 0xc0014ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002194960) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).Peek(0xc002194960, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc002f3bad0) /usr/local/go/src/bufio/bufio.go:138 +0x4f net/http.(*conn).readRequest(0xc001a2e000, 0x258c760, 0xc00044d680, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:962 +0xb3b net/http.(*conn).serve(0xc001a2e000, 0x258c760, 0xc00044d680) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 350 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795778, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc003717818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc003717800, 0xc003b4c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc003717800, 0xc003b4c000, 0x1000, 0x1000, 0x0, 0x0, 0x15e9340) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000424e20, 0xc003b4c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc0037aac30, 0xc003b4c000, 0x1000, 0x1000, 0x5, 0x0, 0x38800000000) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002dd1560) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).ReadSlice(0xc002dd1560, 0xa, 0x7fc9b378ea30, 0xc00168b9a8, 0x40de86, 0xc00022c100, 0x100) /usr/local/go/src/bufio/bufio.go:359 +0x3d bufio.(*Reader).ReadLine(0xc002dd1560, 0xc00168b9b0, 0xc000060e00, 0x7fc9b383e008, 0x0, 0x0, 0xc00168b9f0) /usr/local/go/src/bufio/bufio.go:388 +0x34 net/textproto.(*Reader).readLineSlice(0xc003df0000, 0xc00022c100, 0xc003717800, 0x0, 0x0, 0x43309c) /usr/local/go/src/net/textproto/reader.go:57 +0x6c net/textproto.(*Reader).ReadLine(...) /usr/local/go/src/net/textproto/reader.go:38 net/http.readRequest(0xc002dd1560, 0x0, 0xc00022c100, 0x0, 0x0) /usr/local/go/src/net/http/request.go:1012 +0x92 net/http.(*conn).readRequest(0xc001a2e500, 0x258c760, 0xc000822880, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:965 +0x15f net/http.(*conn).serve(0xc001a2e500, 0x258c760, 0xc000822880) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 855 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).UnsafeWriteBuffer(0xc002c41c80, 0x25a7080, 0xc002c56540, 0x0, 0xc002dfcd78) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:67 +0x126 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).write(0xc002c41c80, 0xc0029fdae0, 0xa, 0xa, 0x25a3601, 0xc002c4cde0, 0xc0045d8140) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:54 +0xc7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).Write(...) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:32 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x25a6d80, 0xc00036bbd0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:262 +0x341 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*tableBuffer).Do(0xc00036bcc0, 0xc002c41cc0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/table.go:64 +0xd6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x258c760, 0xc002c41c40, 0xc002c41c80, 0xc004058d20, 0xc002df3730) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:240 +0xe7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc003188e40, 0xc003188de0, 0xc002c41c00, 0x258c760, 0xc002c41c40, 0xc002c41c80) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:109 +0xd6 created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:106 +0x1a7 goroutine 297 [IO wait]: goroutine 297 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795918, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc000581c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000581c80, 0xc001537000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc000581c80, 0xc001537000, 0x1000, 0x1000, 0xc002e059e8, 0x4d33dd, 0xc000581c80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0000c2300, 0xc001537000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc0035a8660, 0xc001537000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002195080) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).Peek(0xc002195080, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc002e05ad0) /usr/local/go/src/bufio/bufio.go:138 +0x4f net/http.(*conn).readRequest(0xc0004588c0, 0x258c760, 0xc00044da40, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:962 +0xb3b net/http.(*conn).serve(0xc0004588c0, 0x258c760, 0xc00044da40) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 830 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b37955d8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0005db798, 0x72, 0x0, 0x1, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0005db780, 0xc00458c611, 0x1, 0x1, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc0005db780, 0xc00458c611, 0x1, 0x1, 0xc0007adc58, 0xc0004f3768, 0x48309c) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0000c26b8, 0xc00458c611, 0x1, 0x1, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).backgroundRead(0xc00458c600) /usr/local/go/src/net/http/server.go:677 +0x58 created by net/http.(*connReader).startBackgroundRead /usr/local/go/src/net/http/server.go:673 +0xd4 goroutine 833 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext(0x258c760, 0xc0031ab280, 0x25aace0, 0xc0062642d0, 0xc002a72ff0, 0xa, 0xa, 0xc0031ab240, 0xffffffffffffffff, 0x0, ...) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:114 +0x25b github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable(0xc002a03da0, 0x7fc99ef1b700, 0xc0062644b0, 0x25ade20, 0xc0031a69c0, 0xc003df15f0, 0xc002bf0978, 0x0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:239 +0x169 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).Process(0xc002a03da0, 0x4d5eccd93a5f1dca, 0xb8e093295a108e, 0x7fc99ef1b700, 0xc0062644b0, 0x1, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:228 +0x6e0 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.processMessage(0x258c820, 0xc002a038f0, 0x25935a0, 0xc002a03da0, 0x25710c0, 0xc0041bfc20, 0xc002bcbe00, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/transport.go:206 +0x2d6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*consecutiveTransport).processMessages(0xc002891f20, 0x258c820, 0xc002a038f0, 0xa) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/transport.go:159 +0xc6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*poolDispatcher).run(0xc002891b00, 0x258c820, 0xc002a038f0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/dispatcher.go:132 +0x65 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*poolDispatcher).Start.func1(0xc002891b00, 0x258c820, 0xc002a038f0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/dispatcher.go:86 +0xcd created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*poolDispatcher).Start /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/dispatcher.go:63 +0x7e goroutine 869 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).UnsafeWriteBuffer(0xc00313e040, 0x25a7080, 0xc0031ab200, 0x0, 0xc002dfad78) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:67 +0x126 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).write(0xc00313e040, 0xc001a2f2c0, 0xa, 0xa, 0x25a3601, 0xc0031a6900, 0xc0045d8140) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:54 +0xc7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).Write(...) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:32 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x25a6d80, 0xc0062641e0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:262 +0x341 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*tableBuffer).Do(0xc006264280, 0xc00313e080, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/table.go:64 +0xd6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x258c760, 0xc00313e000, 0xc00313e040, 0x44345f, 0xc000093c80) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:240 +0xe7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc004251980, 0xc004251920, 0xc003135fc0, 0x258c760, 0xc00313e000, 0xc00313e040) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:109 +0xd6 created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:106 +0x1a7 goroutine 857 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).UnsafeWriteBuffer(0xc002c56b40, 0x25a7080, 0xc002c93d00, 0x0, 0xc002dfbd78) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:67 +0x126 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).write(0xc002c56b40, 0xc0029fdd60, 0xa, 0xa, 0x25a3601, 0xc002c4dda0, 0xc0045d8140) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:54 +0xc7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).Write(...) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:32 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x25a6d80, 0xc0062641e0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:262 +0x341 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*tableBuffer).Do(0xc006264230, 0xc002c56b80, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/table.go:64 +0xd6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x258c760, 0xc002c56b00, 0xc002c56b40, 0xc004058d20, 0xc0004f1730) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:240 +0xe7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc003189080, 0xc003189020, 0xc002c56ac0, 0x258c760, 0xc002c56b00, 0xc002c56b40) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:109 +0xd6 created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:106 +0x1a7 goroutine 188 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795d28, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc000280618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000280600, 0xc0001ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc000280600, 0xc0001ce000, 0x1000, 0x1000, 0x0, 0x3030320000000000, 0xed7107cde) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000424d68, 0xc0001ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc002e0c120, 0xc0001ce000, 0x1000, 0x1000, 0xc0003a8700, 0x315e2fc, 0x2) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002dd0ba0) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).ReadSlice(0xc002dd0ba0, 0xa, 0xc002f3db28, 0xc002f3d9a8, 0x40de86, 0xc0030a1b00, 0x100) /usr/local/go/src/bufio/bufio.go:359 +0x3d bufio.(*Reader).ReadLine(0xc002dd0ba0, 0xc002f3d9b0, 0xc000060700, 0x7fc9b383e008, 0x0, 0x0, 0xc002f3d9f0) /usr/local/go/src/bufio/bufio.go:388 +0x34 net/textproto.(*Reader).readLineSlice(0xc0035a8900, 0xc0030a1b00, 0xc000280600, 0x0, 0x0, 0x43309c) /usr/local/go/src/net/textproto/reader.go:57 +0x6c net/textproto.(*Reader).ReadLine(...) /usr/local/go/src/net/textproto/reader.go:38 net/http.readRequest(0xc002dd0ba0, 0x0, 0xc0030a1b00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:1012 +0x92 net/http.(*conn).readRequest(0xc0000d4e60, 0x258c760, 0xc0003a85c0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:965 +0x15f net/http.(*conn).serve(0xc0000d4e60, 0x258c760, 0xc0003a85c0) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 197 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc00017d540) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2027 +0x132 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc002c67b20, 0xc00017d540) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x5b created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x155 goroutine 206 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc0000d2140, 0xc002c67bb0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2074 +0x26f github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc002c67bb0, 0xc0000d2140) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x65 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x131 goroutine 210 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc00017cc80, 0xc002c67bf0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2074 +0x26f github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc002c67bf0, 0xc00017cc80) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x65 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:426 +0x131 goroutine 209 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc00017cc80) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2027 +0x132 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc002c67be0, 0xc00017cc80) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x5b created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:505 +0x155 goroutine 211 [select]: github.com/influxdata/influxdb/tsdb.(*Store).monitorShards(0xc00020fc00) /go/src/github.com/influxdata/influxdb/tsdb/store.go:1935 +0x19e github.com/influxdata/influxdb/tsdb.(*Store).Open.func1(0xc00020fc00) /go/src/github.com/influxdata/influxdb/tsdb/store.go:232 +0x4b created by github.com/influxdata/influxdb/tsdb.(*Store).Open /go/src/github.com/influxdata/influxdb/tsdb/store.go:230 +0x30f goroutine 212 [select]: github.com/influxdata/influxdb/services/subscriber.(*Service).run(0xc0003ba780) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:239 +0x296 github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func1(0xc0003ba780) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:98 +0x61 created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:96 +0x17e goroutine 213 [select, 1 minutes]: github.com/influxdata/influxdb/services/subscriber.(*Service).waitForMetaUpdates(0xc0003ba780) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:165 +0xcf github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func2(0xc0003ba780) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:102 +0x61 created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:100 +0x1a3 goroutine 214 [select]: github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics(0xc000454000) /go/src/github.com/influxdata/influxdb/monitor/service.go:434 +0x9fa created by github.com/influxdata/influxdb/monitor.(*Monitor).Open /go/src/github.com/influxdata/influxdb/monitor/service.go:126 +0x2de goroutine 215 [select, 1 minutes]: github.com/influxdata/influxdb/services/precreator.(*Service).runPrecreation(0xc00043d580) /go/src/github.com/influxdata/influxdb/services/precreator/service.go:76 +0x10a created by github.com/influxdata/influxdb/services/precreator.(*Service).Open /go/src/github.com/influxdata/influxdb/services/precreator/service.go:54 +0x22b goroutine 216 [select, 1 minutes]: github.com/influxdata/influxdb/tcp.(*listener).Accept(0xc00077f2f0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/tcp/mux.go:236 +0x11d github.com/influxdata/influxdb/services/snapshotter.(*Service).serve(0xc00045c5f0) /go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:94 +0x85 created by github.com/influxdata/influxdb/services/snapshotter.(*Service).Open /go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:68 +0x92 goroutine 217 [select]: github.com/influxdata/influxdb/services/continuous_querier.(*Service).backgroundLoop(0xc0002107e0) /go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:215 +0x198 created by github.com/influxdata/influxdb/services/continuous_querier.(*Service).Open /go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:133 +0x11f goroutine 218 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795df8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc00011f918, 0x72, 0x0, 0x0, 0x168abea) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc00011f900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8 net.(*netFD).accept(0xc00011f900, 0xc002e01d68, 0x78c914, 0xc000126180) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc00221f420, 0x5f7e85e2, 0xc002e01d68, 0x4cab26) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc00221f420, 0xc002e01db8, 0x18, 0xc002decc00, 0x78be44) /usr/local/go/src/net/tcpsock.go:261 +0x47 net/http.(*Server).Serve(0xc0001260e0, 0x2584720, 0xc00221f420, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2896 +0x280 net/http.Serve(...) /usr/local/go/src/net/http/server.go:2468 github.com/influxdata/influxdb/services/httpd.(*Service).serve(0xc000458780, 0x2584720, 0xc00221f420) /go/src/github.com/influxdata/influxdb/services/httpd/service.go:250 +0x9c github.com/influxdata/influxdb/services/httpd.(*Service).serveTCP(0xc000458780) /go/src/github.com/influxdata/influxdb/services/httpd/service.go:238 +0x3c created by github.com/influxdata/influxdb/services/httpd.(*Service).Open /go/src/github.com/influxdata/influxdb/services/httpd/service.go:187 +0x65c goroutine 219 [select, 1 minutes]: github.com/influxdata/influxdb/services/retention.(*Service).run(0xc0000ca5a0) /go/src/github.com/influxdata/influxdb/services/retention/service.go:78 +0xe09 github.com/influxdata/influxdb/services/retention.(*Service).Open.func1(0xc0000ca5a0) /go/src/github.com/influxdata/influxdb/services/retention/service.go:51 +0x61 created by github.com/influxdata/influxdb/services/retention.(*Service).Open /go/src/github.com/influxdata/influxdb/services/retention/service.go:51 +0x1a9 goroutine 220 [select, 1 minutes]: github.com/influxdata/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0xc000275080) /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:174 +0x178 created by github.com/influxdata/influxdb/cmd/influxd/run.(*Command).Run /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:155 +0xeb0 goroutine 239 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795b88, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc0005db598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0005db580, 0xc002e69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc0005db580, 0xc002e69000, 0x1000, 0x1000, 0x0, 0x0, 0x15e9340) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000425388, 0xc002e69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc0035a8570, 0xc002e69000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc002dd1260) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).ReadSlice(0xc002dd1260, 0xa, 0xc002e03998, 0xc002e039a8, 0x40de86, 0xc001384600, 0x100) /usr/local/go/src/bufio/bufio.go:359 +0x3d bufio.(*Reader).ReadLine(0xc002dd1260, 0xc002e039b0, 0xc000060700, 0x7fc9b383e6d0, 0x0, 0x1, 0xc002e039f0) /usr/local/go/src/bufio/bufio.go:388 +0x34 net/textproto.(*Reader).readLineSlice(0xc0037aae40, 0xc001384600, 0xc0005db580, 0x0, 0x0, 0x43309c) /usr/local/go/src/net/textproto/reader.go:57 +0x6c net/textproto.(*Reader).ReadLine(...) /usr/local/go/src/net/textproto/reader.go:38 net/http.readRequest(0xc002dd1260, 0x0, 0xc001384600, 0x0, 0x0) /usr/local/go/src/net/http/request.go:1012 +0x92 net/http.(*conn).readRequest(0xc0000d5360, 0x258c760, 0xc00043ca40, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:965 +0x15f net/http.(*conn).serve(0xc0000d5360, 0x258c760, 0xc00043ca40) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 275 [IO wait]: internal/poll.runtime_pollWait(0x7fc9b3795ab8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc000580998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000580980, 0xc004809000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf net.(*netFD).Read(0xc000580980, 0xc004809000, 0x1000, 0x1000, 0x0, 0x0, 0x15e9340) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0000c21c0, 0xc004809000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:184 +0x68 net/http.(*connReader).Read(0xc002107590, 0xc004809000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /usr/local/go/src/net/http/server.go:785 +0xf4 bufio.(*Reader).fill(0xc001f78a20) /usr/local/go/src/bufio/bufio.go:100 +0x103 bufio.(*Reader).ReadSlice(0xc001f78a20, 0xa, 0xc002f3f998, 0xc002f3f9a8, 0x40de86, 0xc001385200, 0x100) /usr/local/go/src/bufio/bufio.go:359 +0x3d bufio.(*Reader).ReadLine(0xc001f78a20, 0xc002f3f9b0, 0xc000122a80, 0x7fc9b383e6d0, 0x0, 0x1, 0xc002f3f9f0) /usr/local/go/src/bufio/bufio.go:388 +0x34 net/textproto.(*Reader).readLineSlice(0xc002e0f140, 0xc001385200, 0xc000580980, 0x0, 0x0, 0x4ca22d) /usr/local/go/src/net/textproto/reader.go:57 +0x6c net/textproto.(*Reader).ReadLine(...) /usr/local/go/src/net/textproto/reader.go:38 net/http.readRequest(0xc001f78a20, 0x0, 0xc001385200, 0x0, 0x0) /usr/local/go/src/net/http/request.go:1012 +0x92 net/http.(*conn).readRequest(0xc0000d5900, 0x258c760, 0xc0006a5d80, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:965 +0x15f net/http.(*conn).serve(0xc0000d5900, 0x258c760, 0xc0006a5d80) /usr/local/go/src/net/http/server.go:1817 +0x6d4 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2928 +0x384 goroutine 867 [select]: github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).UnsafeWriteBuffer(0xc003135180, 0x25a7080, 0xc003135a40, 0x0, 0xc002dfed78) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:67 +0x126 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).write(0xc003135180, 0xc001a2f040, 0xa, 0xa, 0x25a3601, 0xc002aa59e0, 0xc0045d8140) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:54 +0xc7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.(*StreamWriter).Write(...) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:32 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x25a6d80, 0xc00036bbd0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:262 +0x341 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute.(*tableBuffer).Do(0xc00036bd10, 0xc0031351c0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/execute/table.go:64 +0xd6 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x258c760, 0xc003135140, 0xc003135180, 0x44345f, 0xc000093c80) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/stdlib/universe/filter.go:240 +0xe7 github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc004251740, 0xc0042516e0, 0xc003135100, 0x258c760, 0xc003135140, 0xc003135180) /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:109 +0xd6 created by github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table.StreamWithContext /go/src/github.com/influxdata/influxdb/vendor/github.com/influxdata/flux/internal/execute/table/stream.go:106 +0x1a7 ```
browdus commented 3 years ago

We experienced crashes with matching stack traces on influxdb v1.8.6