search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.83k stars 3.55k forks source link

Panic due to concurrent map modification #14362

Open GFriedrich opened 5 years ago

GFriedrich commented 5 years ago

I'm running version 1.7.6 of InfluxDB and stumbled into an application panic. (I also highly doubt that this would be better with version 1.7.7.)

Steps to reproduce: I currently can't reproduce the issue with certainty, it only happens from time to time when trying to run several delete queries at once.

Expected behavior: No panic

Actual behavior: A panic is thrown which results in an InfluxDB restart.

Logs:

2019-07-17T07:06:27.183380+02:00 metrics influxd: fatal error: concurrent map iteration and map write
2019-07-17T07:06:27.185441+02:00 metrics influxd: goroutine 152537573 [running]:
2019-07-17T07:06:27.185931+02:00 metrics influxd: runtime.throw(0x12f0502, 0x26)
2019-07-17T07:06:27.186208+02:00 metrics influxd: /usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc0000aebf8 sp=0xc0000aebc8 pc=0x42df92
2019-07-17T07:06:27.186483+02:00 metrics influxd: runtime.mapiternext(0xc0000aed38)
2019-07-17T07:06:27.186756+02:00 metrics influxd: /usr/local/go/src/runtime/map.go:790 +0x525 fp=0xc0000aec80 sp=0xc0000aebf8 pc=0x410945
2019-07-17T07:06:27.187068+02:00 metrics influxd: github.com/influxdata/influxdb/tsdb/index/tsi1.(*logTagKey).TagValueIterator(0xc0da8807f0, 0x0, 0x4)
2019-07-17T07:06:27.187335+02:00 metrics influxd: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:1362 +0xf8 fp=0xc0000aeda8 sp=0xc0000aec80 pc=0xf13218
2019-07-17T07:06:27.187598+02:00 metrics influxd: github.com/influxdata/influxdb/tsdb/index/tsi1.tagKeyMergeElem.TagValueIterator(0xc1db5e8d20, 0x4, 0x5, 0x0, 0x0)
2019-07-17T07:06:27.187893+02:00 metrics influxd: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/tsi1.go:271 +0xdb fp=0xc0000aee80 sp=0xc0000aeda8 pc=0xf2524b
2019-07-17T07:06:27.188166+02:00 metrics influxd: github.com/influxdata/influxdb/tsdb/index/tsi1.(*tagKeyMergeElem).TagValueIterator(0xc2454445c0, 0xc07d724240, 0x22)
2019-07-17T07:06:27.188424+02:00 metrics influxd: <autogenerated>:1 +0x4e fp=0xc0000aeeb8 sp=0xc0000aee80 pc=0xf2ad1e
2019-07-17T07:06:27.188672+02:00 metrics influxd: github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).DropMeasurement(0xc057d2f200, 0xc07d724240, 0x22, 0x30, 0x0, 0x0)
2019-07-17T07:06:27.188963+02:00 metrics influxd: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:587 +0x18e fp=0xc0000aef60 sp=0xc0000aeeb8 pc=0xf1a2fe
2019-07-17T07:06:27.189339+02:00 metrics influxd: github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement.func1(0xc079e8a854, 0xc0002f5950, 0xc1a9c41320, 0xc07d724240, 0x22, 0x30)
2019-07-17T07:06:27.189607+02:00 metrics influxd: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:609 +0x4e fp=0xc0000aefb0 sp=0xc0000aef60 pc=0xf26bfe
2019-07-17T07:06:27.189920+02:00 metrics influxd: runtime.goexit()
2019-07-17T07:06:27.190249+02:00 metrics influxd: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000aefb8 sp=0xc0000aefb0 pc=0x45d971
2019-07-17T07:06:27.190515+02:00 metrics influxd: created by github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement
2019-07-17T07:06:27.190790+02:00 metrics influxd: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:603 +0xf6
foobar commented 5 years ago

I'm seeing this issue with v1.7.7

foobar commented 5 years ago

The comments say deletes for the same measurement should be ordered but the code seems to only synchronize with writes. So it results in concurrently calling into sh.DeleteMeasurement for the same measurement. Is this the expected behavior?

https://github.com/influxdata/influxdb/blob/db362197fbc98d380d7dfaf7c69ca97ced38ea1e/tsdb/store.go#L940-L947

foobar commented 5 years ago

Hi @russorat is anyone looking into this issue?

russorat commented 5 years ago

@foobar just for my edification, could you describe the use case as to why you are running so many delete queries on the DB?

It might be a little while until the team investigates this. if you believe you've found the issue, we'd be happy to review a community contribution.

foobar commented 5 years ago

@foobar just for my edification, could you describe the use case as to why you are running so many delete queries on the DB?

In my case, it's when a DELETE times out and canceled but the server doesn't stop the processing. Then a retry DELETE comes in, so two deletes on the same measurement run concurrently.

foobar commented 5 years ago

@russorat I've got a possible fix with PR #14606. Could you please get it reviewed?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

russorat commented 4 years ago

pr still in review. pinged the reviewer to remind them.

toomasp commented 4 years ago

Still happens on 1.7.9 - but we're not doing any DELETE statements. InfluxDB is just running and ingesting new data during this. Could be the retention policy doing its thing.

influxd[16407]: fatal error: concurrent map iteration and map write
influxd[16407]: goroutine 447999 [running]:
influxd[16407]: runtime.throw(0x1554ff8, 0x26)
influxd[16407]: /usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc039943bf0 sp=0xc039943bc0 pc=0x42f482
influxd[16407]: runtime.mapiternext(0xc039943d50)
influxd[16407]: /usr/local/go/src/runtime/map.go:860 +0x597 fp=0xc039943c78 sp=0xc039943bf0 pc=0x410a87
influxd[16407]: github.com/influxdata/influxdb/tsdb/index/tsi1.(*logTagKey).TagValueIterator(0xc028187c40, 0x0, 0x1)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:1362 +0xe6 fp=0xc039943dc0 sp=0xc039943c78 pc=0x10f4dd6
influxd[16407]: github.com/influxdata/influxdb/tsdb/index/tsi1.tagKeyMergeElem.TagValueIterator(0xc0301fbfd0, 0x1, 0x1, 0xc028187c40, 0xc0301fbf01)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/tsi1.go:271 +0xd8 fp=0xc039943e80 sp=0xc039943dc0 pc=0x1107258
influxd[16407]: github.com/influxdata/influxdb/tsdb/index/tsi1.(*tagKeyMergeElem).TagValueIterator(0xc024bd8c80, 0x214da01, 0xc024bd8c80)
influxd[16407]: <autogenerated>:1 +0x4e fp=0xc039943eb8 sp=0xc039943e80 pc=0x110d1ee
influxd[16407]: github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).DropMeasurement(0xc014b5ba00, 0xc02d9b1070, 0xe, 0x10, 0x0, 0x0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:603 +0x17b fp=0xc039943f60 sp=0xc039943eb8 pc=0x10fc09b
influxd[16407]: github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement.func1(0xc02d9b106c, 0xc00c16da40, 0xc031e98d80, 0xc02d9b1070, 0xe, 0x10)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:609 +0x4e fp=0xc039943fb0 sp=0xc039943f60 pc=0x1108a3e
influxd[16407]: runtime.goexit()
influxd[16407]: /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc039943fb8 sp=0xc039943fb0 pc=0x45e5c1
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:603 +0x11d
influxd[16407]: goroutine 1 [chan receive, 163 minutes]:
influxd[16407]: main.(*Main).Run(0xc0005c3f58, 0xc00003c190, 0x2, 0x2, 0xc0005c3f68, 0x11be253)
influxd[16407]: /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:90 +0x32c
influxd[16407]: main.main()
influxd[16407]: /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45 +0x141
influxd[16407]: goroutine 5 [syscall, 163 minutes]:
influxd[16407]: os/signal.signal_recv(0x0)
influxd[16407]: /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
influxd[16407]: os/signal.loop()
influxd[16407]: /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
influxd[16407]: created by os/signal.init.0
influxd[16407]: /usr/local/go/src/os/signal/signal_unix.go:29 +0x41
influxd[16407]: goroutine 8 [select]:
influxd[16407]: github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0000aab90)
influxd[16407]: /go/src/github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
influxd[16407]: created by github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view.init.0
influxd[16407]: /go/src/github.com/influxdata/influxdb/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57
influxd[16407]: goroutine 11 [IO wait, 163 minutes]:
influxd[16407]: internal/poll.runtime_pollWait(0x7fe1a16adec8, 0x72, 0x0)
influxd[16407]: /usr/local/go/src/runtime/netpoll.go:182 +0x56
influxd[16407]: internal/poll.(*pollDesc).wait(0xc00041c618, 0x72, 0x0, 0x0, 0x15209c3)
influxd[16407]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
influxd[16407]: internal/poll.(*pollDesc).waitRead(...)
influxd[16407]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
influxd[16407]: internal/poll.(*FD).Accept(0xc00041c600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
influxd[16407]: /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba
influxd[16407]: net.(*netFD).accept(0xc00041c600, 0x0, 0x0, 0x0)
influxd[16407]: /usr/local/go/src/net/fd_unix.go:238 +0x42
influxd[16407]: net.(*TCPListener).accept(0xc00013b648, 0x47d0de, 0xc0007d5320, 0x47cffd)
influxd[16407]: /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
influxd[16407]: net.(*TCPListener).Accept(0xc00013b648, 0x0, 0x0, 0x0, 0x0)
influxd[16407]: /usr/local/go/src/net/tcpsock.go:260 +0x48
influxd[16407]: github.com/influxdata/influxdb/tcp.(*Mux).Serve(0xc0007d5320, 0x214c9e0, 0xc00013b648, 0x1529ca7, 0xe)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tcp/mux.go:75 +0x92
influxd[16407]: created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open
influxd[16407]: /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:387 +0x31d
influxd[16407]: goroutine 1030 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc000396240, 0xc019c3c1b0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x25d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc019c3c1b0, 0xc000396240)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x5b
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x131
influxd[16407]: goroutine 1004 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc0003a5e60, 0xc019c77fb0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x25d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc019c77fb0, 0xc0003a5e60)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x5b
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x131
influxd[16407]: goroutine 1017 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc00037d320)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc019c3c0a0, 0xc00037d320)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x51
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x129
influxd[16407]: goroutine 1029 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc000396240)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc019c3c190, 0xc000396240)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x51
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x129
influxd[16407]: goroutine 1019 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc0005410e0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc019c3c0c0, 0xc0005410e0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x51
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x129
influxd[16407]: goroutine 1020 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc0005410e0, 0xc019c3c0e0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x25d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc019c3c0e0, 0xc0005410e0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x5b
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x131
influxd[16407]: goroutine 1014 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc00056d7a0, 0xc019c3c070)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x25d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc019c3c070, 0xc00056d7a0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x5b
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x131
influxd[16407]: goroutine 1013 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc00056d7a0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc019c3c050, 0xc00056d7a0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x51
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x129
influxd[16407]: goroutine 1011 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc00056c120)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc019c3c020, 0xc00056c120)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x51
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x129
influxd[16407]: goroutine 1012 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc00056c120, 0xc019c3c030)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x25d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc019c3c030, 0xc00056c120)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x5b
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x131
influxd[16407]: goroutine 420124 [chan send, 10 minutes]:
influxd[16407]: github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
influxd[16407]: /go/src/github.com/influxdata/influxdb/pkg/limiter/fixed.go:40
influxd[16407]: github.com/influxdata/influxdb/tsdb.(*Store).DeleteMeasurement.func1(0xc01922a000, 0x0, 0x0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:937 +0x72
influxd[16407]: github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1(0xc01b723380, 0xc01c532780, 0xc01922a000)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:995 +0x48
influxd[16407]: created by github.com/influxdata/influxdb/tsdb.(*Store).walkShards
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:994 +0x80
influxd[16407]: goroutine 1006 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc0008167e0, 0xc019c77fd0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x25d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc019c77fd0, 0xc0008167e0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x5b
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x131
influxd[16407]: goroutine 437263 [chan send, 4 minutes]:
influxd[16407]: github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
influxd[16407]: /go/src/github.com/influxdata/influxdb/pkg/limiter/fixed.go:40
influxd[16407]: github.com/influxdata/influxdb/tsdb.(*Store).DeleteMeasurement.func1(0xc00a952240, 0x0, 0x0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:937 +0x72
influxd[16407]: github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1(0xc00c4ded20, 0xc0065f2de0, 0xc00a952240)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:995 +0x48
influxd[16407]: created by github.com/influxdata/influxdb/tsdb.(*Store).walkShards
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:994 +0x80
influxd[16407]: goroutine 420114 [chan send, 10 minutes]:
influxd[16407]: github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
influxd[16407]: /go/src/github.com/influxdata/influxdb/pkg/limiter/fixed.go:40
influxd[16407]: github.com/influxdata/influxdb/tsdb.(*Store).DeleteMeasurement.func1(0xc01b810480, 0x0, 0x0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:937 +0x72
influxd[16407]: github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1(0xc01b723380, 0xc01c532780, 0xc01b810480)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:995 +0x48
influxd[16407]: created by github.com/influxdata/influxdb/tsdb.(*Store).walkShards
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:994 +0x80
influxd[16407]: goroutine 1005 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc0008167e0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11d
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc019c77fc0, 0xc0008167e0)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x51
influxd[16407]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x129
influxd[16407]: goroutine 1003 [select]:
influxd[16407]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc0003a5e60)
influxd[16407]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0x11d