search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.34k stars 3.38k forks source link

[Bug] invalid memory address or nil pointer dereference #11410

Open AtlanCI opened 9 months ago

AtlanCI commented 9 months ago

Describe the bug I'm starting the loki all-in-one program and I'm getting panic.

To Reproduce Steps to reproduce the behavior:

  1. use this config 
    
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9060

ruler: storage: type: local local: directory: /home/server/loki/ruler rule_path: /tmp/scratch enable_api: true

common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /tmp/loki

schema_config: configs:

storage_config: boltdb: directory: /home/server/loki/index

filesystem: directory: /home/server/loki/chunks

limits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h query_timeout: 5m

max_query_lookback: 672h

table_manager: retention_deletes_enabled: true retention_period: 672h

2. Started Loki (2.9.2)

**Expected behavior**
normal startup

**Environment:**
 - Infrastructure:  bare-metal
 - OS: CentOS Linux release 7.6.1810 (Core)

**Screenshots, Promtail config, or terminal output**
**stack log**
```shell
panic: runtime error: invalid memory address or nil pointer dereference
goroutine 475 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2548240, 0xc000b3bbd8})
        /drone/src/pkg/storage/stores/shipper/util/util.go:95 +0x47
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
        /drone/src/pkg/storage/stores/shipper/util/util.go:54 +0x45
panic({0x2548240?, 0xc000b3bbd8?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
        /drone/src/vendor/go.etcd.io/bbolt/freelist.go:265 +0x1a
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
        /drone/src/vendor/go.etcd.io/bbolt/db.go:323 +0xaf
sync.(*Once).doSlow(0xc000b3ed08?, 0x10?)
        /usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000b3eb40?)
        /drone/src/vendor/go.etcd.io/bbolt/db.go:316 +0x45
go.etcd.io/bbolt.Open({0xc000af2dc0, 0x36}, 0x1b6?, 0xc0010fbf48)
        /drone/src/vendor/go.etcd.io/bbolt/db.go:293 +0x458
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
        /drone/src/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000af2dc0, 0x36}, 0xc000b525a0)
        /drone/src/pkg/storage/stores/shipper/util/util.go:62 +0x105
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile in goroutine 1
        /drone/src/pkg/storage/stores/shipper/util/util.go:42 +0x86
yashkundu commented 9 months ago

Hi, @JStickler , I would like to pick it up, can you please assign this to me.

JStickler commented 8 months ago

@yashkundu Sorry just saw that you'd tagged me, I've been out sick. No need to wait to have something assigned to you, you can work on the fix and submit a PR at any time.

matheushent commented 6 months ago

Out of the sudden, I started being hit by this error. Full trace is below:

panic: runtime error: invalid memory address or nil pointer dereference
goroutine 290 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc0000107f8})
    /src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
    /src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc0000107f8})
    /usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
    /src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
    /src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc0004021c8?, 0x10?)
    /usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
    /usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000402000?)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc000649260, 0x2b}, 0x1b6?, 0xc000517f48)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
    /src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000649260, 0x2b}, 0xc0005468a0)
    /src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
    /src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc000649260, 0x2b})
    /src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708c20, 0x20}, 0xc00073e9e0)
    /src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708c20, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
    /src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
    /src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
    /src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
    /src/loki/pkg/loki/loki.go:422 +0x56
main.main()
    /src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creapanic: runtime error: invalid memory address or nil pointer dereference
goroutine 303 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc000010e28})
    /src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
    /src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc000010e28})
    /usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
    /src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
    /src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc000844408?, 0x10?)
    /usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
    /usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000844240?)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc0008367b0, 0x2b}, 0x1b6?, 0xc000517f48)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
    /src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc0008367b0, 0x2b}, 0xc000547aa0)
    /src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
    /src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc0008367b0, 0x2b})
    /src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
    /src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
    /src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
    /src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
    /src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
    /src/loki/pkg/loki/loki.go:422 +0x56
main.main()
    /src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creapanic: runtime error: invalid memory address or nil pointer dereference
goroutine 308 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc000010f60})
    /src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
    /src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc000010f60})
    /usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
    /src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
    /src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc000844f48?, 0x10?)
    /usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
    /usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000844d80?)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc000836a50, 0x2b}, 0x1b6?, 0xc000517f48)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
    /src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000836a50, 0x2b}, 0xc000547e60)
    /src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
    /src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc000836a50, 0x2b})
    /src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
    /src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
    /src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
    /src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
    /src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
    /src/loki/pkg/loki/loki.go:422 +0x56
main.main()
    /src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creapanic: runtime error: invalid memory address or nil pointer dereference
goroutine 311 [running]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.logPanic({0x2187c00, 0xc000011008})
    /src/loki/pkg/storage/stores/shipper/util/util.go:95 +0x4d
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile.func1()
    /src/loki/pkg/storage/stores/shipper/util/util.go:54 +0x4b
panic({0x2187c00, 0xc000011008})
    /usr/local/go/src/runtime/panic.go:884 +0x213
go.etcd.io/bbolt.(*freelist).read(0x0?, 0x0?)
    /src/loki/vendor/go.etcd.io/bbolt/freelist.go:265 +0x26
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
    /src/loki/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc000845608?, 0x10?)
    /usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
    /usr/local/go/src/sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0xc000845440?)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:316 +0x47
go.etcd.io/bbolt.Open({0xc000836c00, 0x2b}, 0x1b6?, 0xc000517f48)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
github.com/grafana/loki/pkg/storage/chunk/client/local.OpenBoltdbFile(...)
    /src/loki/pkg/storage/chunk/client/local/boltdb_index_client.go:404
github.com/grafana/loki/pkg/storage/stores/shipper/util.safeOpenBoltDbFile({0xc000836c00, 0x2b}, 0xc00005f260)
    /src/loki/pkg/storage/stores/shipper/util/util.go:62 +0x10a
created by github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile
    /src/loki/pkg/storage/stores/shipper/util/util.go:42 +0x98

goroutine 1 [chan receive]:
github.com/grafana/loki/pkg/storage/stores/shipper/util.SafeOpenBoltdbFile({0xc000836c00, 0x2b})
    /src/loki/pkg/storage/stores/shipper/util/util.go:43 +0xb0
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
    /src/loki/pkg/storage/stores/shipper/index/table.go:373 +0x3ee
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
    /src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/factory.go:190 +0xa13
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:268 +0xd46
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
    /src/loki/pkg/storage/store.go:164 +0x213
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:147 +0xa3b
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
    /src/loki/pkg/loki/modules.go:632 +0x594
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
    /src/loki/pkg/loki/loki.go:422 +0x56
main.main()
    /src/loki/cmd/loki/main.go:105 +0xe55

goroutine 52 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
github.com/go-kit/log.WithFlushPeriod.func1.1()
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5
created by github.com/go-kit/log.WithFlushPeriod.func1
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde
created by github.com/uber/jaeger-client-go.NewRemoteReporter
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66
created by github.com/grafana/dskit/services.(*BasicService).AddListener
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207
creaunexpected fault address 0x7fa3a956c008
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fa3a956c008 pc=0x197beb1]

goroutine 1 [running]:
runtime.throw({0x234deaf?, 0x18?})
    /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc0009a6bb0 sp=0xc0009a6b80 pc=0x438f3d
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:834 +0x125 fp=0xc0009a6c10 sp=0xc0009a6bb0 pc=0x44fcc5
go.etcd.io/bbolt.(*elemRef).isLeaf(...)
    /src/loki/vendor/go.etcd.io/bbolt/cursor.go:387
go.etcd.io/bbolt.(*Cursor).first(0xc0009a6d20)
    /src/loki/vendor/go.etcd.io/bbolt/cursor.go:170 +0x51 fp=0xc0009a6c70 sp=0xc0009a6c10 pc=0x197beb1
go.etcd.io/bbolt.(*Cursor).First(0xc0009a6d20)
    /src/loki/vendor/go.etcd.io/bbolt/cursor.go:36 +0x118 fp=0xc0009a6cf0 sp=0xc0009a6c70 pc=0x197b718
go.etcd.io/bbolt.(*Bucket).ForEach(0xc000843898, 0xc0009a6d60)
    /src/loki/vendor/go.etcd.io/bbolt/bucket.go:389 +0x70 fp=0xc0009a6d50 sp=0xc0009a6cf0 pc=0x19791d0
go.etcd.io/bbolt.(*Tx).ForEach(0xc000845aa0?, 0xc0009a6da8?)
    /src/loki/vendor/go.etcd.io/bbolt/tx.go:128 +0x49 fp=0xc0009a6d88 sp=0xc0009a6d50 pc=0x19888c9
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir.func1(0xc0009a6de0?)
    /src/loki/pkg/storage/stores/shipper/index/table.go:382 +0x39 fp=0xc0009a6db8 sp=0xc0009a6d88 pc=0x1b96339
go.etcd.io/bbolt.(*DB).View(0xc000836d20?, 0xc0009a6f38)
    /src/loki/vendor/go.etcd.io/bbolt/db.go:772 +0x82 fp=0xc0009a6e20 sp=0xc0009a6db8 pc=0x197f7e2
github.com/grafana/loki/pkg/storage/stores/shipper/index.loadBoltDBsFromDir({0xc000708cc0, 0x20}, 0xc00073e9e0)
    /src/loki/pkg/storage/stores/shipper/index/table.go:381 +0x63b fp=0xc0009a6fa8 sp=0xc0009a6e20 pc=0x1b95edb
github.com/grafana/loki/pkg/storage/stores/shipper/index.LoadTable({0xc000708cc0, 0x20}, {0xc00083e000, 0x45}, {0x7fa3a97f8280, 0xc0009f41e0}, 0x0?, 0x0?)
    /src/loki/pkg/storage/stores/shipper/index/table.go:75 +0x46 fp=0xc0009a7000 sp=0xc0009a6fa8 pc=0x1b928a6
github.com/grafana/loki/pkg/storage/stores/shipper/index.(*TableManager).loadTables(0xc000273290)
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:226 +0x674 fp=0xc0009a71f8 sp=0xc0009a7000 pc=0x1b97f34
github.com/grafana/loki/pkg/storage/stores/shipper/index.NewTableManager({{0xc00083e000, 0x45}, {0xc0008f9800, 0xf}, 0x1176592e000, 0x0}, {0x7fa3a97f8280, 0xc0009f41e0}, {0x27fef90, 0xc0009f19e0})
    /src/loki/pkg/storage/stores/shipper/index/table_manager.go:65 +0x24b fp=0xc0009a7320 sp=0xc0009a71f8 pc=0x1b9662b
github.com/grafana/loki/pkg/storage/stores/shipper.(*indexClient).init(0xc00053a800, {0x280f318, 0xc00009bfe0}, {0x7fa3a97f81f8, 0xc0009c59f8}, 0x7fa3d0ea9f48?, {0xc0009e9d40, 0x1, 0x1}, {0x27fef90, ...})
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:104 +0x2ef fp=0xc0009a7718 sp=0xc0009a7320 pc=0x1b98eef
github.com/grafana/loki/pkg/storage/stores/shipper.NewShipper({{{0xc0008f9800, 0xf}, {0xc0008f9870, 0xa}, {0x234faf7, 0x6}, {0xc0008f9820, 0xf}, 0x2260ff9290000, 0x45d964b800, ...}, ...}, ...)
    /src/loki/pkg/storage/stores/shipper/shipper_index_client.go:73 +0x1b7 fp=0xc0009a79d0 sp=0xc0009a7718 pc=0x1b989b7
github.com/grafana/loki/pkg/storage.NewIndexClient({_, _}, {{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/factory.go:190 +0xa13 fp=0xc0009a8a10 sp=0xc0009a79d0 pc=0x1bd58f3
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod(0xc000911000, {{0x17465dcc000}, {0xc0008f97a0, 0xe}, {0xc0008f9770, 0xa}, {0xc0008f9788, 0x3}, {{0xc0008f9750, 0xb}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:268 +0xd46 fp=0xc0009ab558 sp=0xc0009a8a10 pc=0x1bdbe06
github.com/grafana/loki/pkg/storage.(*store).init(0xc000911000)
    /src/loki/pkg/storage/store.go:164 +0x213 fp=0xc0009ab778 sp=0xc0009ab558 pc=0x1bda973
github.com/grafana/loki/pkg/storage.NewStore({{{{0x0}, 0x4000000000000000, 0x4024000000000000, {{...}, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, {...}, {...}, {...}, ...}, ...}, ...}, ...}, ...)
    /src/loki/pkg/storage/store.go:147 +0xa3b fp=0xc0009abbe0 sp=0xc0009ab778 pc=0x1bda5fb
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000340000)
    /src/loki/pkg/loki/modules.go:632 +0x594 fp=0xc0009adb90 sp=0xc0009abbe0 pc=0x1d58214
github.com/grafana/loki/pkg/loki.(*Loki).initStore-fm()
    <autogenerated>:1 +0x26 fp=0xc0009adba8 sp=0xc0009adb90 pc=0x1d62dc6
github.com/grafana/dskit/modules.(*Manager).initModule(0xc0005310f8, {0x7ffd29592b18, 0x8}, 0x84f576?, 0x5666e6?)
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x20a fp=0xc0009adcb0 sp=0xc0009adba8 pc=0x1a5b5ea
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc00073e700?, {0xc0008fb170, 0x1, 0x20a8da0?})
    /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 +0xf8 fp=0xc0009addd0 sp=0xc0009adcb0 pc=0x1a5b358
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000340000, {0xc000866b40?})
    /src/loki/pkg/loki/loki.go:422 +0x56 fp=0xc0009adf48 sp=0xc0009addd0 pc=0x1d4da96
main.main()
    /src/loki/cmd/loki/main.go:105 +0xe55 fp=0xc0009b1f80 sp=0xc0009adf48 pc=0x1d64455
runtime.main()
    /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc0009b1fe0 sp=0xc0009b1f80 pc=0x43b827
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0009b1fe8 sp=0xc0009b1fe0 pc=0x46fc41

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007afb0 sp=0xc00007af90 pc=0x43bc56
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc00007afe0 sp=0xc00007afb0 pc=0x43ba90
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007afe8 sp=0xc00007afe0 pc=0x46fc41
created by runtime.init.6
    /usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007b780 sp=0xc00007b760 pc=0x43bc56
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc00007b7c8 sp=0xc00007b780 pc=0x425dde
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00007b7e0 sp=0xc00007b7c8 pc=0x41b046
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007b7e8 sp=0xc00007b7e0 pc=0x46fc41
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc000098000?, 0x27d62e8?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007bf70 sp=0xc00007bf50 pc=0x43bc56
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x3a079c0)
    /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc00007bfa0 sp=0xc00007bf70 pc=0x423cb3
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc00007bfc8 sp=0xc00007bfa0 pc=0x4242a5
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x41afe6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x46fc41
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x43bfd2?, 0x43b9c5?, 0x0?, 0x0?, 0xc00007a770?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007a628 sp=0xc00007a608 pc=0x43bc56
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00007a7e0 sp=0xc00007a628 pc=0x41a087
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007a7e8 sp=0xc00007a7e0 pc=0x46fc41
created by runtime.createfing
    /usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x4c7772aa202?, 0xc00009cba0?, 0xa0?, 0xc7?, 0xc00007c7a8?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007c750 sp=0xc00007c730 pc=0x43bc56
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00007c7e0 sp=0xc00007c750 pc=0x41cdb1
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x46fc41
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x4c77768cddb?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000076750 sp=0xc000076730 pc=0x43bc56
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000767e0 sp=0xc000076750 pc=0x41cdb1
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000767e8 sp=0xc0000767e0 pc=0x46fc41
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 52 [select]:
runtime.gopark(0xc000077f48?, 0x2?, 0x0?, 0x0?, 0xc000077f14?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000077d90 sp=0xc000077d70 pc=0x43bc56
runtime.selectgo(0xc000077f48, 0xc000077f10, 0x0?, 0x0, 0x3?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000077ed0 sp=0xc000077d90 pc=0x44bf1e
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xb9 fp=0xc000077fe0 sp=0xc000077ed0 pc=0x16855f9
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x46fc41
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
    /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x136

goroutine 54 [select]:
runtime.gopark(0xc000078788?, 0x3?, 0x90?, 0x0?, 0xc000078772?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000785f8 sp=0xc0000785d8 pc=0x43bc56
runtime.selectgo(0xc000078788, 0xc00007876c, 0xc0004cc200?, 0x0, 0xc000078788?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000078738 sp=0xc0000785f8 pc=0x44bf1e
go.opencensus.io/stats/view.(*worker).start(0xc0004cc200)
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:276 +0xad fp=0xc0000787c8 sp=0xc000078738 pc=0x16fcfad
go.opencensus.io/stats/view.init.0.func1()
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x26 fp=0xc0000787e0 sp=0xc0000787c8 pc=0x16fc226
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x46fc41
created by go.opencensus.io/stats/view.init.0
    /src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 121 [select]:
runtime.gopark(0xc0002ccf58?, 0x2?, 0xf0?, 0x5f?, 0xc0002ccf4c?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002ccdd8 sp=0xc0002ccdb8 pc=0x43bc56
runtime.selectgo(0xc0002ccf58, 0xc0002ccf48, 0xc0002ccf40?, 0x0, 0xc000969408?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002ccf18 sp=0xc0002ccdd8 pc=0x44bf1e
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00087e0d0, 0xc000969400)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x89 fp=0xc0002ccf88 sp=0xc0002ccf18 pc=0x8d1989
github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00087e0d0)
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x6d fp=0xc0002ccfc8 sp=0xc0002ccf88 pc=0x8d184d
github.com/uber/jaeger-client-go.NewRemotelyControlledSampler.func1()
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x26 fp=0xc0002ccfe0 sp=0xc0002ccfc8 pc=0x8d0f46
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002ccfe8 sp=0xc0002ccfe0 pc=0x46fc41
created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler
    /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x15b

goroutine 107 [chan receive]:
runtime.gopark(0xc0009a1ed0?, 0x441d75?, 0x18?, 0x1f?, 0x458e6f?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0009a1ec0 sp=0xc0009a1ea0 pc=0x43bc56
runtime.chanrecv(0xc0008b09c0, 0xc0009a1fb0, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0009a1f50 sp=0xc0009a1ec0 pc=0x406f9d
runtime.chanrecv2(0x5f5e100?, 0x853125?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0009a1f78 sp=0xc0009a1f50 pc=0x406ad8
github.com/go-kit/log.WithFlushPeriod.func1.1()
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:74 +0xa5 fp=0xc0009a1fe0 sp=0xc0009a1f78 pc=0x564c25
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0009a1fe8 sp=0xc0009a1fe0 pc=0x46fc41
created by github.com/go-kit/log.WithFlushPeriod.func1
    /src/loki/vendor/github.com/go-kit/log/line_buffer.go:70 +0x6a

goroutine 123 [select]:
runtime.gopark(0xc0002cdf88?, 0x2?, 0x4?, 0x0?, 0xc0002cdf6c?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002cddf0 sp=0xc0002cddd0 pc=0x43bc56
runtime.selectgo(0xc0002cdf88, 0xc0002cdf68, 0xc000536480?, 0x0, 0x405d3d?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002cdf30 sp=0xc0002cddf0 pc=0x44bf1e
github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000534a10, 0xc000536480?)
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xbc fp=0xc0002cdfc0 sp=0xc0002cdf30 pc=0x8bbefc
github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn.func1()
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x2a fp=0xc0002cdfe0 sp=0xc0002cdfc0 pc=0x8bbe0a
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cdfe8 sp=0xc0002cdfe0 pc=0x46fc41
created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn
    /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 124 [select]:
runtime.gopark(0xc000078f78?, 0x2?, 0x60?, 0x78?, 0xc000078f14?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000078d80 sp=0xc000078d60 pc=0x43bc56
runtime.selectgo(0xc000078f78, 0xc000078f10, 0xc000078f28?, 0x0, 0x28042c8?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000078ec0 sp=0xc000078d80 pc=0x44bf1e
github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000537680)
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xde fp=0xc000078fc8 sp=0xc000078ec0 pc=0x8ce3fe
github.com/uber/jaeger-client-go.NewRemoteReporter.func1()
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x26 fp=0xc000078fe0 sp=0xc000078fc8 pc=0x8ce086
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x46fc41
created by github.com/uber/jaeger-client-go.NewRemoteReporter
    /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x250

goroutine 180 [chan receive]:
runtime.gopark(0xc0009ff200?, 0xc00009aae0?, 0xe0?, 0xaa?, 0xc0002cefa8?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002ceee8 sp=0xc0002ceec8 pc=0x43bc56
runtime.chanrecv(0xc0009e1ec0, 0xc0002cefc8, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0002cef78 sp=0xc0002ceee8 pc=0x406f9d
runtime.chanrecv2(0x2807380?, 0xc000667890?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0002cefa0 sp=0xc0002cef78 pc=0x406ad8
github.com/grafana/dskit/services.(*BasicService).AddListener.func1()
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:344 +0x66 fp=0xc0002cefe0 sp=0xc0002cefa0 pc=0x9edca6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cefe8 sp=0xc0002cefe0 pc=0x46fc41
created by github.com/grafana/dskit/services.(*BasicService).AddListener
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:343 +0x1ca

goroutine 181 [select]:
runtime.gopark(0xc0014f3e88?, 0x2?, 0x8?, 0xdc?, 0xc0014f3e3c?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014f3ca0 sp=0xc0014f3c80 pc=0x43bc56
runtime.selectgo(0xc0014f3e88, 0xc0014f3e38, 0xc0000a7a90?, 0x0, 0xc0009e1f18?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0014f3de0 sp=0xc0014f3ca0 pc=0x44bf1e
github.com/grafana/dskit/kv/memberlist.(*KV).running(0xc000990c00, {0x28042c8, 0xc0000a7a90})
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/memberlist_client.go:492 +0x18e fp=0xc0014f3ee0 sp=0xc0014f3de0 pc=0xe74c6e
github.com/grafana/dskit/kv/memberlist.(*KV).running-fm({0x28042c8?, 0xc0000a7a90?})
    <autogenerated>:1 +0x39 fp=0xc0014f3f08 sp=0xc0014f3ee0 pc=0xe874f9
github.com/grafana/dskit/services.(*BasicService).main(0xc000929f40)
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:190 +0x1e8 fp=0xc0014f3fc8 sp=0xc0014f3f08 pc=0x9eca48
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1.2()
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x26 fp=0xc0014f3fe0 sp=0xc0014f3fc8 pc=0x9ec666
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0014f3fe8 sp=0xc0014f3fe0 pc=0x46fc41
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1
    /src/loki/vendor/github.com/grafana/dskit/services/basic_service.go:119 +0x10a

goroutine 200 [select]:
runtime.gopark(0xc0002c9f80?, 0x2?, 0x40?, 0xd0?, 0xc0002c9f64?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002c9de0 sp=0xc0002c9dc0 pc=0x43bc56
runtime.selectgo(0xc0002c9f80, 0xc0002c9f60, 0xc0002c9f58?, 0x0, 0xc0009f9680?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002c9f20 sp=0xc0002c9de0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2d00, 0x195?, 0xc0009f9680?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9 fp=0xc0002c9fb8 sp=0xc0002c9f20 pc=0x133e7b9
github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache.func1()
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x2e fp=0xc0002c9fe0 sp=0xc0002c9fb8 pc=0x133e6ae
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002c9fe8 sp=0xc0002c9fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 213 [select]:
runtime.gopark(0xc0002cbf80?, 0x2?, 0xa0?, 0xdb?, 0xc0002cbf64?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002cbde0 sp=0xc0002cbdc0 pc=0x43bc56
runtime.selectgo(0xc0002cbf80, 0xc0002cbf60, 0xc0002cbf58?, 0x0, 0xc0009f9e60?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002cbf20 sp=0xc0002cbde0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/cache.(*FifoCache).runPruneJob(0xc0009c2dd0, 0x0?, 0xc0009f9e60?)
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:225 +0xd9 fp=0xc0002cbfb8 sp=0xc0002cbf20 pc=0x133e7b9
github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache.func1()
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x2e fp=0xc0002cbfe0 sp=0xc0002cbfb8 pc=0x133e6ae
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cbfe8 sp=0xc0002cbfe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/cache.NewFifoCache
    /src/loki/pkg/storage/chunk/cache/fifo_cache.go:214 +0x1194

goroutine 226 [chan receive]:
runtime.gopark(0xc0002cace8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0002cad50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002cacd0 sp=0xc0002cacb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0002caf18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0002cad60 sp=0xc0002cacd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0008f94b0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0002cad88 sp=0xc0002cad60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0002cafc8 sp=0xc0002cad88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0002cafe0 sp=0xc0002cafc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cafe8 sp=0xc0002cafe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 227 [chan receive]:
runtime.gopark(0xc0003a2ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a2d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a2cd0 sp=0xc0003a2cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a2f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a2d60 sp=0xc0003a2cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0004aaa00?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a2d88 sp=0xc0003a2d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a2fc8 sp=0xc0003a2d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a2fe0 sp=0xc0003a2fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a2fe8 sp=0xc0003a2fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 228 [chan receive]:
runtime.gopark(0xc0003a0ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a0d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a0cd0 sp=0xc0003a0cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a0f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a0d60 sp=0xc0003a0cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0008f94b0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a0d88 sp=0xc0003a0d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a0fc8 sp=0xc0003a0d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a0fe0 sp=0xc0003a0fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a0fe8 sp=0xc0003a0fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 229 [chan receive]:
runtime.gopark(0xc0003a1ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a1d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a1cd0 sp=0xc0003a1cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a1f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a1d60 sp=0xc0003a1cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0xc0008f94b0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a1d88 sp=0xc0003a1d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a1fc8 sp=0xc0003a1d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a1fe0 sp=0xc0003a1fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a1fe8 sp=0xc0003a1fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 230 [chan receive]:
runtime.gopark(0xc00039fce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00039fd50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00039fcd0 sp=0xc00039fcb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00039ff18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00039fd60 sp=0xc00039fcd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00039fd88 sp=0xc00039fd60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00039ffc8 sp=0xc00039fd88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00039ffe0 sp=0xc00039ffc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00039ffe8 sp=0xc00039ffe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 231 [chan receive]:
runtime.gopark(0xc000569ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000569d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000569cd0 sp=0xc000569cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000569f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000569d60 sp=0xc000569cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000569d88 sp=0xc000569d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000569fc8 sp=0xc000569d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000569fe0 sp=0xc000569fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000569fe8 sp=0xc000569fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 232 [chan receive]:
runtime.gopark(0xc00056ace8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00056ad50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00056acd0 sp=0xc00056acb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00056af18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00056ad60 sp=0xc00056acd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00056ad88 sp=0xc00056ad60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00056afc8 sp=0xc00056ad88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00056afe0 sp=0xc00056afc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00056afe8 sp=0xc00056afe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 233 [chan receive]:
runtime.gopark(0xc00056bce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00056bd50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00056bcd0 sp=0xc00056bcb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00056bf18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00056bd60 sp=0xc00056bcd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00056bd88 sp=0xc00056bd60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00056bfc8 sp=0xc00056bd88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00056bfe0 sp=0xc00056bfc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00056bfe8 sp=0xc00056bfe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 234 [chan receive]:
runtime.gopark(0xc000568ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000568d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000568cd0 sp=0xc000568cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000568f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000568d60 sp=0xc000568cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000568d88 sp=0xc000568d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000568fc8 sp=0xc000568d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000568fe0 sp=0xc000568fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000568fe8 sp=0xc000568fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 235 [chan receive]:
runtime.gopark(0xc000565ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000565d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000565cd0 sp=0xc000565cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000565f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000565d60 sp=0xc000565cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000565d88 sp=0xc000565d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000565fc8 sp=0xc000565d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000565fe0 sp=0xc000565fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 236 [chan receive]:
runtime.gopark(0xc000564ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000564d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000564cd0 sp=0xc000564cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000564f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000564d60 sp=0xc000564cd0 pc=0x406f9d
runtime.chanrecv2(0x1?, 0xc0003c7500?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000564d88 sp=0xc000564d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000564fc8 sp=0xc000564d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000564fe0 sp=0xc000564fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000564fe8 sp=0xc000564fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 237 [chan receive]:
runtime.gopark(0xc000566ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000566d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000566cd0 sp=0xc000566cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000566f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000566d60 sp=0xc000566cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000566d88 sp=0xc000566d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000566fc8 sp=0xc000566d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000566fe0 sp=0xc000566fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000566fe8 sp=0xc000566fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 238 [chan receive]:
runtime.gopark(0xc000567ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000567d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000567cd0 sp=0xc000567cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000567f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000567d60 sp=0xc000567cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000567d88 sp=0xc000567d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000567fc8 sp=0xc000567d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000567fe0 sp=0xc000567fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000567fe8 sp=0xc000567fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 239 [chan receive]:
runtime.gopark(0xc000351ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc000351d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000351cd0 sp=0xc000351cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc000351f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000351d60 sp=0xc000351cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000351d88 sp=0xc000351d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc000351fc8 sp=0xc000351d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc000351fe0 sp=0xc000351fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000351fe8 sp=0xc000351fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 240 [chan receive]:
runtime.gopark(0xc0003a5ce8?, 0x41b411?, 0xc0?, 0xb0?, 0xc0003a5d50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a5cd0 sp=0xc0003a5cb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc0003a5f18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0003a5d60 sp=0xc0003a5cd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0003a5d88 sp=0xc0003a5d60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc0003a5fc8 sp=0xc0003a5d88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc0003a5fe0 sp=0xc0003a5fc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a5fe8 sp=0xc0003a5fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 241 [chan receive]:
runtime.gopark(0xc00039ece8?, 0x41b411?, 0xc0?, 0xb0?, 0xc00039ed50?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00039ecd0 sp=0xc00039ecb0 pc=0x43bc56
runtime.chanrecv(0xc0009ea720, 0xc00039ef18, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00039ed60 sp=0xc00039ecd0 pc=0x406f9d
runtime.chanrecv2(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00039ed88 sp=0xc00039ed60 pc=0x406ad8
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).worker(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:142 +0x207 fp=0xc00039efc8 sp=0xc00039ed88 pc=0x1378187
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func1()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x26 fp=0xc00039efe0 sp=0xc00039efc8 pc=0x1377be6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00039efe8 sp=0xc00039efe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:91 +0x1bf

goroutine 242 [select]:
runtime.gopark(0xc00099c778?, 0x2?, 0x0?, 0x0?, 0xc00099c744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00099c5b0 sp=0xc00099c590 pc=0x43bc56
runtime.selectgo(0xc00099c778, 0xc00099c740, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00099c6f0 sp=0xc00099c5b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc00099c7c8 sp=0xc00099c6f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc00099c7e0 sp=0xc00099c7c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00099c7e8 sp=0xc00099c7e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 243 [select]:
runtime.gopark(0xc00099cf78?, 0x2?, 0x0?, 0x0?, 0xc00099cf44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00099cdb0 sp=0xc00099cd90 pc=0x43bc56
runtime.selectgo(0xc00099cf78, 0xc00099cf40, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00099cef0 sp=0xc00099cdb0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc00099cfc8 sp=0xc00099cef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc00099cfe0 sp=0xc00099cfc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00099cfe8 sp=0xc00099cfe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 244 [select]:
runtime.gopark(0xc0003c3778?, 0x2?, 0x0?, 0x0?, 0xc0003c3744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c35b0 sp=0xc0003c3590 pc=0x43bc56
runtime.selectgo(0xc0003c3778, 0xc0003c3740, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c36f0 sp=0xc0003c35b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c37c8 sp=0xc0003c36f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c37e0 sp=0xc0003c37c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c37e8 sp=0xc0003c37e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 245 [select]:
runtime.gopark(0xc0003c3f78?, 0x2?, 0x12?, 0x0?, 0xc0003c3f44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c3db0 sp=0xc0003c3d90 pc=0x43bc56
runtime.selectgo(0xc0003c3f78, 0xc0003c3f40, 0x0?, 0x0, 0xd40010?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c3ef0 sp=0xc0003c3db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c3fc8 sp=0xc0003c3ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c3fe0 sp=0xc0003c3fc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c3fe8 sp=0xc0003c3fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 246 [select]:
runtime.gopark(0xc0003c4778?, 0x2?, 0x2?, 0x0?, 0xc0003c4744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c45b0 sp=0xc0003c4590 pc=0x43bc56
runtime.selectgo(0xc0003c4778, 0xc0003c4740, 0xd40010?, 0x0, 0x1?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c46f0 sp=0xc0003c45b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c47c8 sp=0xc0003c46f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c47e0 sp=0xc0003c47c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c47e8 sp=0xc0003c47e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 247 [select]:
runtime.gopark(0xc0003c4f78?, 0x2?, 0x0?, 0x0?, 0xc0003c4f44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c4db0 sp=0xc0003c4d90 pc=0x43bc56
runtime.selectgo(0xc0003c4f78, 0xc0003c4f40, 0xc0003c4e70?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c4ef0 sp=0xc0003c4db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c4fc8 sp=0xc0003c4ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c4fe0 sp=0xc0003c4fc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c4fe8 sp=0xc0003c4fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 248 [select]:
runtime.gopark(0xc0003c5778?, 0x2?, 0x0?, 0x0?, 0xc0003c5744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c55b0 sp=0xc0003c5590 pc=0x43bc56
runtime.selectgo(0xc0003c5778, 0xc0003c5740, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c56f0 sp=0xc0003c55b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c57c8 sp=0xc0003c56f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c57e0 sp=0xc0003c57c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c57e8 sp=0xc0003c57e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 249 [select]:
runtime.gopark(0xc0003c5f78?, 0x2?, 0x10?, 0x0?, 0xc0003c5f44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c5db0 sp=0xc0003c5d90 pc=0x43bc56
runtime.selectgo(0xc0003c5f78, 0xc0003c5f40, 0x0?, 0x0, 0x12?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c5ef0 sp=0xc0003c5db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c5fc8 sp=0xc0003c5ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c5fe0 sp=0xc0003c5fc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c5fe8 sp=0xc0003c5fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 250 [select]:
runtime.gopark(0xc0003c6778?, 0x2?, 0x1?, 0x0?, 0xc0003c6744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c65b0 sp=0xc0003c6590 pc=0x43bc56
runtime.selectgo(0xc0003c6778, 0xc0003c6740, 0x12?, 0x0, 0x2?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c66f0 sp=0xc0003c65b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c67c8 sp=0xc0003c66f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c67e0 sp=0xc0003c67c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c67e8 sp=0xc0003c67e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 251 [select]:
runtime.gopark(0xc0003c6f78?, 0x2?, 0x0?, 0x0?, 0xc0003c6f44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c6db0 sp=0xc0003c6d90 pc=0x43bc56
runtime.selectgo(0xc0003c6f78, 0xc0003c6f40, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c6ef0 sp=0xc0003c6db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c6fc8 sp=0xc0003c6ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c6fe0 sp=0xc0003c6fc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c6fe8 sp=0xc0003c6fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 252 [select]:
runtime.gopark(0xc000447778?, 0x2?, 0xe0?, 0x4e?, 0xc000447744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004475b0 sp=0xc000447590 pc=0x43bc56
runtime.selectgo(0xc000447778, 0xc000447740, 0xc00030d2b0?, 0x0, 0xc00030d370?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0004476f0 sp=0xc0004475b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0004477c8 sp=0xc0004476f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0004477e0 sp=0xc0004477c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004477e8 sp=0xc0004477e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 253 [select]:
runtime.gopark(0xc000447f78?, 0x2?, 0x40?, 0x4?, 0xc000447f44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000447db0 sp=0xc000447d90 pc=0x43bc56
runtime.selectgo(0xc000447f78, 0xc000447f40, 0xc00030c520?, 0x0, 0xc00030c540?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000447ef0 sp=0xc000447db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc000447fc8 sp=0xc000447ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc000447fe0 sp=0xc000447fc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000447fe8 sp=0xc000447fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 254 [select]:
runtime.gopark(0xc0003c0778?, 0x2?, 0x0?, 0x0?, 0xc0003c0744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c05b0 sp=0xc0003c0590 pc=0x43bc56
runtime.selectgo(0xc0003c0778, 0xc0003c0740, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c06f0 sp=0xc0003c05b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c07c8 sp=0xc0003c06f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c07e0 sp=0xc0003c07c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c07e8 sp=0xc0003c07e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 255 [select]:
runtime.gopark(0xc0003c0f78?, 0x2?, 0x0?, 0x0?, 0xc0003c0f44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c0db0 sp=0xc0003c0d90 pc=0x43bc56
runtime.selectgo(0xc0003c0f78, 0xc0003c0f40, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c0ef0 sp=0xc0003c0db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c0fc8 sp=0xc0003c0ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c0fe0 sp=0xc0003c0fc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c0fe8 sp=0xc0003c0fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 256 [select]:
runtime.gopark(0xc0003c1778?, 0x2?, 0x0?, 0x0?, 0xc0003c1744?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c15b0 sp=0xc0003c1590 pc=0x43bc56
runtime.selectgo(0xc0003c1778, 0xc0003c1740, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c16f0 sp=0xc0003c15b0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c17c8 sp=0xc0003c16f0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c17e0 sp=0xc0003c17c8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c17e8 sp=0xc0003c17e0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 257 [select]:
runtime.gopark(0xc0003c1f78?, 0x2?, 0x0?, 0x0?, 0xc0003c1f44?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c1db0 sp=0xc0003c1d90 pc=0x43bc56
runtime.selectgo(0xc0003c1f78, 0xc0003c1f40, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c1ef0 sp=0xc0003c1db0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/chunk/fetcher.(*Fetcher).asyncWriteBackCacheQueueProcessLoop(0xc000737860)
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:116 +0xa5 fp=0xc0003c1fc8 sp=0xc0003c1ef0 pc=0x1377cc5
github.com/grafana/loki/pkg/storage/chunk/fetcher.New.func2()
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x26 fp=0xc0003c1fe0 sp=0xc0003c1fc8 pc=0x1377b86
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c1fe8 sp=0xc0003c1fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/chunk/fetcher.New
    /src/loki/pkg/storage/chunk/fetcher/fetcher.go:98 +0x259

goroutine 263 [select]:
runtime.gopark(0xc0003a3f88?, 0x2?, 0x68?, 0xdd?, 0xc0003a3f4c?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003a3dd0 sp=0xc0003a3db0 pc=0x43bc56
runtime.selectgo(0xc0003a3f88, 0xc0003a3f48, 0xc000064428?, 0x0, 0xc0003a3f58?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003a3f10 sp=0xc0003a3dd0 pc=0x44bf1e
github.com/grafana/loki/pkg/storage/stores/indexshipper/uploads.(*tableManager).loop(0xc000741dc0)
    /src/loki/pkg/storage/stores/indexshipper/uploads/table_manager.go:65 +0x15d fp=0xc0003a3fc8 sp=0xc0003a3f10 pc=0x1b8e33d
github.com/grafana/loki/pkg/storage/stores/indexshipper/uploads.NewTableManager.func1()
    /src/loki/pkg/storage/stores/indexshipper/uploads/table_manager.go:51 +0x26 fp=0xc0003a3fe0 sp=0xc0003a3fc8 pc=0x1b8e1a6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003a3fe8 sp=0xc0003a3fe0 pc=0x46fc41
created by github.com/grafana/loki/pkg/storage/stores/indexshipper/uploads.NewTableManager
    /src/loki/pkg/storage/stores/indexshipper/uploads/table_manager.go:51 +0x294

goroutine 141 [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002c8cd0 sp=0xc0002c8cb0 pc=0x43bc56
runtime.netpollblock(0x0?, 0x40482f?, 0x0?)
    /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0002c8d08 sp=0xc0002c8cd0 pc=0x434297
internal/poll.runtime_pollWait(0x7fa3a9b0db38, 0x72)
    /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0002c8d28 sp=0xc0002c8d08 pc=0x469d69
internal/poll.(*pollDesc).wait(0xc0001c0800?, 0x26c0?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0002c8d50 sp=0xc0002c8d28 pc=0x4e5f12
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0001c0800)
    /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0002c8df8 sp=0xc0002c8d50 pc=0x4eb81d
net.(*netFD).accept(0xc0001c0800)
    /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc0002c8eb0 sp=0xc0002c8df8 pc=0x6ee9b5
net.(*TCPListener).accept(0xc0002f5308)
    /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0002c8ed8 sp=0xc0002c8eb0 pc=0x70a1a5
net.(*TCPListener).Accept(0xc0002f5308)
    /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc0002c8f08 sp=0xc0002c8ed8 pc=0x708f5d
github.com/grafana/dskit/kv/memberlist.(*TCPTransport).tcpListen(0xc000846400, {0x2801bd0, 0xc0002f5308})
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/tcp_transport.go:202 +0xa9 fp=0xc0002c8fb8 sp=0xc0002c8f08 pc=0xe81ac9
github.com/grafana/dskit/kv/memberlist.NewTCPTransport.func2()
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/tcp_transport.go:178 +0x2e fp=0xc0002c8fe0 sp=0xc0002c8fb8 pc=0xe819ae
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002c8fe8 sp=0xc0002c8fe0 pc=0x46fc41
created by github.com/grafana/dskit/kv/memberlist.NewTCPTransport
    /src/loki/vendor/github.com/grafana/dskit/kv/memberlist/tcp_transport.go:178 +0x7d3

goroutine 142 [select]:
runtime.gopark(0xc000079f98?, 0x2?, 0x0?, 0x0?, 0xc000079f7c?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000079e00 sp=0xc000079de0 pc=0x43bc56
runtime.selectgo(0xc000079f98, 0xc000079f78, 0x405d3d?, 0x0, 0xc000079fa0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000079f40 sp=0xc000079e00 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).streamListen(0xc0000d5a20)
    /src/loki/vendor/github.com/hashicorp/memberlist/net.go:222 +0x96 fp=0xc000079fc8 sp=0xc000079f40 pc=0xe4e8d6
github.com/hashicorp/memberlist.newMemberlist.func3()
    /src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:227 +0x26 fp=0xc000079fe0 sp=0xc000079fc8 pc=0xe4b3a6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.newMemberlist
    /src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:227 +0x9c5

goroutine 143 [select]:
runtime.gopark(0xc00007cf98?, 0x2?, 0x0?, 0x0?, 0xc00007cf8c?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007ce00 sp=0xc00007cde0 pc=0x43bc56
runtime.selectgo(0xc00007cf98, 0xc00007cf88, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00007cf40 sp=0xc00007ce00 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).packetListen(0xc0000d5a20)
    /src/loki/vendor/github.com/hashicorp/memberlist/net.go:346 +0xc5 fp=0xc00007cfc8 sp=0xc00007cf40 pc=0xe4fc25
github.com/hashicorp/memberlist.newMemberlist.func4()
    /src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:228 +0x26 fp=0xc00007cfe0 sp=0xc00007cfc8 pc=0xe4b346
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.newMemberlist
    /src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:228 +0xa0a

goroutine 144 [select]:
runtime.gopark(0xc00007d718?, 0x2?, 0x0?, 0x0?, 0xc00007d714?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007d598 sp=0xc00007d578 pc=0x43bc56
runtime.selectgo(0xc00007d718, 0xc00007d710, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00007d6d8 sp=0xc00007d598 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).packetHandler(0xc0000d5a20)
    /src/loki/vendor/github.com/hashicorp/memberlist/net.go:490 +0x77 fp=0xc00007d7c8 sp=0xc00007d6d8 pc=0xe50d17
github.com/hashicorp/memberlist.newMemberlist.func5()
    /src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:229 +0x26 fp=0xc00007d7e0 sp=0xc00007d7c8 pc=0xe4b2e6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007d7e8 sp=0xc00007d7e0 pc=0x46fc41
created by github.com/hashicorp/memberlist.newMemberlist
    /src/loki/vendor/github.com/hashicorp/memberlist/memberlist.go:229 +0xa4d

goroutine 273 [select]:
runtime.gopark(0xc00007df78?, 0x2?, 0x40?, 0x0?, 0xc00007df54?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007ddd8 sp=0xc00007ddb8 pc=0x43bc56
runtime.selectgo(0xc00007df78, 0xc00007df50, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00007df18 sp=0xc00007ddd8 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0x0?, 0x12a05f200, 0xc000480240, 0xc0009eac00, 0xc0002bafc0)
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:149 +0xb7 fp=0xc00007dfa8 sp=0xc00007df18 pc=0xe5c517
github.com/hashicorp/memberlist.(*Memberlist).schedule.func2()
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:121 +0x36 fp=0xc00007dfe0 sp=0xc00007dfa8 pc=0xe5c3d6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:121 +0x218

goroutine 274 [select]:
runtime.gopark(0xc0003c7f90?, 0x2?, 0x0?, 0x0?, 0xc0003c7f6c?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c7de8 sp=0xc0003c7dc8 pc=0x43bc56
runtime.selectgo(0xc0003c7f90, 0xc0003c7f68, 0x0?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c7f28 sp=0xc0003c7de8 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).pushPullTrigger(0xc0000d5a20, 0xc0009eac00)
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:173 +0xb7 fp=0xc0003c7fc0 sp=0xc0003c7f28 pc=0xe5c697
github.com/hashicorp/memberlist.(*Memberlist).schedule.func3()
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:127 +0x2a fp=0xc0003c7fe0 sp=0xc0003c7fc0 pc=0xe5c36a
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c7fe8 sp=0xc0003c7fe0 pc=0x46fc41
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:127 +0x345

goroutine 275 [select]:
runtime.gopark(0xc0003c7778?, 0x2?, 0xc0?, 0x82?, 0xc0003c7754?)
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0003c75d8 sp=0xc0003c75b8 pc=0x43bc56
runtime.selectgo(0xc0003c7778, 0xc0003c7750, 0x0?, 0x0, 0xc0003c7750?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0003c7718 sp=0xc0003c75d8 pc=0x44bf1e
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc000469dd0?, 0xbebc200, 0xc0004802a0, 0xc0009eac00, 0xc0002bb010)
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:149 +0xb7 fp=0xc0003c77a8 sp=0xc0003c7718 pc=0xe5c517
github.com/hashicorp/memberlist.(*Memberlist).schedule.func4()
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:133 +0x36 fp=0xc0003c77e0 sp=0xc0003c77a8 pc=0xe5c316
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0003c77e8 sp=0xc0003c77e0 pc=0x46fc41
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
    /src/loki/vendor/github.com/hashicorp/memberlist/state.go:133 +0x498

My environment: EKS 1.23 CPU arch: amd64

dbg-raghulkrishna commented 6 months ago

any update on this? i am facing the same issue

hinorashi commented 4 months ago

encountering the same here:

hinorashi commented 4 months ago

encountering the same here:

  • EKS 1.29
  • helm chart 5.48.0
  • loki 2.9.6

I fixed it but without any help from loki pod log or event: the disk is full -> extend the PV