Execute source panic after some time, panic: runtime error: invalid memory address or nil pointer dereference

[robsliwi]

Steps to reproduce: Really haven't found any way to reproduce the behavior but I'll try to keep an eye on it and add information as I good around it

1. Use Telegraf with OPC UA plugin for data collection
2. Wait some time

Expected behavior: Most of the time everything is working fine and all the values can be written, retrieved, and such. We're really happy using InfluxDB.

Actual behavior: We stumbled across missing points in our measurements. A small investigation revealed the following error in the InfluxDB systemd service:

Dec 14 10:08:12 chub.localdomain influxd-systemd-start.sh[77435]: ts=2021-12-14T15:08:12.267358Z lvl=info msg="Execute source panic" log_id=0XR0e8O0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 64756223 [running]:
runtime/debug.Stack()
    /home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.131.0/execute/executor.go:286 +0x205
panic({0x35c9e00, 0x49e7930})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x35e57c0, 0xc006f52a80})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc0031cc060, {0x39711d8, 0xc00208a000}, 0xc000f45bd0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x394bf00, {0x39711d8, 0xc00208a000}, 0xc006f526c0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc001fe7600, 0x61b8b35c)
    /home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0053c4680, {0x39711d8, 0xc00208a000}, {0x3942728, 0xc001fe7600}, 0xc004ca4fe0)
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0053c4680, {0x39711d8, 0xc00208a000})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0053c4680, {0x39711d8, 0xc00208a000})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x3972cc0, 0xc0053c4680})
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.131.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.131.0/execute/executor.go:261 +0x228
"

But it was running with active state and hasn't fully crashed. The Data Explorer was fully functional, for example. Creating a new bucket, letting Telegraf write in the new one and switching back to the old bucket after some minutes allowed Telegraf to write in the first bucket again. The mentioned Execute source panic error did not appear.

Environment info:

• System info: Run uname -srm and copy the output here

• InfluxDB version: Run influxd version and copy the output here

• Other relevant environment details: Container runtime, disk info, etc

• Linux 4.18.0-301.1.el8.x86_64 x86_64

• InfluxDB 2.0.9 (git: d1233b7951) build_date: 2021-10-01T21:09:53Z

• running on bare metal without any Docker or such, enough disk space, using an SSD here, server with a Intel(R) Xeon(R) E-2136 CPU @ 3.30GHz

Config: All default, nothing changed.

Logs: See Actual behavior section.

Performance: Going to execute the following lines next time the bug happens.

# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.

Is there any other way to get a little bit more into the details and trace the bug down? Where could I start?

[nicolasj92]

We have the same problem, also with the OPC-UA plugin. The whole database seems corrupted now after it ran fine for ~1 month

[Thedude7054]

Bump, also experiencing this issue. Database has been running fine for about 2 days, using the vSphere exporter in Telegraf as the only InfluxDB datasource. Telegraf is also failing to write to InfluxDB. Influx UI responds but fails reading from the bucket.

Running InfluxDB v2.1.1

ts=2022-04-01T08:56:47.212618Z lvl=info msg="Welcome to InfluxDB" log_id=0_a5J9R0000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z
ts=2022-04-01T08:56:47.845270Z lvl=info msg="Resources opened" log_id=0_a5J9R0000 service=bolt path=/var/lib/influxdb2/influxd.bolt
ts=2022-04-01T08:56:47.845413Z lvl=info msg="Resources opened" log_id=0_a5J9R0000 service=sqlite path=/var/lib/influxdb2/influxd.sqlite
ts=2022-04-01T08:56:49.061701Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0_a5J9R0000 bolt_path=/var/lib/influxdb2/influxd.bolt
ts=2022-04-01T08:56:49.061790Z lvl=info msg="Using data dir" log_id=0_a5J9R0000 service=storage-engine service=store path=/var/lib/influxdb2/engine/data
ts=2022-04-01T08:56:49.061811Z lvl=info msg="Compaction settings" log_id=0_a5J9R0000 service=storage-engine service=store max_concurrent_compactions=3 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2022-04-01T08:56:49.061817Z lvl=info msg="Open store (start)" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open op_event=start
ts=2022-04-01T08:56:49.116873Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.133172Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/3/000000029-000000002.tsm id=0 duration=15.866ms
ts=2022-04-01T08:56:49.133249Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/3 duration=54.576ms
ts=2022-04-01T08:56:49.157232Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.158427Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.159158Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.160155Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.160658Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a63ddb895df27578/autogen/2/000000002-000000002.tsm id=0 duration=0.126ms
ts=2022-04-01T08:56:49.160769Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a63ddb895df27578/autogen/2 duration=96.456ms
ts=2022-04-01T08:56:49.219364Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/4/000000026-000000002.tsm id=0 duration=60.842ms
ts=2022-04-01T08:56:49.219474Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/4 duration=131.524ms
ts=2022-04-01T08:56:49.248221Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/1/000000090-000000002.tsm id=0 duration=58.682ms
ts=2022-04-01T08:56:49.248387Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/1 duration=183.013ms
ts=2022-04-01T08:56:49.263480Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/5/000000007-000000002.tsm id=0 duration=103.963ms
ts=2022-04-01T08:56:49.263607Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/5 duration=198.334ms
ts=2022-04-01T08:56:49.264076Z lvl=info msg="Open store (end)" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=202.259ms
ts=2022-04-01T08:56:49.264107Z lvl=info msg="Starting retention policy enforcement service" log_id=0_a5J9R0000 service=retention check_interval=30m
ts=2022-04-01T08:56:49.264116Z lvl=info msg="Starting precreation service" log_id=0_a5J9R0000 service=shard-precreation check_interval=10m advance_period=30m
ts=2022-04-01T08:56:49.264152Z lvl=info msg="Starting query controller" log_id=0_a5J9R0000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
ts=2022-04-01T08:56:49.265277Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0_a5J9R0000 max_select_point=0 max_select_series=0 max_select_buckets=0
ts=2022-04-01T08:56:49.869667Z lvl=info msg=Listening log_id=0_a5J9R0000 service=tcp-listener transport=http addr=:8086 port=8086
ts=2022-04-01T08:56:49.869744Z lvl=info msg=Starting log_id=0_a5J9R0000 service=telemetry interval=8h


-04-01T08:57:15.913970Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2264 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc002dcca50})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc002dc6640}, 0xc000b10bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc002dc6640}, 0xc002dcc900)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc002db6900, 0x6246be6b)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00178c270, {0x40a6b18, 0xc002dc6640}, {0x406f510, 0xc002db6900}, 0xc002cbb2f8)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00178c270, {0x40a6b18, 0xc002dc6640})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00178c270, {0x40a6b18, 0xc002dc6640})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00178c270})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"


-04-01T08:57:22.502975Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2307 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc002dcdb00})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc00198c680}, 0xc0022b0bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc00198c680}, 0xc002dcdad0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc002e5b680, 0x6246be72)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc001b6a8f0, {0x40a6b18, 0xc00198c680}, {0x406f510, 0xc002e5b680}, 0xc001dcb618)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc001b6a8f0, {0x40a6b18, 0xc00198c680})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc001b6a8f0, {0x40a6b18, 0xc00198c680})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc001b6a8f0})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"


-04-01T08:57:24.371660Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2296 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc0020219e0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc00129a100}, 0xc002015bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc00129a100}, 0xc002021890)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc0011b8b00, 0x6246be74)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00178cd00, {0x40a6b18, 0xc00129a100}, {0x406f510, 0xc0011b8b00}, 0xc001cf30b8)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00178cd00, {0x40a6b18, 0xc00129a100})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00178cd00, {0x40a6b18, 0xc00129a100})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00178cd00})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"


-04-01T08:57:32.163376Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2355 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc002e1fc20})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc0019cda00}, 0xc0022b0bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0019cda00}, 0xc002e1fb30)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc0019ab680, 0x6246be7c)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0025d2a90, {0x40a6b18, 0xc0019cda00}, {0x406f510, 0xc0019ab680}, 0xc0025125b8)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0025d2a90, {0x40a6b18, 0xc0019cda00})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0025d2a90, {0x40a6b18, 0xc0019cda00})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0025d2a90})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"

[jvinolas]

We are having the same error

[AlaNDGT]

getting the same issue, it can't load tag values when trying to build a query on the UI Version: v2.2.0

[mashirali]

Bump, also experiencing this issue. Database has been running fine for about 2 days, using the vSphere exporter in Telegraf as the only InfluxDB datasource. Telegraf is also failing to write to InfluxDB. Influx UI responds but fails reading from the bucket.

Running InfluxDB v2.1.1

ts=2022-04-01T08:56:47.212618Z lvl=info msg="Welcome to InfluxDB" log_id=0_a5J9R0000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z
ts=2022-04-01T08:56:47.845270Z lvl=info msg="Resources opened" log_id=0_a5J9R0000 service=bolt path=/var/lib/influxdb2/influxd.bolt
ts=2022-04-01T08:56:47.845413Z lvl=info msg="Resources opened" log_id=0_a5J9R0000 service=sqlite path=/var/lib/influxdb2/influxd.sqlite
ts=2022-04-01T08:56:49.061701Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0_a5J9R0000 bolt_path=/var/lib/influxdb2/influxd.bolt
ts=2022-04-01T08:56:49.061790Z lvl=info msg="Using data dir" log_id=0_a5J9R0000 service=storage-engine service=store path=/var/lib/influxdb2/engine/data
ts=2022-04-01T08:56:49.061811Z lvl=info msg="Compaction settings" log_id=0_a5J9R0000 service=storage-engine service=store max_concurrent_compactions=3 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2022-04-01T08:56:49.061817Z lvl=info msg="Open store (start)" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open op_event=start
ts=2022-04-01T08:56:49.116873Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.133172Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/3/000000029-000000002.tsm id=0 duration=15.866ms
ts=2022-04-01T08:56:49.133249Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/3 duration=54.576ms
ts=2022-04-01T08:56:49.157232Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.158427Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.159158Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.160155Z lvl=info msg="index opened with 8 partitions" log_id=0_a5J9R0000 service=storage-engine index=tsi
ts=2022-04-01T08:56:49.160658Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/a63ddb895df27578/autogen/2/000000002-000000002.tsm id=0 duration=0.126ms
ts=2022-04-01T08:56:49.160769Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/a63ddb895df27578/autogen/2 duration=96.456ms
ts=2022-04-01T08:56:49.219364Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/4/000000026-000000002.tsm id=0 duration=60.842ms
ts=2022-04-01T08:56:49.219474Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/4 duration=131.524ms
ts=2022-04-01T08:56:49.248221Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/1/000000090-000000002.tsm id=0 duration=58.682ms
ts=2022-04-01T08:56:49.248387Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/1 duration=183.013ms
ts=2022-04-01T08:56:49.263480Z lvl=info msg="Opened file" log_id=0_a5J9R0000 service=storage-engine engine=tsm1 service=filestore path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/5/000000007-000000002.tsm id=0 duration=103.963ms
ts=2022-04-01T08:56:49.263607Z lvl=info msg="Opened shard" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/engine/data/d21c8d0f1a721bd9/autogen/5 duration=198.334ms
ts=2022-04-01T08:56:49.264076Z lvl=info msg="Open store (end)" log_id=0_a5J9R0000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=202.259ms
ts=2022-04-01T08:56:49.264107Z lvl=info msg="Starting retention policy enforcement service" log_id=0_a5J9R0000 service=retention check_interval=30m
ts=2022-04-01T08:56:49.264116Z lvl=info msg="Starting precreation service" log_id=0_a5J9R0000 service=shard-precreation check_interval=10m advance_period=30m
ts=2022-04-01T08:56:49.264152Z lvl=info msg="Starting query controller" log_id=0_a5J9R0000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
ts=2022-04-01T08:56:49.265277Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0_a5J9R0000 max_select_point=0 max_select_series=0 max_select_buckets=0
ts=2022-04-01T08:56:49.869667Z lvl=info msg=Listening log_id=0_a5J9R0000 service=tcp-listener transport=http addr=:8086 port=8086
ts=2022-04-01T08:56:49.869744Z lvl=info msg=Starting log_id=0_a5J9R0000 service=telemetry interval=8h
������
-04-01T08:57:15.913970Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2264 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc002dcca50})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc002dc6640}, 0xc000b10bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc002dc6640}, 0xc002dcc900)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc002db6900, 0x6246be6b)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00178c270, {0x40a6b18, 0xc002dc6640}, {0x406f510, 0xc002db6900}, 0xc002cbb2f8)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00178c270, {0x40a6b18, 0xc002dc6640})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00178c270, {0x40a6b18, 0xc002dc6640})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00178c270})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"
������
-04-01T08:57:22.502975Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2307 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc002dcdb00})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc00198c680}, 0xc0022b0bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc00198c680}, 0xc002dcdad0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc002e5b680, 0x6246be72)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc001b6a8f0, {0x40a6b18, 0xc00198c680}, {0x406f510, 0xc002e5b680}, 0xc001dcb618)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc001b6a8f0, {0x40a6b18, 0xc00198c680})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc001b6a8f0, {0x40a6b18, 0xc00198c680})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc001b6a8f0})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"
������
-04-01T08:57:24.371660Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2296 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc0020219e0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc00129a100}, 0xc002015bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc00129a100}, 0xc002021890)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc0011b8b00, 0x6246be74)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00178cd00, {0x40a6b18, 0xc00129a100}, {0x406f510, 0xc0011b8b00}, 0xc001cf30b8)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00178cd00, {0x40a6b18, 0xc00129a100})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00178cd00, {0x40a6b18, 0xc00129a100})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00178cd00})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"
������
-04-01T08:57:32.163376Z lvl=info msg="Execute source panic" log_id=0_a5J9R0000 service=storage-reads error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 2355 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).do.func1.1()\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205\npanic({0x3c52600, 0x53e34d0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c\npanic({0x3c7bc60, 0xc002e1fc20})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc001a93590, {0x40a6b18, 0xc0019cda00}, 0xc0022b0bd0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f\ngithub.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0019cda00}, 0xc002e1fb30)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390\ngithub.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc0019ab680, 0x6246be7c)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0025d2a90, {0x40a6b18, 0xc0019cda00}, {0x406f510, 0xc0019ab680}, 0xc0025125b8)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0025d2a90, {0x40a6b18, 0xc0019cda00})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0025d2a90, {0x40a6b18, 0xc0019cda00})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168\ngithub.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0025d2a90})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e\ncreated by github.com/influxdata/flux/execute.(*executionState).do\n\t/home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228\n"

I am also getting the same error... sometimes data is shown and sometimes its not showing..