Index out of range in influxdb2 at startup

[amir-enote]

Hello, I'm hitting an interesting error when launching an existent influxdb2 pod in k8s. The stateful set was working great until a recent restart, since then I'm unable to startup the pod properly with the following error: ts=2022-04-08T10:09:57.720955Z lvl=info msg="Opened shard" log_id=0_jAFoh0000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/data/8ec72314c1dc0090/autogen/178 duration=9902.121ms ts=2022-04-08T10:09:57.806059Z lvl=info msg="index opened with 8 partitions" log_id=0_jAFoh0000 service=storage-engine index=tsi panic: runtime error: index out of range [1] with length 0 goroutine 619 [running]: encoding/binary.bigEndian.Uint16(...) /home/circleci/.tools/go/src/encoding/binary/binary.go:101 github.com/influxdata/influxdb/v2/tsdb.ReadSeriesKeyMeasurement(...) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/series_file.go:380 github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).execSeriesEntry(0xc00f9ff3b0, 0xc01e68bc68) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:698 +0xb6f github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).execEntry(0xc01e68bc68, 0x7f8c881831c6) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:638 +0x3e github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).open(0xc00f9ff3b0) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:166 +0x3cf github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).Open(0xc00f9ff3b0) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:115 +0x6b github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).openLogFile(0xc01e79aea0, {0xc0583ca870, 0x404dd20}) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:254 +0x4f github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).Open(0xc01e79aea0) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/partition.go:197 +0x96f github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Index).Open.func1(0xc01e4b8fc0) /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:296 +0x47 created by github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Index).Open /home/circleci/go/src/github.com/influxdata/influxdb/tsdb/index/tsi1/index.go:290 +0x4da stream closed

Not sure how to approach this issue, or what can be done to bypass is, so any suggestion would be grateful.

I'm deploying using the latest chart version, in k8s version 1.19.15 Custom values that are provided to the chart are resources(limit cpu to 2, memory to 4Gi), persistence is enabled, ingress in enabled with tls and an admin user is provided to the chart, the rest is default.

Thanks in advance!

[alespour]

Coul you please provide the full log? This does not seem related to the chart itself, so I would transfer the issue influxdb repository and it is important to know more details about the instance like version etc. Thank you.

[amir-enote]

Here is the beginning of the log, afterwards its alternates as in the previous log an the new one(open shard, reading file, index opened), and then results in the exception as showed previously.

2022-04-11T08:53:18. info found existing boltdb file, skipping setup wrapper {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt"} 2022-04-11T08:53:18. info found existing boltdb file, skipping setup wrapper {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt"} ts=2022-04-11T08:53:18.422315Z lvl=info msg="Welcome to InfluxDB" log_id=0_my4mpG000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z ts=2022-04-11T08:53:18.431151Z lvl=info msg="Resources opened" log_id=0_my4mpG000 service=bolt path=/var/lib/influxdb2/influxd.bolt ts=2022-04-11T08:53:18.431257Z lvl=info msg="Resources opened" log_id=0_my4mpG000 service=sqlite path=/var/lib/influxdb2/influxd.sqlite ts=2022-04-11T08:53:18.448332Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0_my4mpG000 bolt_path=/var/lib/influxdb2/influxd.bolt ts=2022-04-11T08:53:18.450285Z lvl=info msg="Using data dir" log_id=0_my4mpG000 service=storage-engine service=store path=/var/lib/influxdb2/data ts=2022-04-11T08:53:18.450339Z lvl=info msg="Compaction settings" log_id=0_my4mpG000 service=storage-engine service=store max_concurrent_compactions=1 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648 ts=2022-04-11T08:53:18.450356Z lvl=info msg="Open store (start)" log_id=0_my4mpG000 service=storage-engine service=store op_name=tsdb_open op_event=start ts=2022-04-11T08:53:18.483620Z lvl=info msg="index opened with 8 partitions" log_id=0_my4mpG000 service=storage-engine index=tsi ts=2022-04-11T08:53:18.486518Z lvl=info msg="Reading file" log_id=0_my4mpG000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb2/wal/82775262672a1bed/autogen/315/_00001.wal size=876509 ts=2022-04-11T08:53:18.487447Z lvl=info msg="index opened with 8 partitions" log_id=0_my4mpG000 service=storage-engine index=tsi ts=2022-04-11T08:53:18.488224Z lvl=info msg="Reading file" log_id=0_my4mpG000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb2/wal/82775262672a1bed/autogen/312/_00001.wal size=10215367 ts=2022-04-11T08:53:27.998974Z lvl=info msg="Opened shard" log_id=0_my4mpG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/data/82775262672a1bed/autogen/315 duration=9542.869ms ts=2022-04-11T08:53:28.003486Z lvl=info msg="Opened shard" log_id=0_my4mpG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/data/82775262672a1bed/autogen/312 duration=9542.106ms ts=2022-04-11T08:53:28.013328Z lvl=info msg="index opened with 8 partitions" log_id=0_my4mpG000 service=storage-engine index=tsi ts=2022-04-11T08:53:28.018374Z lvl=info msg="index opened with 8 partitions" log_id=0_my4mpG000 service=storage-engine index=tsi ts=2022-04-11T08:53:28.021767Z lvl=info msg="Reading file" log_id=0_my4mpG000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb2/wal/82775262672a1bed/autogen/313/_00001.wal size=831414 ts=2022-04-11T08:53:28.023656Z lvl=info msg="Reading file" log_id=0_my4mpG000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb2/wal/82775262672a1bed/autogen/314/_00001.wal size=1400750 ts=2022-04-11T08:53:28.096284Z lvl=info msg="Opened shard" log_id=0_my4mpG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/data/82775262672a1bed/autogen/313 duration=93.271ms ts=2022-04-11T08:53:28.131066Z lvl=info msg="index opened with 8 partitions" log_id=0_my4mpG000 service=storage-engine index=tsi ts=2022-04-11T08:53:28.136200Z lvl=info msg="Reading file" log_id=0_my4mpG000 service=storage-engine engine=tsm1 service=cacheloader path=/var/lib/influxdb2/wal/82775262672a1bed/autogen/320/_00001.wal size=1491484 ts=2022-04-11T08:53:28.132768Z lvl=info msg="Opened shard" log_id=0_my4mpG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/var/lib/influxdb2/data/82775262672a1bed/autogen/314 duration=129.230ms ts=2022-04-11T08:53:28.160129Z lvl=info msg="index opened with 8 partitions" log_id=0_my4mpG000 service=storage-engine index=tsi

[alespour]

Since you are using the last chart version I assume it's InfluxDB 2.1.1.

[amir-enote]

This is from the log I pasted: Welcome to InfluxDB" log_id=0_my4mpG000 version=2.1.1 commit=657e1839de build_date=2021-11-09T03:03:48Z