search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.97k stars 3.55k forks source link

InfluxDB 2.4.0 runtime error at startup #23700

Open hkml2000 opened 2 years ago

hkml2000 commented 2 years ago

Actual behavior: running for a couple of weeks, InfluxDB suddenly stops and fails to start with "panic: runtime error: slice bounds out of range [:1073741893] with length 1073741824"

Environment info:

Config: { "assets-path": "", "bolt-path": "C:\influxdb\data\influxd.bolt", "e2e-testing": false, "engine-path": "C:\influxdb\data\engine", "feature-flags": null, "flux-log-enabled": false, "hardening-enabled": false, "http-bind-address": ":8086", "http-idle-timeout": 180000000000, "http-read-header-timeout": 10000000000, "http-read-timeout": 0, "http-write-timeout": 0, "influxql-max-select-buckets": 0, "influxql-max-select-point": 0, "influxql-max-select-series": 0, "instance-id": "", "log-level": "info", "metrics-disabled": false, "nats-max-payload-bytes": 0, "nats-port": 0, "no-tasks": false, "pprof-disabled": false, "query-concurrency": 1024, "query-initial-memory-bytes": 0, "query-max-memory-bytes": 0, "query-memory-bytes": 0, "query-queue-size": 1024, "reporting-disabled": false, "secret-store": "bolt", "session-length": 60, "session-renew-disabled": false, "sqlite-path": "C:\influxdb\data\influxd.sqlite", "storage-cache-max-memory-size": 1073741824, "storage-cache-snapshot-memory-size": 26214400, "storage-cache-snapshot-write-cold-duration": "10m0s", "storage-compact-full-write-cold-duration": "4h0m0s", "storage-compact-throughput-burst": 50331648, "storage-max-concurrent-compactions": 0, "storage-max-index-log-file-size": 1048576, "storage-no-validate-field-size": false, "storage-retention-check-interval": "30m0s", "storage-series-file-max-concurrent-snapshot-compactions": 0, "storage-series-id-set-cache-size": 0, "storage-shard-precreator-advance-period": "30m0s", "storage-shard-precreator-check-interval": "10m0s", "storage-tsm-use-madv-willneed": false, "storage-validate-keys": false, "storage-wal-fsync-delay": "0s", "storage-wal-max-concurrent-writes": 0, "storage-wal-max-write-delay": 600000000000, "storage-write-timeout": 10000000000, "store": "disk", "testing-always-allow-setup": false, "tls-cert": "", "tls-key": "", "tls-min-version": "1.2", "tls-strict-ciphers": false, "tracing-type": "", "ui-disabled": false, "vault-addr": "", "vault-cacert": "", "vault-capath": "", "vault-client-cert": "", "vault-client-key": "", "vault-client-timeout": 0, "vault-max-retries": 0, "vault-skip-verify": false, "vault-tls-server-name": "", "vault-token": "" }

Logs: ts=2022-09-01T10:46:39.710647Z lvl=info msg="Welcome to InfluxDB" log_id=0cfAPTNW000 version=v2.4.0 commit=de247bab08 build_date=2022-08-18T19:41:41Z log_level=info ts=2022-09-01T10:46:39.721651Z lvl=info msg="Resources opened" log_id=0cfAPTNW000 service=bolt path=C:\influxdb\data\influxd.bolt ts=2022-09-01T10:46:39.721651Z lvl=info msg="Resources opened" log_id=0cfAPTNW000 service=sqlite path=C:\influxdb\data\influxd.sqlite ts=2022-09-01T10:46:39.738648Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0cfAPTNW000 bolt_path=C:\influxdb\data\influxd.bolt ts=2022-09-01T10:46:39.738648Z lvl=info msg="Using data dir" log_id=0cfAPTNW000 service=storage-engine service=store path=C:\influxdb\data\engine\data ts=2022-09-01T10:46:39.739651Z lvl=info msg="Compaction settings" log_id=0cfAPTNW000 service=storage-engine service=store max_concurrent_compactions=4 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648 ts=2022-09-01T10:46:39.739651Z lvl=info msg="Open store (start)" log_id=0cfAPTNW000 service=storage-engine service=store op_name=tsdb_open op_event=start ts=2022-09-01T10:46:41.112747Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.113746Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.114747Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.114747Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.115748Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\708\000000024-000000003.tsm id=0 duration=1.001ms ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\869\000000008-000000002.tsm id=0 duration=0.999ms ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\869\000000017-000000001.tsm id=2 duration=0.000ms ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\869\000000016-000000002.tsm id=1 duration=0.000ms ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\761\000000024-000000003.tsm id=0 duration=0.000ms ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\869\000000019-000000001.tsm id=4 duration=0.000ms ts=2022-09-01T10:46:41.116747Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\869\000000018-000000001.tsm id=3 duration=0.000ms ts=2022-09-01T10:46:41.117746Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\869\000000020-000000001.tsm id=5 duration=0.999ms ts=2022-09-01T10:46:41.117746Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\3e1f65ab0ae9abf9\autogen\816\000000024-000000003.tsm id=0 duration=0.000ms ts=2022-09-01T10:46:41.120755Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.122749Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.213752Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\394\000012517-000000007.tsm id=3 duration=91.002ms ts=2022-09-01T10:46:41.272757Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004987-000000001.tsm id=11 duration=144.003ms ts=2022-09-01T10:46:41.275762Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004736-000000006.tsm id=1 duration=150.008ms ts=2022-09-01T10:46:41.491776Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\394\000012517-000000004.tsm id=0 duration=370.023ms ts=2022-09-01T10:46:41.734035Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.737030Z lvl=info msg="Reading file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=cacheloader path=C:\influxdb\data\engine\wal\5ae3c808ebcb8ea7\autogen\871\_00001.wal size=4702848 ts=2022-09-01T10:46:41.775452Z lvl=info msg="index opened with 8 partitions" log_id=0cfAPTNW000 service=storage-engine index=tsi ts=2022-09-01T10:46:41.844666Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\394\000012517-000000006.tsm id=2 duration=721.908ms ts=2022-09-01T10:46:41.844666Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004736-000000005.tsm id=0 duration=718.904ms ts=2022-09-01T10:46:41.924666Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\394\000012517-000000005.tsm id=1 duration=801.907ms ts=2022-09-01T10:46:41.960669Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004928-000000003.tsm id=4 duration=687.903ms ts=2022-09-01T10:46:42.021676Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004985-000000001.tsm id=9 duration=97.009ms ts=2022-09-01T10:46:42.044678Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\58ca5e890d8e8fc9\autogen\858\000000012-000000002.tsm id=0 duration=23.002ms ts=2022-09-01T10:46:42.061676Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004896-000000003.tsm id=3 duration=847.913ms ts=2022-09-01T10:46:42.070675Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004986-000000001.tsm id=10 duration=109.001ms ts=2022-09-01T10:46:42.122685Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004960-000000003.tsm id=5 duration=846.912ms ts=2022-09-01T10:46:42.144685Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004864-000000004.tsm id=2 duration=1008.924ms ts=2022-09-01T10:46:42.144685Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004968-000000002.tsm id=6 duration=651.906ms ts=2022-09-01T10:46:42.192693Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004976-000000002.tsm id=7 duration=348.023ms ts=2022-09-01T10:46:42.200689Z lvl=info msg="Opened file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=filestore path=C:\influxdb\data\engine\data\14aba2a311644add\autogen\763\000004984-000000002.tsm id=8 duration=356.018ms ts=2022-09-01T10:46:42.200689Z lvl=info msg="Reading file" log_id=0cfAPTNW000 service=storage-engine engine=tsm1 service=cacheloader path=C:\influxdb\data\engine\wal\14aba2a311644add\autogen\763\_04988.wal size=3187524 ts=2022-09-01T10:46:45.694932Z lvl=info msg="Open store (end)" log_id=0cfAPTNW000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=5955.205ms

panic: runtime error: slice bounds out of range [:1073741893] with length 1073741824

goroutine 1 [running]: github.com/influxdata/influxdb/v2/pkg/mmap.Map({0xc0015ca200, 0x3e}, 0x0) /root/project/pkg/mmap/mmap_windows.go:45 +0x42e github.com/influxdata/influxdb/v2/tsdb.(SeriesIndex).Open.func1(0xc000ad95e0) /root/project/tsdb/series_index.go:69 +0x9f github.com/influxdata/influxdb/v2/tsdb.(SeriesIndex).Open(0xc000ad95e0) /root/project/tsdb/series_index.go:84 +0x25 github.com/influxdata/influxdb/v2/tsdb.(SeriesPartition).Open.func1(0xc0055f84d0) /root/project/tsdb/series_partition.go:88 +0x10a github.com/influxdata/influxdb/v2/tsdb.(SeriesPartition).Open(0xc0055f84d0) /root/project/tsdb/series_partition.go:95 +0x6f github.com/influxdata/influxdb/v2/tsdb.(SeriesFile).Open(0xc004b7ca00) /root/project/tsdb/series_file.go:91 +0x4d5 github.com/influxdata/influxdb/v2/tsdb.(Store).openSeriesFile(0xc000a3b800, {0xc001b79230, 0x10}) /root/project/tsdb/store.go:556 +0x11a github.com/influxdata/influxdb/v2/tsdb.(Store).loadShards(0xc000a3b800, {0x72ca1a8?, 0xc001a5d900}) /root/project/tsdb/store.go:363 +0x12c7 github.com/influxdata/influxdb/v2/tsdb.(Store).Open(0xc000a3b800, {0x72ca1a8, 0xc001a5d900}) /root/project/tsdb/store.go:261 +0x226 github.com/influxdata/influxdb/v2/storage.(Engine).Open(0xc001bc2180, {0x72ca1a8, 0xc001a5d900}) /root/project/storage/engine.go:195 +0x11f github.com/influxdata/influxdb/v2/cmd/influxd/launcher.(Launcher).run(0xc000e38380, {0x72ca1a8?, 0xc001a5d8c0?}, 0xc000868000) /root/project/cmd/influxd/launcher/launcher.go:345 +0x2335 github.com/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1() /root/project/cmd/influxd/launcher/cmd.go:124 +0x2da github.com/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc000169b80?, {0x8bbc008?, 0x0?, 0x0?}) /root/project/kit/cli/viper.go:54 +0x1e github.com/spf13/cobra.(Command).execute(0xc000169b80, {0xc000108200, 0x0, 0x0}) /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x67c github.com/spf13/cobra.(Command).ExecuteC(0xc000169b80) /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x39c github.com/spf13/cobra.(*Command).Execute(...) /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887 main.main() /root/project/cmd/influxd/main.go:61 +0x4b8

hkml2000 commented 2 years ago

Hi, the problem appears after adding downsampling tasks influx runs well during 2 weeks and suddenly crashes, after disabling downsampling task no more crashes

jeffreyssmith2nd commented 2 years ago

The logs you provided above look like they are at startup when it fails, right? Do you happen to have any logs from right before/when influx crashes while the task is enabled?