search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.59k stars 3.54k forks source link

write points fail and report [shard 90] short buffer? #22951

Open lonelyleaf opened 2 years ago

lonelyleaf commented 2 years ago

Steps to reproduce: List the minimal actions needed to reproduce the behavior.

  1. write point use java client

  2. client report 

    com.influxdb.exceptions.InternalServerErrorException: unexpected error writing points to database: [shard 90] short buffer
at com.influxdb.internal.AbstractRestClient.responseToError(AbstractRestClient.java:118) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.internal.AbstractRestClient.execute(AbstractRestClient.java:81) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.AbstractWriteBlockingClient.write(AbstractWriteBlockingClient.java:85) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.WriteApiBlockingImpl.lambda$writePoints$1(WriteApiBlockingImpl.java:137) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]

  3. influxdb logs error too ,and only when write some data trigger this error

Environment info:

docker influx 2.1.1

Logs: Include snippet of errors in log.

2021-12-01T06:25:36.585734Z info    Reading file    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "cacheloader", "path": "/var/lib/influxdb2/engine/wal/cb8da52d13f762fb/autogen/96/_00001.wal", "size": 5712}
2021-12-01T06:25:36.591886Z info    Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/94/000000001-000000001.tsm", "id": 0, "duration": "7.343ms"}
2021-12-01T06:25:36.611728Z info    Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/89/000000003-000000002.tsm", "id": 0, "duration": "0.596ms"}
2021-12-01T06:25:43.419126Z error   Failed to open shard    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "store", "op_name": "tsdb_open", "db_shard_id": 90, "error": "[shard 90] short buffer"}
2021-12-01T06:25:44.798705Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.808490Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.810225Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.851809Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.862204Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T07:00:56.567977Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:05:56.869878Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:03.029356Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.678191Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.694187Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.697167Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.706261Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.065021Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.074078Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.696444Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.705558Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.150872Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.159537Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:48.998667Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:49.007475Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.541653Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.550789Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.432700Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.441095Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.482655Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.492085Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.475071Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.483124Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:29.106772Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T02:55:50.189347Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T02:56:46.791163Z info    Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T02:56:46.791346Z info    Retention policy deletion check (end)   {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.206ms"}
2021-12-02T02:58:51.197395Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:00:50.281280Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:03:44.792549Z info    Cache snapshot (start)  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:44.824314Z info    Snapshot for path written   {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/d985752f6c791b5b/autogen/20", "duration": "31.791ms"}
2021-12-02T03:03:44.824357Z info    Cache snapshot (end)    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.833ms"}
2021-12-02T03:03:49.558056Z info    Cache snapshot (start)  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:49.588914Z info    Snapshot for path written   {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/592474454cfee085/autogen/99", "duration": "30.889ms"}
2021-12-02T03:03:49.588946Z info    Cache snapshot (end)    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "30.919ms"}
2021-12-02T03:03:53.616409Z info    Cache snapshot (start)  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:53.648065Z info    Snapshot for path written   {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/e801bb6a29638f3e/autogen/100", "duration": "31.674ms"}
2021-12-02T03:03:53.648100Z info    Cache snapshot (end)    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.709ms"}
2021-12-02T03:04:25.317616Z info    Unauthorized    {"log_id": "0Y9AZ_e0000", "error": "token required"}
2021-12-02T03:04:55.852016Z info    Execute source panic    {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776506 [running]:
runtime/debug.Stack()
    /home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00f43eed0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0587a6540}, 0xc0617efbd0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0587a6540}, 0xc00f43e7b0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01c044180, 0x61a837d7)
    /home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00fa53860, {0x40a6b18, 0xc0587a6540}, {0x406f510, 0xc01c044180}, 0xc016b2c0b0)
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00fa53860})
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:04:57.479246Z info    Execute source panic    {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776589 [running]:
runtime/debug.Stack()
    /home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d85f710})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc015cda100}, 0xc01758fbd0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc015cda100}, 0xc01d85f020)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d839380, 0x61a837d9)
    /home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00b639a00, {0x40a6b18, 0xc015cda100}, {0x406f510, 0xc01d839380}, 0xc03c8df0e0)
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00b639a00})
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:04:59.934067Z info    Execute source panic    {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776621 [running]:
runtime/debug.Stack()
    /home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01e71b800})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0138f7340}, 0xc03d883bd0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0138f7340}, 0xc01e71b110)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01e70d100, 0x61a837db)
    /home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0181f2d00, {0x40a6b18, 0xc0138f7340}, {0x406f510, 0xc01e70d100}, 0xc07acbf840)
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0181f2d00})
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:05:50.338055Z info    Write failed    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:08:09.183024Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:08:09.183373Z info    Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:08:09.183392Z info    Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:09:00.159441Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:00.159977Z info    Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:00.159995Z info    Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:54.759458Z info    index opened with 8 partitions  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:54.759771Z info    Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:09:54.759784Z info    Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:14:03.878068Z info    Execute source panic    {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903477 [running]:
runtime/debug.Stack()
    /home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00b13f740})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01c242cc0}, 0xc028b75bd0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01c242cc0}, 0xc00b13f5f0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc00257e380, 0x61a839fb)
    /home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8c410, {0x40a6b18, 0xc01c242cc0}, {0x406f510, 0xc00257e380}, 0xc0d402d520)
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8c410})
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:14:07.591332Z info    Execute source panic    {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903622 [running]:
runtime/debug.Stack()
    /home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01c3ea150})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01cf75240}, 0xc016bf8bd0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01cf75240}, 0xc01c3ea000)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01b72c400, 0x61a839ff)
    /home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8cc30, {0x40a6b18, 0xc01cf75240}, {0x406f510, 0xc01b72c400}, 0xc0116b95b0)
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8cc30})
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:14:14.450359Z info    Execute source panic    {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 904050 [running]:
runtime/debug.Stack()
    /home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d8e8db0})
    /home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01daf9340}, 0xc002a4abd0)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01daf9340}, 0xc01d8e8c60)
    /home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d496800, 0x61a83a06)
    /home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0173ff040, {0x40a6b18, 0xc01daf9340}, {0x406f510, 0xc01d496800}, 0xc02bd34880)
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
    /home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0173ff040})
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
    /home/circleci/go/pkg/mod/github.com/influxdata/flux@v0.139.0/execute/executor.go:261 +0x228
2021-12-02T03:15:51.054091Z info    Cache snapshot (start)  {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:15:51.057282Z info    Snapshot for path written   {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/98", "duration": "3.215ms"}
2021-12-02T03:15:51.057308Z info    Cache snapshot (end)    {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "3.238ms"}
2021-12-02T03:17:16.205704Z warn    internal error not returned to client   {"log_id": "0Y9AZ_e0000", "handler": "error_logger", "error": "context canceled"}
2021-12-02T03:26:46.791140Z info    Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T03:26:46.791292Z info    Retention policy deletion check (end)   {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.164ms"}
abelletti commented 2 years ago

FYI, we are seeing this as well. InfluxDB v2.1.1, Python client. Writing a large number of rows, we first see the issue on row 14995. The error definitely appears to be data-dependent. From skimming the InfluxDB source, this comes from extracting an array of strings from a compressed sequence of bytes. Apparently something becomes corrupt and the decoder expects to find more bytes of string than are actually present.

Going to see if I can downgrade to an earlier InfluxDB and check against that.

abelletti commented 2 years ago

In our case, this appears to be caused by corruption in the bucket we're writing to. I've confirmed that writing to a new bucket in a different org works, and even a new bucket in the same org. But writing to this particular bucket/org fails every time, regardless of whether that particular measurement has been purged or not.

lonelyleaf commented 2 years ago

@abelletti You're right, I can fix this issue by create a new bucket, thanks. Hope IInfluxdata can add some feature to fix corrupted data in bucket.

davidby-influx commented 2 years ago

We have seen this when a disk fills up. That can cause corrupt files which persist after a disk resize or move to a larger disk.