search

talariadb / talaria

TalariaDB is a distributed, highly available, and low latency time-series database for Presto
MIT License
215 stars 31 forks source link

[error] writeRequests: Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1074437932 got: 2147483646 #97

Closed atlas-comstock closed 2 years ago

atlas-comstock commented 2 years ago

When instance ingesting SQS msg and compact to ORC file, it crashed after 20mins.

2022/06/03 07:11:57 [error] writeRequests: Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1074437932 got: 2147483646
2022/06/03 07:11:57 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1074437932 got: 2147483646
2022/06/03 07:13:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:13:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:13:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:14:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:14:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:14:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:15:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:15:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:15:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:16:14 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:16:14 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:16:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:17:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:17:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:17:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:18:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:18:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:18:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:19:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:19:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:19:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:20:15 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:20:15 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:20:15 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:21:16 [error] Unable to read: Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:21:16 [error] Key: [6 163 5 141 0 0 0 0 0 0 0 0 0 0 9 170], Version : 3208, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1202413 Offset:1043162081}
2022/06/03 07:21:16 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:22:06 [error] writeRequests: Unable to truncate file: "/data/eventlog/000002.vlog" error: mremap size mismatch: requested: 1074096225 got: 2147483646
2022/06/03 07:22:06 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000002.vlog" error: mremap size mismatch: requested: 1074096225 got: 2147483646
panic: runtime error: slice bounds out of range [1074096225:0]

goroutine 218270 [running]:
github.com/dgraph-io/badger/v3.(*valueLog).write.func2(0x403137e330, 0x403137e330, 0x40107e2120)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/value.go:826 +0x1bc
github.com/dgraph-io/badger/v3.(*valueLog).write(0x40003b33f8, 0x4083065f40, 0x1, 0xa, 0x0, 0x0)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/value.go:884 +0x388
github.com/dgraph-io/badger/v3.(*DB).writeRequests(0x40003b3200, 0x4083065f40, 0x1, 0xa, 0x1d94600, 0x0)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:816 +0x74
github.com/dgraph-io/badger/v3.(*DB).doWrites.func1(0x4083065f40, 0x1, 0xa)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:887 +0x4c
created by github.com/dgraph-io/badger/v3.(*DB).doWrites
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:940 +0x350
atlas-comstock commented 2 years ago 
2022/06/03 07:11:51 [error] writeRequests: Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1073911873 got: 2147483646
2022/06/03 07:11:51 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1073911873 got: 2147483646
2022/06/03 07:13:10 [error] Unable to read: Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:13:10 [error] Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1592503 Offset:1061259076}
2022/06/03 07:13:10 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:14:14 [error] Unable to read: Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:14:14 [error] Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1592503 Offset:1061259076}
2022/06/03 07:14:14 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:15:17 [error] Unable to read: Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:15:17 [error] Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1592503 Offset:1061259076}
2022/06/03 07:15:17 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:16:21 [error] Unable to read: Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:16:21 [error] Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1592503 Offset:1061259076}
2022/06/03 07:16:21 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:17:34 [error] Unable to read: Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:17:34 [error] Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1592503 Offset:1061259076}
2022/06/03 07:17:34 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:18:46 [error] Unable to read: Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:18:46 [error] Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1592503 Offset:1061259076}
2022/06/03 07:18:46 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:19:50 [error] Unable to read: Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 Error: file with ID: 1 not found
2022/06/03 07:19:50 [error] Key: [53 48 0 120 0 0 0 0 0 0 0 0 0 0 8 138], Version : 2828, meta: 66, userMeta: 0 valuePointer: {Fid:1 Len:1592503 Offset:1061259076}
2022/06/03 07:19:50 [error] ServerError: target=storage/compact/compact.go.102, reason=Internal Server Error, msg=compact: unable to read a buffer due to EOF
2022/06/03 07:20:01 [error] writeRequests: Unable to truncate file: "/data/eventlog/000002.vlog" error: mremap size mismatch: requested: 1074595673 got: 2147483646
2022/06/03 07:20:01 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000002.vlog" error: mremap size mismatch: requested: 1074595673 got: 2147483646
panic: runtime error: slice bounds out of range [1074595673:0]

goroutine 180032 [running]:
github.com/dgraph-io/badger/v3.(*valueLog).write.func2(0x4058cf5b00, 0x4058cf5b00, 0x4000288240)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/value.go:826 +0x1bc
github.com/dgraph-io/badger/v3.(*valueLog).write(0x40005d93f8, 0x40160cf680, 0x1, 0xa, 0x0, 0x0)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/value.go:884 +0x388
github.com/dgraph-io/badger/v3.(*DB).writeRequests(0x40005d9200, 0x40160cf680, 0x1, 0xa, 0x1d94600, 0x0)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:816 +0x74
github.com/dgraph-io/badger/v3.(*DB).doWrites.func1(0x40160cf680, 0x1, 0xa)
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:887 +0x4c
created by github.com/dgraph-io/badger/v3.(*DB).doWrites
        /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:940 +0x350
atlas-comstock commented 2 years ago

Fix in https://github.com/talariadb/talaria/pull/98