search

dgraph-io / badger

Fast key-value DB in Go.
https://dgraph.io/badger
Apache License 2.0
13.88k stars 1.18k forks source link

Badger shouldn't just panic when open a bad database #1797

Closed HaoboGu closed 2 months ago

HaoboGu commented 2 years ago

When BadgerDB tries to open a corrupted dababase, sometime it would just panic(seems that this panic only occurs on Windows):

panic: runtime error: index out of range [3] with length 0 [recovered]
    panic: 
== Recovering from initIndex crash ==
File Info: [ID: 38, Size: 408, Zeros: 408]
isEnrypted: false checksumLen: 0 checksum:  indexLen: 0 index: [] 
== Recovered ==

goroutine 80 [running]:
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1.1(0xc0001c1aa0)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:351 +0x165
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1(0xc0001d6180)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:397 +0x6e
panic(0x1d77ca0, 0xc0003bcd68)
    C:/go/src/runtime/panic.go:971 +0x49a
github.com/google/flatbuffers/go.GetInt32(...)
    C:/gocode/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:85
github.com/google/flatbuffers/go.GetUOffsetT(...)
    C:/gocode/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:121
github.com/dgraph-io/badger/v3/fb.GetRootAsTableIndex(...)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/fb/TableIndex.go:14
github.com/dgraph-io/badger/v3/table.(*Table).readTableIndex(0xc0001d6180, 0x0, 0x8, 0xc0001c1a40)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:704 +0x491
github.com/dgraph-io/badger/v3/table.(*Table).initIndex(0xc0001d6180, 0x0, 0x0, 0xc0002a3260)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:462 +0x545
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest(0xc0001d6180, 0x0, 0x0)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:401 +0xab
github.com/dgraph-io/badger/v3/table.OpenTable(0xc000434280, 0x0, 0x200000, 0x0, 0x0, 0x3f847ae147ae147b, 0x1000, 0x0, 0x1, 0xc00053a080, ...)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:308 +0x36e
github.com/dgraph-io/badger/v3.newLevelsController.func1(0xc00043e180, 0xc0002b57b0, 0xc000589000, 0xc0002b5798, 0xc000178d10, 0x7, 0x7, 0xc00050a780, 0x21, 0x6, ...)
    C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/levels.go:150 +0x2d8
created by github.com/dgraph-io/badger/v3.newLevelsController

As a widely used dependency, BadgerDB should not just panic. Those panics cannot be captured and then breaks the whole application.

github-actions[bot] commented 2 years ago

Use Discuss Issues for reporting issues about this repository.

HaoboGu commented 2 years ago

@skrdgraph it's the issue, much appreciated if you can help

skrdgraph commented 2 years ago

@HaoboGu will do our best. For the time being, I'm tagging @pandeyshubham25 (he may have come across this) ...

HaoboGu commented 2 years ago

thanks very much!

skrdgraph commented 2 years ago

@HaoboGu would you be able to provide us with sample steps to reproduce this issue?

some other qs:

HaoboGu commented 2 years ago

  1. badger version: github.com/dgraph-io/badger/v3 v3.2103.2

  2. Yes, I'm using latest release version of badger

  3. Windows 10 1903 (OS build 18362.1082) / go1.16.2 windows/amd64

  4. reproduce:

    • download the corrupted db db.zip
    • open it on Windows using badger.Open(badger.DefaultOptions(dbPath).WithVerifyValueChecksum(true))
    • badger panics then

  5. badger should return an error instead of just panic

skrdgraph commented 2 years ago

i'll try this later in the day, and circle back. pls bear with me as I figure this out (will be discussing with the internal team on this ...)

HaoboGu commented 2 years ago

no problem :D

skrdgraph commented 2 years ago

ok this may not be very helpful, tested this on Mac-OS 12.4 (won't have access to windows machine till tomorrow)

Looks like I get a different error while trying to do this: 

$ badger  --dir ~/Downloads/db/ info show-keys
Listening for /debug HTTP requests at port: 8080
jemalloc enabled: false
Using Go memory

[     2021-05-05T15:08:08Z] MANIFEST      716 B MA
[          1 month earlier] 000002.sst    247 B L6 
[          1 month earlier] 000004.sst    246 B L6 
[          1 month earlier] 000006.sst    247 B L6 
[          1 month earlier] 000008.sst    247 B L6 
[          1 month earlier] 000010.sst    247 B L6 
[          1 month earlier] 000012.sst    247 B L6 
[          1 month earlier] 000014.sst    247 B L6 
[          1 month earlier] 000016.sst    247 B L6 
[          1 month earlier] 000018.sst    247 B L6 
[          1 month earlier] 000023.sst    337 B L6 
[          3 weeks earlier] 000025.sst    247 B L6 
[         21 hours earlier] 000028.sst    408 B L6 
[         21 hours earlier] 000030.sst    247 B L6 
[         21 hours earlier] 000032.sst    247 B L6 
[         19 hours earlier] 000035.sst    410 B L6 
[       30 minutes earlier] 000038.sst    408 B L6 
[       29 minutes earlier] 000041.vlog    20 B VL
[       29 minutes earlier] 000042.vlog  2.0 GiB VL

[EXTRA]
[2021-05-05T15:08:08Z] 00086.mem    128 MiB
[2021-03-08T17:42:34Z] DISCARD      1.0 MiB
[2021-03-08T17:42:34Z] KEYREGISTRY    28 B

[Summary]
Level 0 size:          0 B
Level 1 size:          0 B
Level 2 size:          0 B
Level 3 size:          0 B
Level 4 size:          0 B
Level 5 size:          0 B
Level 6 size:      4.4 KiB
Total SST size:    4.4 KiB
Value log size:    2.0 GiB

Abnormalities:
3 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
Error: failed to open database err: while opening memtables error: while opening fid: 86 error: while updating skiplist error: end offset: 20 < size: 134217728 error: Log truncate required to run DB. This might result in data loss
github.com/dgraph-io/badger/v3.init
    /Users/sudhishkr/scratch/github/dgraph-io/badger/errors.go:101
runtime.doInit
    /usr/local/opt/go/libexec/src/runtime/proc.go:6222
runtime.doInit
    /usr/local/opt/go/libexec/src/runtime/proc.go:6199
runtime.doInit
    /usr/local/opt/go/libexec/src/runtime/proc.go:6199
runtime.main
    /usr/local/opt/go/libexec/src/runtime/proc.go:233
runtime.goexit
    /usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1571

Are you trying to open the DB like this? badger --dir ~/Downloads/db/ info show-keys <<

HaoboGu commented 2 years ago

I got this:

PS C:\Users\ADMIN\Downloads> badger --dir db info show-keys
Listening for /debug HTTP requests at port: 8080
jemalloc enabled: false
Using Go memory

[2021-05-05T15:08:07+08:00] MANIFEST      716 B MA
[          1 month earlier] 000002.sst    247 B L6 
[          1 month earlier] 000004.sst    246 B L6 
[          1 month earlier] 000006.sst    247 B L6 
[          1 month earlier] 000008.sst    247 B L6 
[          1 month earlier] 000010.sst    247 B L6 
[          1 month earlier] 000012.sst    247 B L6 
[          1 month earlier] 000014.sst    247 B L6 
[          1 month earlier] 000016.sst    247 B L6 
[          1 month earlier] 000018.sst    247 B L6 
[          1 month earlier] 000023.sst    337 B L6 
[          3 weeks earlier] 000025.sst    247 B L6 
[         21 hours earlier] 000028.sst    408 B L6 
[         21 hours earlier] 000030.sst    247 B L6 
[         21 hours earlier] 000032.sst    247 B L6 
[         19 hours earlier] 000035.sst    410 B L6 
[       30 minutes earlier] 000038.sst    408 B L6 
[       29 minutes earlier] 000041.vlog    20 B VL
[       29 minutes earlier] 000042.vlog  2.0 GiB VL

[EXTRA]
[2021-05-05T15:08:07+08:00] 00086.mem    128 MiB
[2021-03-08T17:42:33+08:00] DISCARD      1.0 MiB
[2021-03-08T17:42:32+08:00] KEYREGISTRY    28 B

[Summary]
Level 0 size:          0 B
Level 1 size:          0 B
Level 2 size:          0 B
Level 3 size:          0 B
Level 4 size:          0 B
Level 5 size:          0 B
Level 6 size:      4.4 KiB
Total SST size:    4.4 KiB
Value log size:    2.0 GiB

Abnormalities:
3 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
Error: failed to open database err: Read-only mode is not supported on Windows
github.com/dgraph-io/badger/v3.init
        C:/Users/ADMIN/Downloads/badger-3.2103.2/errors.go:94
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6222
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.main
        C:/Program Files/Go/src/runtime/proc.go:233
runtime.goexit
        C:/Program Files/Go/src/runtime/asm_amd64.s:1571
Usage:
  badger info [flags]

Flags:
      --cv-mode string       [none, table, block, tableAndBlock] Specifies when the db should verify checksum for SST. (default "none")
      --discard              Parse and print DISCARD file from value logs.
      --enc-key string       Use the provided encryption key
  -h, --help                 help for info
      --histogram            Show a histogram of the key and value sizes.
      --history              Show all versions of a key
  -l, --lookup string        Hex of the key to lookup
      --read-only            If set to true, DB will be opened in read only mode. If DB has not been closed properly, this option can be set to false to open DB. (default true)
      --show-internal        Show internal keys along with other keys. This option should be used along with --show-key option
      --show-keys            Show keys stored in Badger
      --show-meta            Output item meta data as well (default true)
  -s, --show-tables          If set to true, show tables as well.       
      --truncate             If set to true, it allows truncation of value log files if they have corrupt data.
      --with-prefix string   Consider only the keys with specified prefix

Global Flags:
      --dir string        Directory where the LSM tree files are located. (required)
      --vlog-dir string   Directory where the value log files are located, if different from --dir

failed to open database err: Read-only mode is not supported on Windows
github.com/dgraph-io/badger/v3.init
        C:/Users/ADMIN/Downloads/badger-3.2103.2/errors.go:94
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6222
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.main
        C:/Program Files/Go/src/runtime/proc.go:233
runtime.goexit
        C:/Program Files/Go/src/runtime/asm_amd64.s:1571
skrdgraph commented 2 years ago

your error is probably related to folder permissions -> failed to open database err: Read-only mode is not supported on Windows.

my error is something I need to look further into. @MichelDiz have you see this before?

@HaoboGu - how did you exactly simulate the error you have indicated in your original post https://github.com/dgraph-io/badger/issues/1797#issue-1320470136? You did respond saying

open it on Windows using badger.Open(badger.DefaultOptions(dbPath).WithVerifyValueChecksum(true))

if you could elaborate on the steps it will be helpful.

HaoboGu commented 2 years ago

This is the minimal repro that I can do: image

Here is the main.go:

package main

import (
    "fmt"

    "github.com/dgraph-io/badger/v3"
)

func main() {
    opts := badger.DefaultOptions("C://Users/ADMIN/Downloads/db")
    _, err := badger.Open(opts)
    if err != nil {
        fmt.Print(err)
    }
}

Then go run ./main.go

BenChen-PKI commented 1 year ago

I would suggest in function newLevelsController of levels.go, add a defer function to recover and panic again in the go routine (line 129), so that the panic in this go routine can be caught by the client code. in this way, the application can handle the panic from badgerDB when opening a corrupted db, then delete the corrupted db and create a new db to serve.

github-actions[bot] commented 3 months ago

This issue has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open.