pebbledb: panic during upgrade w/ Cosmos SDK

melekes commented 9 months ago

Refs https://github.com/cometbft/cometbft-db/pull/112#discussion_r1464435355

At the upgrade block, the Cosmos SDK will panic without flushing data to disk or closing dbs properly.

Upgrade guide:

1. After seeing `UPGRADE "xxxx" NEED at height....`, restart the current version with `-X github.com/tendermint/tm-db.ForceSync=1`
2. Restart the new version as normal

Example: Upgrading sifchain from v0.14.0 to v0.15.0

# log:
panic: UPGRADE "0.15.0" NEEDED at height: 8170210: {"binaries":{"linux/amd64":"https://github.com/Sifchain/sifnode/releases/download/v0.15.0/sifnoded-v0.15.0-linux-amd64.zip?checksum=0c03b5846c5a13dcc0d9d3127e4f0cee0aeddcf2165177b2f2e0d60dbcf1a5ea"}}

# step1
git reset --hard
git checkout v0.14.0
go mod edit -replace github.com/tendermint/tm-db=github.com/baabeetaa/tm-db@pebble
go mod tidy
go install -tags pebbledb -ldflags "-w -s -X github.com/cosmos/cosmos-sdk/types.DBBackend=pebbledb -X github.com/tendermint/tm-db.ForceSync=1" ./cmd/sifnoded
$HOME/go/bin/sifnoded start --db_backend=pebbledb

# step 2
git reset --hard
git checkout v0.15.0
go mod edit -replace github.com/tendermint/tm-db=github.com/baabeetaa/tm-db@pebble
go mod tidy
go install -tags pebbledb -ldflags "-w -s -X github.com/cosmos/cosmos-sdk/types.DBBackend=pebbledb" ./cmd/sifnoded
$HOME/go/bin/sifnoded start --db_backend=pebbledb

melekes commented 1 month ago

Is anyone still experiencing this issue?

melekes commented 1 month ago

I will close this, but please comment if the issue persists.

cometbft / cometbft-db

pebbledb: panic during upgrade w/ Cosmos SDK #126