search

cosmos / cosmos-sdk

:chains: A Framework for Building High Value Public Blockchains :sparkles:
https://cosmos.network/
Apache License 2.0
6.23k stars 3.59k forks source link

Can't import/work with genesis.json larger 4GB #16915

Open alpha-omega-labs opened 1 year ago

alpha-omega-labs commented 1 year ago

Hello, In GenesisL1 we have largest state, exported state is ~10GB. Snappy compression lib can't handle anything above ~4GB and throwing "panic: snappy: decoded block is too large" on importing genesis.json to new chain during upgrade. 4gb state might meet Cosmos and other chains, so maybe its a good idea to move to other compression lib. Thank you

chillyvee commented 1 year ago

Is the genesis.json entirely pre-compressed? Or only parts of the data inside genesis.json?

Maybe snappy is used during protobuf portion of JSON decode and not supporting > 4GB?

Is there a particular part of your genesis that is larger than others? Number of accounts? Contract state?

Can you post the backtrace dump of the panic?

alpha-omega-labs commented 1 year ago

Its a genesis.json obtained via "genesisd export" command which is a part of ethermint, so entire state. And this error comes when trying to add this genesis.json with cronos (and also kava) source codes. We have all parts of genesis.json large: more than 700k account (because of L1 coins airdrop/gift via solidity smart contract in our chain to Cosmos and Evmos delegators) and also very large contract state with about 1MLN NFT of molecular data (https://molnft.org) and and also few GB of molecular dataset written entirely onchain and indexed there with onchain search possibility: https://datasetnft.org/protcrud.

We expect also to increase that size to 20gb during recording of all obtained proteins (about 200k) entirely onchain (this will add about 10gb of data to store ~50 years of molecular biology experiments for public view/research).

Will post more info on panic in a few hours from that message, thank you!

alpha-omega-labs commented 1 year ago

This is with cronosd and imported genesis.json (same message on just exported ~10GB and with prettyfied ~14gb)

` cronosd start --log_level=trace 12:05PM INF Unlocking keyring 12:05PM INF starting ABCI with Tendermint 12:05PM INF starting node with ABCI Tendermint in-process panic: snappy: decoded block is too large

goroutine 1 [running]: github.com/golang/snappy.Encode({0xc001f96040?, 0x12?, 0x0?}, {0xca6da5c000?, 0x27661a0?, 0x43b8aa?}) github.com/golang/snappy@v0.0.4/encode.go:22 +0x2d1 github.com/syndtr/goleveldb/leveldb/table.(Writer).writeBlock(0xc001bf0000, 0xc001bf0060, 0xc001bf0060?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:172 +0x139 github.com/syndtr/goleveldb/leveldb/table.(Writer).finishBlock(0xc001bf0000) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:223 +0x46 github.com/syndtr/goleveldb/leveldb/table.(Writer).Append(0xc001bf0000, {0xc803120000, 0x12, 0x26a93c000}, {0xc803120012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:256 +0x253 github.com/syndtr/goleveldb/leveldb.(tWriter).append(0xc000da68a0, {0xc803120000, 0x12, 0x26a93c000}, {0xc803120012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table.go:558 +0x1e5 github.com/syndtr/goleveldb/leveldb.(tOps).createFrom(0x26a93ad15?, {0x4be81b0, 0xc001194000}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table.go:397 +0x16d github.com/syndtr/goleveldb/leveldb.(session).flushMemdb(0xc00160a000, 0xc000ddc280, 0xc0001b6a80, 0x0?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/sessioncompaction.go:35 +0x116 github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal(0xc001db4700) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:597 +0xcb6 github.com/syndtr/goleveldb/leveldb.openDB(0xc00160a000) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:131 +0x43c github.com/syndtr/goleveldb/leveldb.Open({0x4be63d0?, 0xc0001fb810?}, 0x5ba700?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:203 +0x188 github.com/syndtr/goleveldb/leveldb.OpenFile({0xc000500300?, 0x2c0b742?}, 0x0) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:225 +0x5d github.com/cometbft/cometbft-db.NewGoLevelDBWithOpts({0x2c0b742?, 0xc00062aee8?}, {0xc00203a030?, 0x0?}, 0xc000e12c30?) github.com/cometbft/cometbft-db@v0.8.0/goleveldb.go:32 +0x7c github.com/cometbft/cometbft-db.NewGoLevelDB(...) github.com/cometbft/cometbft-db@v0.8.0/goleveldb.go:27 github.com/cometbft/cometbft-db.init.0.func1({0x2c0b742?, 0xc000e3d5c0?}, {0xc00203a030?, 0x9?}) github.com/cometbft/cometbft-db@v0.8.0/goleveldb.go:15 +0x25 github.com/cometbft/cometbft-db.NewDB({0x2c0b742, 0x5}, {0xc000db2500, 0x9}, {0xc00203a030, 0x12}) github.com/cometbft/cometbft-db@v0.8.0/db.go:69 +0x11d github.com/tendermint/tendermint/node.DefaultDBProvider(0xc000512030) github.com/tendermint/tendermint@v0.34.28/node/node.go:78 +0xaf github.com/tendermint/tendermint/node.initDBs(0xc000237040, 0x46eefd8) github.com/tendermint/tendermint@v0.34.28/node/node.go:243 +0x129 github.com/tendermint/tendermint/node.NewNode(0xc000237040, {0x4bd0380, 0xc000ddc6e0}, 0xc002066970, {0x4bae640, 0xc0005122e8}, 0x1?, 0x646f558?, 0xc002066b80, {0x4bd58e0, ...}, ...) github.com/tendermint/tendermint@v0.34.28/node/node.go:718 +0x79 github.com/evmos/ethermint/server.startInProcess(, {{0x0, 0x0, 0x0}, {0x4bfd5c8, 0xc000db66f0}, 0x0, {0xc001ceae90, 0xc}, {0x4befe58, ...}, ...}, ...) github.com/evmos/ethermint@v0.0.0-00010101000000-000000000000/server/start.go:378 +0xf87 github.com/evmos/ethermint/server.StartCmd.func2(0xc001c02000?, {0xc001ce6ec0?, 0x0?, 0x1?}) github.com/evmos/ethermint@v0.0.0-00010101000000-000000000000/server/start.go:158 +0x225 github.com/spf13/cobra.(Command).execute(0xc001c02000, {0xc001ce6ea0, 0x1, 0x1}) github.com/spf13/cobra@v1.6.1/command.go:916 +0x862 github.com/spf13/cobra.(Command).ExecuteC(0xc001bf2000) github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd github.com/spf13/cobra.(Command).Execute(...) github.com/spf13/cobra@v1.6.1/command.go:968 github.com/spf13/cobra.(Command).ExecuteContext(...) github.com/spf13/cobra@v1.6.1/command.go:961 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x251a330?, {0x2c0bdd9, 0x6}, {0xc00061cc30, 0xd}) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/server/cmd/execute.go:36 +0x20f main.main() github.com/crypto-org-chain/cronos/v2/cmd/cronosd/main.go:13 +0x38 `

This is with kava

` kava start --log_level=trace 12:08PM INF Unlocking keyring 12:08PM INF starting ABCI with Tendermint 12:08PM INF starting node with ABCI Tendermint in-process panic: snappy: decoded block is too large

goroutine 1 [running]: github.com/golang/snappy.Encode({0xc0018e4050?, 0x12?, 0x0?}, {0xca6d6bc000?, 0x26d0b80?, 0x41464a?}) github.com/golang/snappy@v0.0.4/encode.go:22 +0x2d1 github.com/syndtr/goleveldb/leveldb/table.(Writer).writeBlock(0xc001782480, 0xc0017824e0, 0xc0017824e0?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:172 +0x139 github.com/syndtr/goleveldb/leveldb/table.(Writer).finishBlock(0xc001782480) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:223 +0x46 github.com/syndtr/goleveldb/leveldb/table.(Writer).Append(0xc001782480, {0xc802d80000, 0x12, 0x26a93c000}, {0xc802d80012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:256 +0x253 github.com/syndtr/goleveldb/leveldb.(tWriter).append(0xc000626ae0, {0xc802d80000, 0x12, 0x26a93c000}, {0xc802d80012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table.go:558 +0x1e5 github.com/syndtr/goleveldb/leveldb.(tOps).createFrom(0x26a93ad15?, {0x37f86b0, 0xc001530180}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table.go:397 +0x16d github.com/syndtr/goleveldb/leveldb.(session).flushMemdb(0xc00003d3b0, 0xc00020a8c0, 0xc00018b420, 0x0?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/sessioncompaction.go:35 +0x116 github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal(0xc000c43340) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:597 +0xcb6 github.com/syndtr/goleveldb/leveldb.openDB(0xc00003d3b0) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:131 +0x43c github.com/syndtr/goleveldb/leveldb.Open({0x37f6d90?, 0xc001351ea0?}, 0x5bc500?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:203 +0x188 github.com/syndtr/goleveldb/leveldb.OpenFile({0xc001422c80?, 0x2b6bbd7?}, 0x0) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:225 +0x5d github.com/cometbft/cometbft-db.NewGoLevelDBWithOpts({0x2b6bbd7?, 0xc00063ef30?}, {0xc0018e5b90?, 0x0?}, 0xc0001fa8f0?) github.com/cometbft/cometbft-db@v0.7.0/goleveldb.go:32 +0x7c github.com/cometbft/cometbft-db.NewGoLevelDB(...) github.com/cometbft/cometbft-db@v0.7.0/goleveldb.go:27 github.com/cometbft/cometbft-db.init.0.func1({0x2b6bbd7?, 0xc000d2c3c0?}, {0xc0018e5b90?, 0x9?}) github.com/cometbft/cometbft-db@v0.7.0/goleveldb.go:15 +0x25 github.com/cometbft/cometbft-db.NewDB({0x2b6bbd7, 0x5}, {0xc00140e510, 0x9}, {0xc0018e5b90, 0x10}) github.com/cometbft/cometbft-db@v0.7.0/db.go:64 +0x11d github.com/tendermint/tendermint/node.DefaultDBProvider(0xc000fc1fc8) github.com/tendermint/tendermint@v0.34.27/node/node.go:78 +0xaf github.com/tendermint/tendermint/node.initDBs(0xc000162000, 0x32e6f50) github.com/tendermint/tendermint@v0.34.27/node/node.go:243 +0x129 github.com/tendermint/tendermint/node.NewNode(0xc000162000, {0x37e0150, 0xc00020a640}, 0xc001b6ba40, {0x37bc5e0, 0xc000fc1db8}, 0x1?, 0x5063a90?, 0xc001b6bcd0, {0x37e5690, ...}, ...) github.com/tendermint/tendermint@v0.34.27/node/node.go:718 +0x79 github.com/evmos/ethermint/server.startInProcess(, {{0x0, 0x0, 0x0}, {0x380dbe8, 0xc000221080}, 0x0, {0x0, 0x0}, {0x3800278, ...}, ...}, ...) github.com/evmos/ethermint@v0.21.0/server/start.go:373 +0xf87 github.com/evmos/ethermint/server.StartCmd.func2(0xc001a2e900?, {0xc001b6b140?, 0x0?, 0x1?}) github.com/evmos/ethermint@v0.21.0/server/start.go:157 +0x225 github.com/spf13/cobra.(Command).execute(0xc001a2e900, {0xc001b6b120, 0x1, 0x1}) github.com/spf13/cobra@v1.6.1/command.go:916 +0x862 github.com/spf13/cobra.(Command).ExecuteC(0xc0019f8900) github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd github.com/spf13/cobra.(Command).Execute(...) github.com/spf13/cobra@v1.6.1/command.go:968 github.com/spf13/cobra.(Command).ExecuteContext(...) github.com/spf13/cobra@v1.6.1/command.go:961 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x2575720?, {0x2b69a3d, 0x4}, {0xc0006189f0, 0xb}) github.com/cosmos/cosmos-sdk@v0.46.11/server/cmd/execute.go:36 +0x20f main.main() github.com/kava-labs/kava/cmd/kava/main.go:16 +0x38

`

alpha-omega-labs commented 1 year ago

genesis.json can be found here (prettified): 135.181.5.216/genesis.json

chillyvee commented 1 year ago

The issue is a bit further down the stack (not exactly cosmos-sdk)

It looks like snappy directly affects goleveldb storage. Goleveldb can turn off compression if using more disk space is acceptable. Since you already have a fork of cometbft-db this can be done without affect on other chains.

pebbledb will not work without breaking up batches (throws error "panic: pebble: batch too large: >= 4.0 G") - hard limit does NOT look adjustable.

Did not look into the other storage backends since the are experimental/less used.

chillyvee commented 1 year ago

Here's something to adjust in your code to get around the snappy problem and you can try again. The disk usage may not be significantly different without compression. (Try and share what you experience)

It appears there might be a problem parsing in the genesis.json contents (not a snappy problem). The genesis.json document is fully read by this point.

The data/state.db is about 9.7G at the time of panic (cronosd tag v1.0.9)

Make this change

https://github.com/cometbft/cometbft-db/blob/main/goleveldb.go#L26

func NewGoLevelDB(name string, dir string) (*GoLevelDB, error) {
        return NewGoLevelDBWithOpts(name, dir, &opt.Options{Compression: opt.NoCompression})
}

The issue when using the genesis.json provided at 135.181.5.216/genesis.json is:

Unlocking keyring
starting ABCI with Tendermint
service start impl=multiAppConn module=proxy msg={} server=node
service start connection=query impl=localClient module=abci-client msg={} server=node
service start connection=snapshot impl=localClient module=abci-client msg={} server=node
service start connection=mempool impl=localClient module=abci-client msg={} server=node
service start connection=consensus impl=localClient module=abci-client msg={} server=node
service start impl=EventBus module=events msg={} server=node
service start impl=PubSub module=pubsub msg={} server=node
service start impl=IndexerService module=txindex msg={} server=node
ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 server=node software-version=1.0.9
ABCI Replay Blocks appHeight=0 module=consensus server=node stateHeight=0 storeHeight=0
initializing blockchain state from genesis.json
panic: value is nil

goroutine 80 [running]:
github.com/cosmos/cosmos-sdk/store/types.AssertValidValue(...)
        github.com/cosmos/cosmos-sdk@v0.46.11/store/types/validity.go:13
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Set(0xc04364e660?, {0xc0db34a340?, 0x1?, 0x1?}, {0x0?, 0xed?, 0xed?})
        github.com/cosmos/cosmos-sdk@v0.46.11/store/gaskv/store.go:49 +0x11c
github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.SetAccount({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/x/auth/keeper/account.go:82 +0x2c4
github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.InitGenesis({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/x/auth/keeper/genesis.go:23 +0x35e
github.com/cosmos/cosmos-sdk/x/auth.AppModule.InitGenesis({{}, {{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, ...}, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/x/auth/module.go:144 +0x125
github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/types/module/module.go:322 +0x423
github.com/crypto-org-chain/cronos/app.(*App).InitChainer(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...)
        github.com/crypto-org-chain/cronos/app/app.go:835 +0x1cf
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc000c668c0, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
        github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/abci.go:71 +0x3f5
github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc000e22240, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
        github.com/tendermint/tendermint@v0.34.27/abci/client/local_client.go:272 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0x0?, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
        github.com/tendermint/tendermint@v0.34.27/proxy/app_conn.go:77 +0x55
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, {{0x0, ...}, ...}, ...}, ...)
        github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:319 +0xd78
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000de9990, {0x3c45640, 0xc000e300d0})
        github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, ...}, ...)
        github.com/tendermint/tendermint@v0.34.27/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc00153c3c0, {0x3c29dc0, 0xc0015fb900}, 0xc00051e810, {0x3c08e20, 0xc000273158}, 0x1?, 0x53d2568?, 0xc00051ea30, {0x3c2f600, ...}, ...)
        github.com/tendermint/tendermint@v0.34.27/node/node.go:779 +0x597
github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x3c565e8, 0xc000e3db60}, 0x0, {0x0, 0x0}, {0x3c49d58, ...}, ...}, ...)
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:330 +0xb67
github.com/evmos/ethermint/server.StartCmd.func2.2()
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:144 +0x5e
github.com/evmos/ethermint/server.wrapCPUProfile.func2()
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:640 +0x29
created by github.com/evmos/ethermint/server.wrapCPUProfile
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:639 +0x37b
alpha-omega-labs commented 1 year ago

how to properly apply this? Can you please provide commit if you have time, thank you very much! "unc NewGoLevelDB(name string, dir string) (*GoLevelDB, error) { return NewGoLevelDBWithOpts(name, dir, &opt.Options{Compression: opt.NoCompression}) }

alpha-omega-labs commented 1 year ago

Since you suggested to turn off compression, did following:

versiondb/extsort/sort_test.go: SnappyCompression: true, versiondb/client/convert_to_sst.go: SnappyCompression: true, versiondb/client/restore_app_db.go: SnappyCompression: true,

Changed to false

Rebuild

Started with that genesis.json

Got similar error, but after 5m waiting and ~90GB RAM consumed (did on 300gb swap)

time cronosd start 9:49PM INF Unlocking keyring 9:49PM INF starting ABCI with Tendermint 9:49PM INF starting node with ABCI Tendermint in-process 9:54PM INF service start impl=multiAppConn module=proxy msg={} server=node 9:54PM INF service start connection=query impl=localClient module=abci-client msg={} server=node 9:54PM INF service start connection=snapshot impl=localClient module=abci-client msg={} server=node 9:54PM INF service start connection=mempool impl=localClient module=abci-client msg={} server=node 9:54PM INF service start connection=consensus impl=localClient module=abci-client msg={} server=node 9:54PM INF service start impl=EventBus module=events msg={} server=node 9:54PM INF service start impl=PubSub module=pubsub msg={} server=node 9:54PM INF service start impl=IndexerService module=txindex msg={} server=node 9:54PM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 server=node software-version=memiavl/v0.0.3-10-gc823329 9:54PM INF ABCI Replay Blocks appHeight=0 module=consensus server=node stateHeight=0 storeHeight=0 panic: snappy: decoded block is too large [recovered] panic: snappy: decoded block is too large [recovered] panic: snappy: decoded block is too large

goroutine 44 [running]: github.com/syndtr/goleveldb/leveldb.(DB).mCompaction.func1() github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:763 +0xc5 panic({0x27f39a0, 0xc000136370}) runtime/panic.go:884 +0x213 github.com/syndtr/goleveldb/leveldb.(DB).compactionTransact.func1() github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:159 +0x9b panic({0x27f39a0, 0xc000136370}) runtime/panic.go:884 +0x213 github.com/golang/snappy.Encode({0xc001cb8048?, 0x12?, 0x0?}, {0xd2e1534000?, 0x27661a0?, 0x43b8aa?}) github.com/golang/snappy@v0.0.4/encode.go:22 +0x2d1 github.com/syndtr/goleveldb/leveldb/table.(Writer).writeBlock(0xc001678240, 0xc0016782a0, 0xc0016782a0?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:172 +0x139 github.com/syndtr/goleveldb/leveldb/table.(Writer).finishBlock(0xc001678240) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:223 +0x46 github.com/syndtr/goleveldb/leveldb/table.(Writer).Append(0xc001678240, {0xd076bf8000, 0x12, 0x26a93c000}, {0xd076bf8012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table/writer.go:256 +0x253 github.com/syndtr/goleveldb/leveldb.(tWriter).append(0xc002580660, {0xd076bf8000, 0x12, 0x26a93c000}, {0xd076bf8012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table.go:558 +0x1e5 github.com/syndtr/goleveldb/leveldb.(tOps).createFrom(0xc00120d860?, {0x4be81b0, 0xc000ee8800}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/table.go:397 +0x16d github.com/syndtr/goleveldb/leveldb.(session).flushMemdb(0xc000fbc3c0, 0xc0018ed220, 0xc000e68c40, 0xc000f84b10?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/session_compaction.go:35 +0x116 github.com/syndtr/goleveldb/leveldb.(DB).memCompaction.func1(0xc002580501?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:305 +0xa5 github.com/syndtr/goleveldb/leveldb.(compactionTransactFunc).run(0x0?, 0x0?) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:242 +0x1f github.com/syndtr/goleveldb/leveldb.(DB).compactionTransact(0xc0016ca700, {0x2c16240, 0xb}, {0x4bb81b0, 0xc001cd0020}) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:186 +0x217 github.com/syndtr/goleveldb/leveldb.(DB).compactionTransactFunc(...) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:253 github.com/syndtr/goleveldb/leveldb.(DB).memCompaction(0xc0016ca700) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:303 +0x3e8 github.com/syndtr/goleveldb/leveldb.(DB).mCompaction(0xc0016ca700) github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:777 +0x93 created by github.com/syndtr/goleveldb/leveldb.openDB github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:156 +0x5d8

real 5m26.681s user 2m58.130s sys 1m19.140s

drklee3 commented 1 year ago

I have an open pr on cometbft that fixes this issue without needing to disable snappy compression: https://github.com/cometbft/cometbft/pull/1017

It removes saving the entire genesis json string into the database in a single key, which causes snappy to panic.

Disabling the snappy compression should work too with the cometbft-db modification above so that it avoids the step that causes it.

alpha-omega-labs commented 1 year ago

Amazing its already addressed by Kava, thank you! might we consider forking to Kava instead of Cronos. What do you think?

chillyvee commented 1 year ago

@drklee3 is correct. Not saving the entire genesis is another way to do this (but might break the /genesis API endpoint )

@alpha-omega-labs Sounds like you might work with Kava code, but if you need help making the suggested edit above to turn off compression, write back

alpha-omega-labs commented 1 year ago

@drklee3 is correct. Not saving the entire genesis is another way to do this (but might break the /genesis API endpoint )

@alpha-omega-labs Sounds like you might work with Kava code, but if you need help making the suggested edit above to turn off compression, write back

yes please, it would be very very helpful if you provide code way to disable compression in current cronosd or kava source code. Here is a direct code fork of cronos since we started to experiment with it because evmos went close source. https://github.com/alpha-omega-labs/genesisL1 If you can provide commit to disable compression there (or in kava) that would be highly appreciated! Thank you!

alpha-omega-labs commented 1 year ago

Here's something to adjust in your code to get around the snappy problem and you can try again. The disk usage may not be significantly different without compression. (Try and share what you experience)

It appears there might be a problem parsing in the genesis.json contents (not a snappy problem). The genesis.json document is fully read by this point.

The data/state.db is about 9.7G at the time of panic (cronosd tag v1.0.9)

Make this change

https://github.com/cometbft/cometbft-db/blob/main/goleveldb.go#L26

func NewGoLevelDB(name string, dir string) (*GoLevelDB, error) {
        return NewGoLevelDBWithOpts(name, dir, &opt.Options{Compression: opt.NoCompression})
}

The issue when using the genesis.json provided at 135.181.5.216/genesis.json is:

Unlocking keyring
starting ABCI with Tendermint
service start impl=multiAppConn module=proxy msg={} server=node
service start connection=query impl=localClient module=abci-client msg={} server=node
service start connection=snapshot impl=localClient module=abci-client msg={} server=node
service start connection=mempool impl=localClient module=abci-client msg={} server=node
service start connection=consensus impl=localClient module=abci-client msg={} server=node
service start impl=EventBus module=events msg={} server=node
service start impl=PubSub module=pubsub msg={} server=node
service start impl=IndexerService module=txindex msg={} server=node
ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 server=node software-version=1.0.9
ABCI Replay Blocks appHeight=0 module=consensus server=node stateHeight=0 storeHeight=0
initializing blockchain state from genesis.json
panic: value is nil

goroutine 80 [running]:
github.com/cosmos/cosmos-sdk/store/types.AssertValidValue(...)
        github.com/cosmos/cosmos-sdk@v0.46.11/store/types/validity.go:13
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Set(0xc04364e660?, {0xc0db34a340?, 0x1?, 0x1?}, {0x0?, 0xed?, 0xed?})
        github.com/cosmos/cosmos-sdk@v0.46.11/store/gaskv/store.go:49 +0x11c
github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.SetAccount({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/x/auth/keeper/account.go:82 +0x2c4
github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.InitGenesis({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/x/auth/keeper/genesis.go:23 +0x35e
github.com/cosmos/cosmos-sdk/x/auth.AppModule.InitGenesis({{}, {{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, ...}, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/x/auth/module.go:144 +0x125
github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.46.11/types/module/module.go:322 +0x423
github.com/crypto-org-chain/cronos/app.(*App).InitChainer(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...)
        github.com/crypto-org-chain/cronos/app/app.go:835 +0x1cf
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc000c668c0, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
        github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/abci.go:71 +0x3f5
github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc000e22240, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
        github.com/tendermint/tendermint@v0.34.27/abci/client/local_client.go:272 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0x0?, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
        github.com/tendermint/tendermint@v0.34.27/proxy/app_conn.go:77 +0x55
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, {{0x0, ...}, ...}, ...}, ...)
        github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:319 +0xd78
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000de9990, {0x3c45640, 0xc000e300d0})
        github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, ...}, ...)
        github.com/tendermint/tendermint@v0.34.27/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc00153c3c0, {0x3c29dc0, 0xc0015fb900}, 0xc00051e810, {0x3c08e20, 0xc000273158}, 0x1?, 0x53d2568?, 0xc00051ea30, {0x3c2f600, ...}, ...)
        github.com/tendermint/tendermint@v0.34.27/node/node.go:779 +0x597
github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x3c565e8, 0xc000e3db60}, 0x0, {0x0, 0x0}, {0x3c49d58, ...}, ...}, ...)
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:330 +0xb67
github.com/evmos/ethermint/server.StartCmd.func2.2()
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:144 +0x5e
github.com/evmos/ethermint/server.wrapCPUProfile.func2()
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:640 +0x29
created by github.com/evmos/ethermint/server.wrapCPUProfile
        github.com/evmos/ethermint@v0.6.1-0.20221101220534-a8ea4eceb6d9/server/start.go:639 +0x37b

Yes, you are right! Was able to get your suggestion work on both kava and cronos and got that error as you do. genesis.json originally exported as single line and was prettified to be more human readable. Might this was an issue. Will try to put original genesis.json and see what happens.

alpha-omega-labs commented 1 year ago

Got same error with non modified at all genesis.json

At the same time genesis.json seems to be visibly good - where to dig for error?

initializing blockchain state from genesis.json panic: value is nil

goroutine 1 [running]: github.com/cosmos/cosmos-sdk/store/types.AssertValidValue(...) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/store/types/validity.go:13 github.com/cosmos/cosmos-sdk/store/gaskv.(Store).Set(0x61f2560?, {0xc0fd4560a0?, 0x1?, 0x1?}, {0x0?, 0xed?, 0xed?}) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/store/gaskv/store.go:49 +0x11c github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.SetAccount({{0x4bb7548, 0xc000ee6c20}, {0x4be7b90, 0xc00139f980}, {{0x4be7b90, 0xc00139f980}, 0xc0000148b0, {0x4bb7548, 0xc000ee6c90}, {0x4bb7598, ...}, ...}, ...}, ...) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/x/auth/keeper/account.go:82 +0x2da github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.InitGenesis({{0x4bb7548, 0xc000ee6c20}, {0x4be7b90, 0xc00139f980}, {{0x4be7b90, 0xc00139f980}, 0xc0000148b0, {0x4bb7548, 0xc000ee6c90}, {0x4bb7598, ...}, ...}, ...}, ...) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/x/auth/keeper/genesis.go:23 +0x285 github.com/cosmos/cosmos-sdk/x/auth.AppModule.InitGenesis({{}, {{0x4bb7548, 0xc000ee6c20}, {0x4be7b90, 0xc00139f980}, {{0x4be7b90, 0xc00139f980}, 0xc0000148b0, {0x4bb7548, 0xc000ee6c90}, ...}, ...}, ...}, ...) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/x/auth/module.go:144 +0x125 github.com/cosmos/cosmos-sdk/types/module.(Manager).InitGenesis(, {{0x4bd4338, 0xc000056090}, {0x4be85a0, 0xc00183f6c0}, {{0x0, 0x0}, {0xc0019d2010, 0xc}, 0x67049f, ...}, ...}, ...) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/types/module/module.go:322 +0x363 github.com/crypto-org-chain/cronos/v2/app.(*App).InitChainer(, {{0x4bd4338, 0xc000056090}, {0x4be85a0, 0xc00183f6c0}, {{0x0, 0x0}, {0xc0019d2010, 0xc}, 0x67049f, ...}, ...}, ...) github.com/crypto-org-chain/cronos/v2/app/app.go:836 +0x1cf github.com/cosmos/cosmos-sdk/baseapp.(BaseApp).InitChain(0xc0015a8380, {{0x1, 0xed9ed0804, 0x0}, {0xc0019d2010, 0xc}, 0xc00013f420, {0xc000f0c000, 0x2c, 0x2c}, ...}) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/baseapp/abci.go:71 +0x3f5 github.com/tendermint/tendermint/abci/client.(localClient).InitChainSync(0xc00115ad20, {{0x1, 0xed9ed0804, 0x0}, {0xc0019d2010, 0xc}, 0xc00013f420, {0xc000f0c000, 0x2c, 0x2c}, ...}) github.com/tendermint/tendermint@v0.34.28/abci/client/local_client.go:272 +0x118 github.com/tendermint/tendermint/proxy.(appConnConsensus).InitChainSync(0xc0001327b8?, {{0x1, 0xed9ed0804, 0x0}, {0xc0019d2010, 0xc}, 0xc00013f420, {0xc000f0c000, 0x2c, 0x2c}, ...}) github.com/tendermint/tendermint@v0.34.28/proxy/app_conn.go:77 +0x55 github.com/tendermint/tendermint/consensus.(Handshaker).ReplayBlocks(, {{{0xb, 0x0}, {0x2c0d9fe, 0x7}}, {0xc0019d2010, 0xc}, 0x67049f, 0x0, {{0x0, ...}, ...}, ...}, ...) github.com/tendermint/tendermint@v0.34.28/consensus/replay.go:319 +0xf18 github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00069cf68, {0x4beb4e8, 0xc0010ee0d0}) github.com/tendermint/tendermint@v0.34.28/consensus/replay.go:268 +0x3d4 github.com/tendermint/tendermint/node.doHandshake({, }, {{{0xb, 0x0}, {0x2c0d9fe, 0x7}}, {0xc0019d2010, 0xc}, 0x67049f, 0x0, ...}, ...) github.com/tendermint/tendermint@v0.34.28/node/node.go:329 +0x1b8 github.com/tendermint/tendermint/node.NewNode(0xc00014a8c0, {0x4bd0460, 0xc001505400}, 0xc000f26670, {0x4bae720, 0xc0010f2df8}, 0x1?, 0x646f558?, 0xc000f26880, {0x4bd59c0, ...}, ...) github.com/tendermint/tendermint@v0.34.28/node/node.go:779 +0x597 github.com/evmos/ethermint/server.startInProcess(, {{0x0, 0x0, 0x0}, {0x4bfd6a8, 0xc0012c05d0}, 0x0, {0x0, 0x0}, {0x4beff38, ...}, ...}, ...) github.com/evmos/ethermint@v0.0.0-00010101000000-000000000000/server/start.go:378 +0xf87 github.com/evmos/ethermint/server.StartCmd.func2(0xc000f22000?, {0x646f558?, 0x0?, 0x0?}) github.com/evmos/ethermint@v0.0.0-00010101000000-000000000000/server/start.go:158 +0x225 github.com/spf13/cobra.(Command).execute(0xc000f22000, {0x646f558, 0x0, 0x0}) github.com/spf13/cobra@v1.6.1/command.go:916 +0x862 github.com/spf13/cobra.(Command).ExecuteC(0xc000f14000) github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd github.com/spf13/cobra.(Command).Execute(...) github.com/spf13/cobra@v1.6.1/command.go:968 github.com/spf13/cobra.(Command).ExecuteContext(...) github.com/spf13/cobra@v1.6.1/command.go:961 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x251a350?, {0x2c0bdd7, 0x6}, {0xc001171520, 0xd}) github.com/cosmos/cosmos-sdk@v0.46.14-0.20230705083228-b39cdb2e74ad/server/cmd/execute.go:36 +0x20f main.main() github.com/crypto-org-chain/cronos/v2/cmd/cronosd/main.go:13 +0x38

alpha-omega-labs commented 1 year ago

searching for a null in genesis.json returned: Null value at app_state.authz.authorization.1.authorization.max_tokens ... ... ... Null value at app_state.authz.authorization.1355.authorization.max_tokens

Null value at app_state.ibc.client_genesis.clients.0.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.0.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.1.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.1.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.2.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.2.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.3.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.3.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.4.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.4.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.5.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.5.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.6.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.6.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.7.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.7.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.8.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.8.client_state.proof_specs.1.inner_spec.empty_child

Null value at app_state.params

Where is most likely fail do you think?