search

quicksilver-zone / quicksilver

Quicksilver
https://app.quicksilver.zone
Other
62 stars 60 forks source link

Sync error #113

Closed shmutalov closed 2 years ago

shmutalov commented 2 years ago

There is a sync error on clean node (rhapsody-5):

8:04PM INF executed block height=16510 module=state num_invalid_txs=0 num_valid_txs=0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdcbb13]
goroutine 33551 [running]:
github.com/cosmos/iavl.(*nodeDB).incrVersionReaders(0x2569908?, 0x20?)
        github.com/cosmos/iavl@v0.17.3/nodedb.go:619 +0x33
github.com/cosmos/iavl.newExporter(0xc0104e0a50)
        github.com/cosmos/iavl@v0.17.3/export.go:45 +0xc8
github.com/cosmos/iavl.(*ImmutableTree).Export(...)
        github.com/cosmos/iavl@v0.17.3/immutable_tree.go:142
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Export(0xc00f216660?, 0x2559838?)
        github.com/cosmos/cosmos-sdk@v0.45.4/store/iavl/store.go:248 +0x52
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Snapshot(0xc001202000, 0x4074, {0x25502e0, 0xc01045c360})
        github.com/cosmos/cosmos-sdk@v0.45.4/store/rootmulti/store.go:693 +0x551
github.com/cosmos/cosmos-sdk/snapshots.(*Manager).createSnapshot(0xc001222460, 0xc2a506?, 0xc0001dc600?)
        github.com/cosmos/cosmos-sdk@v0.45.4/snapshots/manager.go:180 +0xb1
created by github.com/cosmos/cosmos-sdk/snapshots.(*Manager).Create
        github.com/cosmos/cosmos-sdk@v0.45.4/snapshots/manager.go:167 +0x1ee

Node restarts, then sync continues from last synced block and... crashes at random block again

Quicksilver version:

shmutalov@silver:~$ quicksilverd version --long
name: quicksilver
server_name: quicksilverd
version: v0.3.0
commit: 5f3df37f05439274056f3e0ebcbd44412cff28f1
build_tags: netgo,
go: go version go1.18.3 linux/amd64
build_deps:
- filippo.io/edwards25519@v1.0.0-beta.2
- github.com/99designs/keyring@v1.1.6 => github.com/cosmos/keyring@v1.1.7-0.20210622111912-ef00f8ac3d76
- github.com/ChainSafe/go-schnorrkel@v0.0.0-20200405005733-88cbf1b4c40d
- github.com/Workiva/go-datastructures@v1.0.53
- github.com/armon/go-metrics@v0.3.10
- github.com/beorn7/perks@v1.0.1
- github.com/bgentry/speakeasy@v0.1.0
- github.com/btcsuite/btcd@v0.22.1
- github.com/cenkalti/backoff/v4@v4.1.1
- github.com/cespare/xxhash/v2@v2.1.2
- github.com/coinbase/rosetta-sdk-go@v0.7.0
- github.com/confio/ics23/go@v0.7.0
- github.com/cosmos/btcutil@v1.0.4
- github.com/cosmos/cosmos-proto@v1.0.0-alpha7
- github.com/cosmos/cosmos-sdk@v0.45.4 => github.com/ingenuity-build/cosmos-sdk@v0.45.4-ls
- github.com/cosmos/go-bip39@v1.0.0
- github.com/cosmos/iavl@v0.17.3
- github.com/cosmos/ibc-go/v3@v3.0.0
- github.com/davecgh/go-spew@v1.1.1
- github.com/desertbit/timer@v0.0.0-20180107155436-c41aec40b27f
- github.com/dvsekhvalnov/jose2go@v0.0.0-20200901110807-248326c1351b
- github.com/felixge/httpsnoop@v1.0.1
- github.com/fsnotify/fsnotify@v1.5.4
- github.com/go-kit/kit@v0.12.0
- github.com/go-kit/log@v0.2.0
- github.com/go-logfmt/logfmt@v0.5.1
- github.com/godbus/dbus@v0.0.0-20190726142602-4481cbc300e2
- github.com/gogo/gateway@v1.1.0
- github.com/gogo/protobuf@v1.3.3 => github.com/regen-network/protobuf@v1.3.3-alpha.regen.1
- github.com/golang/protobuf@v1.5.2
- github.com/golang/snappy@v0.0.4
- github.com/google/btree@v1.0.1
- github.com/google/orderedcode@v0.0.1
- github.com/gorilla/handlers@v1.5.1
- github.com/gorilla/mux@v1.8.0
- github.com/gorilla/websocket@v1.5.0
- github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0
- github.com/grpc-ecosystem/grpc-gateway@v1.16.0
- github.com/gsterjov/go-libsecret@v0.0.0-20161001094733-a6f4afe4910c
- github.com/gtank/merlin@v0.1.1
- github.com/gtank/ristretto255@v0.1.2
- github.com/hashicorp/go-immutable-radix@v1.3.1
- github.com/hashicorp/golang-lru@v0.5.5-0.20210104140557-80c98217689d
- github.com/hashicorp/hcl@v1.0.0
- github.com/hdevalence/ed25519consensus@v0.0.0-20210204194344-59a8610d2b87
- github.com/improbable-eng/grpc-web@v0.15.0
- github.com/klauspost/compress@v1.13.6
- github.com/lib/pq@v1.10.6
- github.com/libp2p/go-buffer-pool@v0.0.2
- github.com/magiconair/properties@v1.8.6
- github.com/mattn/go-isatty@v0.0.14
- github.com/matttproud/golang_protobuf_extensions@v1.0.1
- github.com/mimoo/StrobeGo@v0.0.0-20181016162300-f8f6d4d2b643
- github.com/minio/highwayhash@v1.0.2
- github.com/mitchellh/mapstructure@v1.5.0
- github.com/mtibben/percent@v0.2.1
- github.com/pelletier/go-toml@v1.9.5
- github.com/pkg/errors@v0.9.1
- github.com/pmezard/go-difflib@v1.0.0
- github.com/prometheus/client_golang@v1.12.2
- github.com/prometheus/client_model@v0.2.0
- github.com/prometheus/common@v0.32.1
- github.com/prometheus/procfs@v0.7.3
- github.com/rakyll/statik@v0.1.7
- github.com/rcrowley/go-metrics@v0.0.0-20201227073835-cf1acfcdf475
- github.com/regen-network/cosmos-proto@v0.3.1
- github.com/rs/cors@v1.8.2
- github.com/rs/zerolog@v1.26.1
- github.com/spf13/afero@v1.8.2
- github.com/spf13/cast@v1.4.1
- github.com/spf13/cobra@v1.4.0
- github.com/spf13/jwalterweatherman@v1.1.0
- github.com/spf13/pflag@v1.0.5
- github.com/spf13/viper@v1.11.0
- github.com/stretchr/testify@v1.7.1
- github.com/subosito/gotenv@v1.2.0
- github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7
- github.com/tendermint/btcd@v0.1.1
- github.com/tendermint/crypto@v0.0.0-20191022145703-50d29ede1e15
- github.com/tendermint/go-amino@v0.16.0
- github.com/tendermint/tendermint@v0.34.19
- github.com/tendermint/tm-db@v0.6.7 => github.com/notional-labs/tm-db@v0.6.7-0.20220413113722-16f0d618a72b
- go.opencensus.io@v0.23.0
- golang.org/x/crypto@v0.0.0-20220411220226-7b82a4e95df4
- golang.org/x/net@v0.0.0-20220412020605-290c469a71a5
- golang.org/x/sys@v0.0.0-20220422013727-9388b58f7150
- golang.org/x/term@v0.0.0-20210927222741-03fcf44c2211
- golang.org/x/text@v0.3.7
- google.golang.org/genproto@v0.0.0-20220407144326-9054f6ed7bac
- google.golang.org/grpc@v1.46.2
- google.golang.org/protobuf@v1.28.0
- gopkg.in/ini.v1@v1.66.4
- gopkg.in/yaml.v2@v2.4.0
- gopkg.in/yaml.v3@v3.0.0-20210107192922-496545a6307b
- nhooyr.io/websocket@v1.8.7
cosmos_sdk_version: v0.45.4

It seems crash on every 2000 blocks synced

joe-bowman commented 2 years ago

Hey thanks for this report; can you dump here your contents of config.toml and app.toml please?

shmutalov commented 2 years ago

@joe-bowman

config.toml and app.toml as you requested.

joe-bowman commented 2 years ago

Thank you; there looks to be nothing problematic here, and nobody else has reported an issue.

Could you try resyncing with: snapshot-interval = 0 - this disables snapshot creation.

What is the memory on the machine?

shmutalov commented 2 years ago

I have already synced, but it was interrupted many times before full sync.

4x CPU and 16 GB RAM

shmutalov commented 2 years ago

New one:

6:29AM INF IBC fungible token transfer amount=1 module=x/ibc-transfer receiver=cosmos1896r2pq7j7qal26t6k9whqehqy3tzsn8t8se0j sender=quick1896r2pq7j7qal26t6k9whqehqy3tzsn8qrqtkq token=uqck
panic: runtime error: index out of range [23] with length 23 [recovered]
        panic: runtime error: index out of range [23] with length 23
last events:
0: scBlockReceived{48146#F6BA7E7009249C2EC4AA91C059F1EE7BBE43C2FF39E177ED2B60CA46BFBF0C7D from aed219bea697ffc9f69eacdb3ea2e86b55874a39}
1: scBlockReceived{48148#B002B3D23E8242EA6574A59ECEB04A27A0604E61FA33F62B898D5C758E72D111 from f8a583eee2db1166739a4a294403c2a0c63a58d9}
2: scBlockReceived{48149#136F7840E66308A86C55BB772E956D211BE3449C21C820D34B7FCC790CA5BF53 from 0bd73406b15bee362c17c8d2d9276f500d04b3d1}
3: scBlockReceived{48150#0ECFCE2ED170889F27D868A1B158CEB624676F0A7B8B5B19062C52BC8172C513 from aed219bea697ffc9f69eacdb3ea2e86b55874a39}
4: scFinishedEv{after try prune}
5: scBlockReceived{48147#2F3736BD8B9BB1F767D09D33726ACCCA4A6D09CDB8214777789864568ED19500 from bc04d4a854e39dbadb543c0531ba1d5d15bad008}
6: scBlockReceived{48145#EC5CEAA7CB68D020958F9F2355DD4C4EECD22C20E1828742768A7D9D8A9A6D36 from 0bd73406b15bee362c17c8d2d9276f500d04b3d1}
7: scBlockReceived{48143#197A5D6ADB1680D20D188CAA2554C0880A793E4D43736E7797DB06A0D8FBBA79 from aed219bea697ffc9f69eacdb3ea2e86b55874a39}
8: scBlockReceived{48141#F816471ECF04CBBBDD02E3D0B6088AD826BD5A99775E05440DC7CAC1CF6D8C1F from bc04d4a854e39dbadb543c0531ba1d5d15bad008}
9: scBlockReceived{48144#D4AA9A237BEFE954BD1B8E08DF3C382D194A76487EE237A84C8F07AFEB5EA417 from f8a583eee2db1166739a4a294403c2a0c63a58d9}
10: scBlockReceived{48142#C7BD2C98917003D81A231B0D7734E57FBF34D002588592CFD53C65185ECE9216 from 0bd73406b15bee362c17c8d2d9276f500d04b3d1}
goroutine 74 [running]:
github.com/tendermint/tendermint/blockchain/v2.(*Routine).start.func1()
        github.com/tendermint/tendermint@v0.34.19/blockchain/v2/routine.go:77 +0x24e
panic({0x1b13de0, 0xc006b7f728})
        runtime/panic.go:838 +0x207
github.com/tendermint/tendermint/state.execBlockOnProxyApp.func1(0x40f407?, 0x10?)
        github.com/tendermint/tendermint@v0.34.19/state/execution.go:287 +0x1cf
github.com/tendermint/tendermint/abci/client.(*localClient).callback(0x20?, 0x0?, 0x0?)
        github.com/tendermint/tendermint@v0.34.19/abci/client/local_client.go:329 +0x2d
github.com/tendermint/tendermint/abci/client.(*localClient).DeliverTxAsync(0xc000e41500, {{0xc00b0ac300?, 0xcc3e64?, 0xc000e41500?}})
        github.com/tendermint/tendermint@v0.34.19/abci/client/local_client.go:94 +0x2fd
github.com/tendermint/tendermint/proxy.(*appConnConsensus).DeliverTxAsync(0xc007098220?, {{0xc00b0ac300?, 0x20?, 0xb?}})
        github.com/tendermint/tendermint@v0.34.19/proxy/app_conn.go:85 +0x26
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2579680?, 0xc0010db440}, {0x257e890, 0xc000c37450}, 0xc000001c20, {0x2584768, 0xc00002ce20}, 0xbc10?)
        github.com/tendermint/tendermint@v0.34.19/state/execution.go:320 +0x822
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc002621db0, 0x8}}, {0xc002621dc0, 0xd}, 0x1, 0xbc10, {{0xc00a4a6fc0, ...}, ...}, ...}, ...)
        github.com/tendermint/tendermint@v0.34.19/state/execution.go:140 +0x171
github.com/tendermint/tendermint/blockchain/v2.(*pContext).applyBlock(0xc000da6420, {{0xc002d898e0, 0x20, 0x20}, {0x2, {0xc002d894e0, 0x20, 0x20}}}, 0xc002621dc0?)
        github.com/tendermint/tendermint@v0.34.19/blockchain/v2/processor_context.go:33 +0x14f
github.com/tendermint/tendermint/blockchain/v2.(*pcState).handle(0xc0044e0d20, {0x25632c0?, 0x33a4568?})
        github.com/tendermint/tendermint@v0.34.19/blockchain/v2/processor.go:181 +0x3a8
github.com/tendermint/tendermint/blockchain/v2.(*Routine).start(0xc000152a10)
        github.com/tendermint/tendermint@v0.34.19/blockchain/v2/routine.go:94 +0x232
created by github.com/tendermint/tendermint/blockchain/v2.(*BlockchainReactor).startSync
        github.com/tendermint/tendermint@v0.34.19/blockchain/v2/reactor.go:152 +0x17d
quicksilverd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
quicksilverd.service: Failed with result 'exit-code'.
joe-bowman commented 2 years ago

These are both Tendermint issues, rather than Quicksilver specific - and we've seen no recurrences thus far. Closing as stale. Will re-open if they start presenting themselves again.