search

cosmos / gaia

Cosmos Hub
https://hub.cosmos.network
Apache License 2.0
464 stars 678 forks source link

our nodes can't connect to seeds suddenly stopped syncing after running for months #676

Closed zebpay-peswani closed 3 years ago

zebpay-peswani commented 3 years ago

Summary of Bug

Hi Suddenly our 3 nodes stopped syncing. I tried changing seeds but does not workd here are the logs https://pastebin.com/GUmdVFeN please help it's urgent

Version

gaiacli version -> 2.0.15

gaiad version -> 2.0.15

Steps to Reproduce

It was working fine for months and stopped suddenly

tac0turtle commented 3 years ago

There was a chain upgrade. You will need to download the new version of gaia (v4.0.3) get the new genesis located at github.com/cosmos/mainnet.

zebpay-peswani commented 3 years ago

can i upgrade from 2.0.13 to 4 directly or i need 2.0.15?

tac0turtle commented 3 years ago

You will need to reset, you state and download the new version (v4.0.3).

zebpay-peswani commented 3 years ago

perl -i -pe 's/^halt-time =.*/halt-time = 1613628000/' /blockchain/.gaiad/config/app.toml

cat /blockchain/.gaiad/data/priv_validator_state.json | jq '.height'

// data is in /blockchain/.gaiad

this line always gives 0 height however our node s synced to height 5200791 . The data folder is about 254GB

also I upgraded to 2.0.15

gaiad export --home=/blockchain/.gaiad --height=5200791 > cosmoshub_3_genesis_export.json

above command does not export the whole data and finish in seconds however in docs it say it may take an hour

zebpay-peswani commented 3 years ago

I upgrade one of our node but getting below error when starting

ubuntu@ip-172-31-6-22:/blockchain/.gaia/config$ gaiad start --home=/blockchain/.gaia
12:27PM INF starting ABCI with Tendermint
panic: proto: Int64Value: wiretype end group for non-group

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/store/rootmulti.getLatestVersion(0x2090aa0, 0xc0001243b8, 0x2079740)
    github.com/cosmos/cosmos-sdk@v0.41.2/store/rootmulti/store.go:908 +0x10b
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).LoadLatestVersion(0xc00016e320, 0x2079740, 0xc000100400)
    github.com/cosmos/cosmos-sdk@v0.41.2/store/rootmulti/store.go:158 +0x37
github.com/cosmos/cosmos-sdk/baseapp.DefaultStoreLoader(0x2098100, 0xc00016e320, 0xc0001214d0, 0xc00126d308)
    github.com/cosmos/cosmos-sdk@v0.41.2/baseapp/baseapp.go:258 +0x31
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).LoadLatestVersion(0xc00009f6c0, 0xc0010b8c10, 0x7f9d48eeaf80)
    github.com/cosmos/cosmos-sdk@v0.41.2/baseapp/baseapp.go:248 +0x42
github.com/cosmos/gaia/v4/app.NewGaiaApp(0x2079700, 0xc00104bbc0, 0x2090aa0, 0xc0001243b8, 0x0, 0x0, 0xc0016eba01, 0xc0016e7e60, 0x7ffd78772d95, 0x11, ...)
    github.com/cosmos/gaia/v4/app/app.go:409 +0x68ea
github.com/cosmos/gaia/v4/cmd/gaiad/cmd.newApp(0x2079700, 0xc00104bbc0, 0x2090aa0, 0xc0001243b8, 0x0, 0x0, 0x203a800, 0xc000f6ca20, 0x98, 0x195a1e0)
    github.com/cosmos/gaia/v4/cmd/gaiad/cmd/root.go:177 +0x9cf
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000efa1c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20816c0, 0xc000dfc6a0, ...)
    github.com/cosmos/cosmos-sdk@v0.41.2/server/start.go:236 +0x1dc
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000e75680, 0xc000ed1cd0, 0x0, 0x1, 0x0, 0x0)
    github.com/cosmos/cosmos-sdk@v0.41.2/server/start.go:120 +0x1c8
github.com/spf13/cobra.(*Command).execute(0xc000e75680, 0xc000ed1cc0, 0x1, 0x1, 0xc000e75680, 0xc000ed1cc0)
    github.com/spf13/cobra@v1.1.3/command.go:852 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000dc2780, 0x0, 0x0, 0xc000dfa360)
    github.com/spf13/cobra@v1.1.3/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v1.1.3/command.go:897
github.com/spf13/cobra.(*Command).ExecuteContext(...)
    github.com/spf13/cobra@v1.1.3/command.go:890
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc000dc2780, 0xc000dfa360, 0x12, 0x20941e0, 0xc000dfc6a0)
    github.com/cosmos/cosmos-sdk@v0.41.2/server/cmd/execute.go:36 +0x265
main.main()

Here is the version details

name: gaia
server_name: gaiad
version: 4.0.2
commit: 6d46572f3273423ad9562cf249a86ecc8206e207
build_tags: netgo,ledger
go: go version go1.15.8 linux/amd64
build_deps:
- github.com/99designs/keyring@v1.1.6
- github.com/ChainSafe/go-schnorrkel@v0.0.0-20200405005733-88cbf1b4c40d
- github.com/Workiva/go-datastructures@v1.0.52
- github.com/aristanetworks/goarista@v0.0.0-20170210015632-ea17b1a17847
- github.com/armon/go-metrics@v0.3.6
- github.com/beorn7/perks@v1.0.1
- github.com/bgentry/speakeasy@v0.1.0
- github.com/btcsuite/btcd@v0.21.0-beta
- github.com/btcsuite/btcutil@v1.0.2
- github.com/cespare/xxhash/v2@v2.1.1
- github.com/confio/ics23/go@v0.6.3
- github.com/cosmos/cosmos-sdk@v0.41.2
- github.com/cosmos/go-bip39@v1.0.0
- github.com/cosmos/iavl@v0.15.3
- github.com/cosmos/ledger-cosmos-go@v0.11.1
- github.com/cosmos/ledger-go@v0.9.2
- github.com/davecgh/go-spew@v1.1.1
- github.com/dvsekhvalnov/jose2go@v0.0.0-20200901110807-248326c1351b
- github.com/enigmampc/btcutil@v1.0.3-0.20200723161021-e2fb6adb2a25
- github.com/ethereum/go-ethereum@v1.9.25
- github.com/felixge/httpsnoop@v1.0.1
- github.com/fsnotify/fsnotify@v1.4.9
- github.com/go-kit/kit@v0.10.0
- github.com/go-logfmt/logfmt@v0.5.0
- github.com/godbus/dbus@v0.0.0-20190726142602-4481cbc300e2
- github.com/gogo/gateway@v1.1.0
- github.com/gogo/protobuf@v1.3.3 => github.com/regen-network/protobuf@v1.3.3-alpha.regen.1
- github.com/golang/protobuf@v1.4.3
- github.com/golang/snappy@v0.0.3-0.20201103224600-674baa8c7fc3
- github.com/google/btree@v1.0.0
- github.com/gorilla/handlers@v1.5.1
- github.com/gorilla/mux@v1.8.0
- github.com/gorilla/websocket@v1.4.2
- github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2
- github.com/grpc-ecosystem/grpc-gateway@v1.16.0
- github.com/gsterjov/go-libsecret@v0.0.0-20161001094733-a6f4afe4910c
- github.com/gtank/merlin@v0.1.1
- github.com/gtank/ristretto255@v0.1.2
- github.com/hashicorp/go-immutable-radix@v1.0.0
- github.com/hashicorp/golang-lru@v0.5.4
- github.com/hashicorp/hcl@v1.0.0
- github.com/libp2p/go-buffer-pool@v0.0.2
- github.com/magiconair/properties@v1.8.4
- github.com/mattn/go-isatty@v0.0.12
- github.com/matttproud/golang_protobuf_extensions@v1.0.1
- github.com/mimoo/StrobeGo@v0.0.0-20181016162300-f8f6d4d2b643
- github.com/minio/highwayhash@v1.0.1
- github.com/mitchellh/go-homedir@v1.1.0
- github.com/mitchellh/mapstructure@v1.1.2
- github.com/mtibben/percent@v0.2.1
- github.com/pelletier/go-toml@v1.8.0
- github.com/pkg/errors@v0.9.1
- github.com/pmezard/go-difflib@v1.0.0
- github.com/prometheus/client_golang@v1.8.0
- github.com/prometheus/client_model@v0.2.0
- github.com/prometheus/common@v0.15.0
- github.com/prometheus/procfs@v0.2.0
- github.com/rakyll/statik@v0.1.7
- github.com/rcrowley/go-metrics@v0.0.0-20200313005456-10cdbea86bc0
- github.com/regen-network/cosmos-proto@v0.3.1
- github.com/rs/cors@v1.7.0
- github.com/rs/zerolog@v1.20.0
- github.com/spf13/afero@v1.3.4
- github.com/spf13/cast@v1.3.1
- github.com/spf13/cobra@v1.1.3
- github.com/spf13/jwalterweatherman@v1.1.0
- github.com/spf13/pflag@v1.0.5
- github.com/spf13/viper@v1.7.1
- github.com/stretchr/testify@v1.7.0
- github.com/subosito/gotenv@v1.2.0
- github.com/syndtr/goleveldb@v1.0.1-0.20200815110645-5c35d600f0ca
- github.com/tendermint/btcd@v0.1.1
- github.com/tendermint/crypto@v0.0.0-20191022145703-50d29ede1e15
- github.com/tendermint/go-amino@v0.16.0
- github.com/tendermint/tendermint@v0.34.7
- github.com/tendermint/tm-db@v0.6.4
- github.com/zondax/hid@v0.9.0
- golang.org/x/crypto@v0.0.0-20201221181555-eec23a3978ad
- golang.org/x/net@v0.0.0-20201021035429-f5854403a974
- golang.org/x/sys@v0.0.0-20201015000850-e3ed0017c211
- golang.org/x/term@v0.0.0-20201117132131-f5c789dd3221
- golang.org/x/text@v0.3.3
- google.golang.org/genproto@v0.0.0-20210114201628-6edceaf6022f
- google.golang.org/grpc@v1.35.0
- google.golang.org/protobuf@v1.25.0
- gopkg.in/ini.v1@v1.51.0
- gopkg.in/yaml.v2@v2.4.0
- gopkg.in/yaml.v3@v3.0.0-20200313102051-9f266ea9e77c
tac0turtle commented 3 years ago

You don't need to export. You can use the genesis file located at github.com/cosmos/mainnet.

can you also run gaiad version

zebpay-peswani commented 3 years ago

can you please help in above error I given the version also

tac0turtle commented 3 years ago

The above errors stem from version mismatches. Can you run unsafe_rest_all and try again

zebpay-peswani commented 3 years ago 
ubuntu@ip-172-31-6-22:/blockchain/.gaia$ gaiad start --home=/blockchain/.gaia
1:22PM INF starting ABCI with Tendermint
1:22PM INF Starting multiAppConn service impl={"Logger":{}} module=proxy
1:22PM INF Starting localClient service connection=query impl="marshaling error: json: unsupported type: abcicli.Callback" module=abci-client
1:22PM INF Starting localClient service connection=snapshot impl="marshaling error: json: unsupported type: abcicli.Callback" module=abci-client
1:22PM INF Starting localClient service connection=mempool impl="marshaling error: json: unsupported type: abcicli.Callback" module=abci-client
1:22PM INF Starting localClient service connection=consensus impl="marshaling error: json: unsupported type: abcicli.Callback" module=abci-client
1:22PM INF Starting EventBus service impl={"Logger":{}} module=events
1:22PM INF Starting PubSub service impl={"Logger":{}} module=pubsub
1:22PM INF Starting IndexerService service impl={"Logger":{}} module=txindex
1:22PM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=
1:22PM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
tac0turtle commented 3 years ago

Yup and now you wait a while for it to start. You can skip this by using state sync, but then you won't have the complete state.

zebpay-peswani commented 3 years ago

After running overnight I got below error. 

5:58PM INF Saving AddrBook to file book=/blockchain/.gaia/config/addrbook.json module=p2p size=0
5:58PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
5:58PM INF No addresses to dial. Falling back to seeds module=pex
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/server/config.GetConfig(0xc000fa8120, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/cosmos/cosmos-sdk@v0.41.2/server/config/config.go:197 +0xc19
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000f2c240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20816c0, 0xc000e1e4e0, ...)
        github.com/cosmos/cosmos-sdk@v0.41.2/server/start.go:264 +0x66f
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000e68a00, 0xc000f0db10, 0x0, 0x1, 0x0, 0x0)
        github.com/cosmos/cosmos-sdk@v0.41.2/server/start.go:120 +0x1c8
github.com/spf13/cobra.(*Command).execute(0xc000e68a00, 0xc000f0db00, 0x1, 0x1, 0xc000e68a00, 0xc000f0db00)
        github.com/spf13/cobra@v1.1.3/command.go:852 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000c5d900, 0x0, 0x0, 0xc000e0c8e0)
        github.com/spf13/cobra@v1.1.3/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.1.3/command.go:897
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.1.3/command.go:890
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc000c5d900, 0xc000e0c8e0, 0x12, 0x20941e0, 0xc000e1e4e0)
        github.com/cosmos/cosmos-sdk@v0.41.2/server/cmd/execute.go:36 +0x265
main.main()
        github.com/cosmos/gaia/v4/cmd/gaiad/main.go:16 +0x45
zebpay-peswani commented 3 years ago

@THEBOLCK79 I did not get?