search

vegaprotocol / vegacapsule

Run a complete Vega Protocol network locally
MIT License
4 stars 7 forks source link

Unable to start network with 3 data nodes #304

Closed MuthuVega closed 1 year ago

MuthuVega commented 1 year ago

I am unable to start a network with 3 data nodes.

This is to test the de-centralised feature and currently not all changes are merged in - so using the below branches for the various repos.

Repo Branches

vega - dehistory-path-fix
vegacapsule - revert-302-revert-300-fix-config-templating
system tests - de-history

Below is the full stack of logs when trying to start a network with 3 data nodes.

Muthus-MacBook-Pro:scripts muthupandianvadivelu$ make vegacapsule-start-network
Check local environment: Start
WORK_DIR=/Users/muthupandianvadivelu/muthu/vega/repos
SYSTEM_TEST_DIR=/Users/muthupandianvadivelu/muthu/vega/repos/system-tests
VEGA_DIR=/Users/muthupandianvadivelu/muthu/vega/repos/vega
DEVOPS_DIR=/Users/muthupandianvadivelu/muthu/vega/repos/devops-infra
VEGATOOLS_DIR=/Users/muthupandianvadivelu/muthu/vega/repos/vegatools
DOCKER_GOCACHE=/Users/muthupandianvadivelu/muthu/vega/repos/system-tests/go/cache
GOPATH=/Users/muthupandianvadivelu/go
GOPRIVATE=
GO111MODULE=on
CGO_ENABLED=0
GOOS=linux
GOARCH=arm64
GOPROXY=
SYSTEM_TESTS_ENV=127.0.0.1
VALIDATOR_NODE_COUNT=2
NON_VALIDATOR_NODE_COUNT=1
SYSTEM_TESTS_PORTBASE=1000
SYSTEM_TESTS_DOCKER_IMAGE_TAG=local
VEGA_CORE_IMAGE_TAG=local
DATA_NODE_IMAGE_TAG=local
TEST_FUNCTION=
TEST_DIRECTORY=tests
TEST_MARK=
TEST_EXTRA_PYTEST_ARGS=
UID=501
GID=20
VEGA_LOG_LEVEL=Info
VEGACAPSULE_CONFIG_FILENAME=/Users/muthupandianvadivelu/muthu/vega/repos/system-tests/vegacapsule/capsule_config.hcl
TESTS_DIR=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata
NETWORK_HOME_PATH=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet
smartcontractsmount=-v /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/smartcontracts:/workspace/smartcontracts -e SMART_CONTRACTS_ADDRESSES=/workspace/smartcontracts/addresses.json
MAINNET_TEST_CASE=
=== Check Successful ===
Check local environment: End
Additional check steps for vegacapsule: Start
VEGACAPSULE_DIR=/Users/muthupandianvadivelu/muthu/vega/repos/vegacapsule
VEGACAPSULE_NET_ADDR=
GOBIN=/Users/muthupandianvadivelu/go/bin
=== Check successful ===
Additional check steps for vegacapsule: End
2022/10/28 11:33:45 generating network
2022/10/28 11:33:45 Initiating faucet with: /usr/local/bin/vega [faucet init --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/faucet --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/faucet/faucet-wallet-pass.txt --output json]
2022/10/28 11:33:45 Initiating Tendermint node "full" with: /usr/local/bin/vega [tm init full --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node4]
2022/10/28 11:33:45 Initiating Tendermint node "validator" with: /usr/local/bin/vega [tm init validator --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node1]
2022/10/28 11:33:45 Initiating Tendermint node "full" with: /usr/local/bin/vega [tm init full --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node3]
2022/10/28 11:33:45 Initiating Tendermint node "full" with: /usr/local/bin/vega [tm init full --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node2]
2022/10/28 11:33:45 Initiating Tendermint node "validator" with: /usr/local/bin/vega [tm init validator --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node0]
2022/10/28 11:33:46 I[2022-10-28|11:33:46.017] Generated private validator                  module=main keyFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node0/config/priv_validator_key.json stateFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node0/data/priv_validator_state.json
I[2022-10-28|11:33:46.018] Generated node key                           module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node0/config/node_key.json
I[2022-10-28|11:33:46.018] Generated genesis file                       module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node0/config/genesis.json

2022/10/28 11:33:46 Initiating node "validator" with: /usr/local/bin/vega [init --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0 --nodewallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt --no-tendermint --output json validator]
2022/10/28 11:33:46 I[2022-10-28|11:33:46.023] Generated private validator                  module=main keyFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node3/config/priv_validator_key.json stateFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node3/data/priv_validator_state.json
I[2022-10-28|11:33:46.024] Generated node key                           module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node3/config/node_key.json
I[2022-10-28|11:33:46.024] Generated genesis file                       module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node3/config/genesis.json

2022/10/28 11:33:46 Initiating node "full" with: /usr/local/bin/vega [init --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node3 --nodewallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node3/node-vega-wallet-pass.txt --no-tendermint --output json full]
2022/10/28 11:33:46 I[2022-10-28|11:33:46.024] Generated private validator                  module=main keyFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node1/config/priv_validator_key.json stateFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node1/data/priv_validator_state.json
I[2022-10-28|11:33:46.024] Generated node key                           module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node1/config/node_key.json
I[2022-10-28|11:33:46.024] Generated genesis file                       module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node1/config/genesis.json

2022/10/28 11:33:46 I[2022-10-28|11:33:46.025] Generated private validator                  module=main keyFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node4/config/priv_validator_key.json stateFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node4/data/priv_validator_state.json
I[2022-10-28|11:33:46.026] Generated node key                           module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node4/config/node_key.json
I[2022-10-28|11:33:46.026] Generated genesis file                       module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node4/config/genesis.json

2022/10/28 11:33:46 Initiating node "validator" with: /usr/local/bin/vega [init --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1 --nodewallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt --no-tendermint --output json validator]
2022/10/28 11:33:46 Initiating node "full" with: /usr/local/bin/vega [init --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node4 --nodewallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node4/node-vega-wallet-pass.txt --no-tendermint --output json full]
2022/10/28 11:33:46 I[2022-10-28|11:33:46.042] Generated private validator                  module=main keyFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node2/config/priv_validator_key.json stateFile=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node2/data/priv_validator_state.json
I[2022-10-28|11:33:46.042] Generated node key                           module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node2/config/node_key.json
I[2022-10-28|11:33:46.042] Generated genesis file                       module=main path=/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node2/config/genesis.json

2022/10/28 11:33:46 Initiating node "full" with: /usr/local/bin/vega [init --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node2 --nodewallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node2/node-vega-wallet-pass.txt --no-tendermint --output json full]
2022/10/28 11:33:46 vega config initialized for node "full" with id 4, paths: "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node4/config/node/config.toml"
2022/10/28 11:33:46 Initiating data node with: /usr/local/bin/vega [datanode init -f --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/data/node4 testnet]
2022/10/28 11:33:46 Creating vega wallet with: /usr/local/bin/vega [wallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0 create --output json --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/vega-wallet-pass.txt --wallet created-wallet]
2022/10/28 11:33:46 Creating vega wallet with: /usr/local/bin/vega [wallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1 create --output json --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/vega-wallet-pass.txt --wallet created-wallet]
2022/10/28 11:33:46 vega config initialized for node "full" with id 3, paths: "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node3/config/node/config.toml"
2022/10/28 11:33:46 Initiating data node with: /usr/local/bin/vega [datanode init -f --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/data/node3 testnet]
2022/10/28 11:33:46 vega config initialized for node "full" with id 2, paths: "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node2/config/node/config.toml"
2022/10/28 11:33:46 Initiating data node with: /usr/local/bin/vega [datanode init -f --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/data/node2 testnet]
2022/10/28 11:33:46 2022-10-28T11:33:46.226+0100    INFO    root    commands/init.go:72 configuration generated successfully    {"path": "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/data/node4/config/data-node/config.toml"}

2022/10/28 11:33:46 node wallet create out: &vega.createWalletOutput{Wallet:struct { Name string "json:\"name\""; FilePath string "json:\"filePath\""; RecoveryPhrase string "json:\"recoveryPhrase\"" }{Name:"created-wallet", FilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/data/wallets/created-wallet", RecoveryPhrase:"damp begin reduce live agent fine treat merge host shoot vibrant sugar spider coffee hamster helmet town patient into subway day clap canal alarm"}, Key:struct { Public string "json:\"publicKey\"" }{Public:"4f87e7926893b9328eb10a75b5a88e5f13fefc5940ee203ffd0df7842b274407"}}
2022/10/28 11:33:46 Importing node vega wallet with: /usr/local/bin/vega [nodewallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt import --output json --chain vega --wallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/vega-wallet-pass.txt --wallet-path /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/data/wallets/created-wallet]
2022/10/28 11:33:46 node wallet create out: &vega.createWalletOutput{Wallet:struct { Name string "json:\"name\""; FilePath string "json:\"filePath\""; RecoveryPhrase string "json:\"recoveryPhrase\"" }{Name:"created-wallet", FilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/data/wallets/created-wallet", RecoveryPhrase:"private hawk pledge retreat future skill mother onion science barrel term lazy gain daughter spray segment digital axis can degree task stand garbage almost"}, Key:struct { Public string "json:\"publicKey\"" }{Public:"5dc85f389e614457cd5cda0e5161355fb3350ded749f62b8cc25c6a1c2b52409"}}
2022/10/28 11:33:46 Importing node vega wallet with: /usr/local/bin/vega [nodewallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt import --output json --chain vega --wallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/vega-wallet-pass.txt --wallet-path /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/data/wallets/created-wallet]
2022/10/28 11:33:46 2022-10-28T11:33:46.282+0100    INFO    root    commands/init.go:72 configuration generated successfully    {"path": "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/data/node3/config/data-node/config.toml"}

2022/10/28 11:33:46 2022-10-28T11:33:46.291+0100    INFO    root    commands/init.go:72 configuration generated successfully    {"path": "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/data/node2/config/data-node/config.toml"}

2022/10/28 11:33:46 node wallet import out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/config/node/wallets.encrypted", TendermintPubkey:"", WalletFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/data/node/wallets/vega/vega.1666953226315677000"}
2022/10/28 11:33:46 Collecting vega node id with: /usr/local/bin/vega [wallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0 info --output json --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/vega-wallet-pass.txt --wallet created-wallet]
2022/10/28 11:33:46 node wallet import out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/config/node/wallets.encrypted", TendermintPubkey:"", WalletFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/data/node/wallets/vega/vega.1666953226317942000"}
2022/10/28 11:33:46 Collecting vega node id with: /usr/local/bin/vega [wallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1 info --output json --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/vega-wallet-pass.txt --wallet created-wallet]
2022/10/28 11:33:46 node wallet info out: &vega.walletInfoOutput{Name:"created-wallet", ID:"37cc5bdd1536f8c3e767a5878f973052afa04c078cb7baec5efd819dfd6c4ba4"}
2022/10/28 11:33:46 Generating node "ethereum" wallet with: /usr/local/bin/vega [nodewallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt generate --output json --chain ethereum --wallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/ethereum-vega-wallet-pass.txt]
2022/10/28 11:33:46 node wallet info out: &vega.walletInfoOutput{Name:"created-wallet", ID:"8970217356fdfdbfadba781c3654cf05fe1a5a6b1cd8dd1b823a3fe6e76e69d3"}
2022/10/28 11:33:46 Generating node "ethereum" wallet with: /usr/local/bin/vega [nodewallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt generate --output json --chain ethereum --wallet-passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/ethereum-vega-wallet-pass.txt]
2022/10/28 11:33:49 ethereum wallet out: &vega.generateNodeWalletOutput{Mnemonic:"", RegistryFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/config/node/wallets.encrypted", WalletFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/data/node/wallets/ethereum/UTC--2022-10-28T10-33-46.446508000Z--d6a25b36b4845a7a8b7887542a6f10ae8441df1c"}
2022/10/28 11:33:49 ethereum wallet out: &vega.generateNodeWalletOutput{Mnemonic:"", RegistryFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/config/node/wallets.encrypted", WalletFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/data/node/wallets/ethereum/UTC--2022-10-28T10-33-46.446508000Z--b93c6baccc4e9abb0be3c415b53d5911bc3c6003"}
2022/10/28 11:33:50 Importing tenderming wallet: [nodewallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt import --output json --chain tendermint --tendermint-home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node1]
2022/10/28 11:33:50 Importing tenderming wallet: [nodewallet --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt import --output json --chain tendermint --tendermint-home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node0]
2022/10/28 11:33:50 tendermint wallet out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/config/node/wallets.encrypted", TendermintPubkey:"bDbFcYltI6icEDRe+4IkvELzivciVUqKP7NuxJmPEa0=", WalletFilePath:""}
2022/10/28 11:33:50 vega config initialized for node "validator" with id 0, paths: "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/config/node/config.toml"
2022/10/28 11:33:50 tendermint wallet out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/config/node/wallets.encrypted", TendermintPubkey:"6CeONViC5bUsmzc+wlWU8GIdC0alV1jAY4VzgiBeTv0=", WalletFilePath:""}
2022/10/28 11:33:50 vega config initialized for node "validator" with id 1, paths: "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/config/node/config.toml"
2022/10/28 11:33:50 Overwriting Tendermint config for nodeset testnet-nodeset-validators-0-validator
2022/10/28 11:33:50 Overwriting Vega config for nodeset testnet-nodeset-validators-0-validator
2022/10/28 11:33:50 Overwriting Tendermint config for nodeset testnet-nodeset-validators-1-validator
2022/10/28 11:33:50 Overwriting Vega config for nodeset testnet-nodeset-validators-1-validator
2022/10/28 11:33:50 Overwriting Tendermint config for nodeset testnet-nodeset-full-4-full
2022/10/28 11:33:50 Overwriting Vega config for nodeset testnet-nodeset-full-4-full
2022/10/28 11:33:50 Overwriting Data Node config for nodeset testnet-nodeset-full-4-full
2022/10/28 11:33:50 Overwriting Tendermint config for nodeset testnet-nodeset-full-3-full
2022/10/28 11:33:50 Overwriting Vega config for nodeset testnet-nodeset-full-3-full
2022/10/28 11:33:50 Overwriting Data Node config for nodeset testnet-nodeset-full-3-full
2022/10/28 11:33:50 Overwriting Tendermint config for nodeset testnet-nodeset-full-2-full
2022/10/28 11:33:50 Overwriting Vega config for nodeset testnet-nodeset-full-2-full
2022/10/28 11:33:50 Overwriting Data Node config for nodeset testnet-nodeset-full-2-full
2022/10/28 11:33:50 Updating genesis with: /usr/local/bin/vega [genesis --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt update --tm-home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node0 --dry-run]
2022/10/28 11:33:52 Updating genesis with: /usr/local/bin/vega [genesis --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1 --passphrase-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt update --tm-home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/tendermint/node1 --dry-run]
2022/10/28 11:33:55 Initiating wallet "wallet-1" with: [wallet init --output json --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/wallet]
2022/10/28 11:33:56 Importing network to wallet "wallet-1" with: [wallet network import --output json --home /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/wallet --from-file /Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/wallet/config.toml]
2022/10/28 11:33:56 generating network success
2022/10/28 11:33:56 starting network
2022/10/28 11:34:00 Update for job: "ganache-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:00 Update for job: "postgres-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:04 Update for job: "ganache-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:04 Update for job: "postgres-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:08 Update for job: "ganache-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:08 Update for job: "postgres-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:12 Update for job: "postgres-1", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:12 Update for job: "ganache-1", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:12 adding node set "testnet-nodeset-validators-0-validator" with default Nomad job definition
2022/10/28 11:34:12 adding node set "testnet-nodeset-validators-1-validator" with default Nomad job definition
2022/10/28 11:34:12 adding node set "testnet-nodeset-full-2-full" with default Nomad job definition
2022/10/28 11:34:12 adding node set "testnet-nodeset-full-3-full" with default Nomad job definition
2022/10/28 11:34:12 adding node set "testnet-nodeset-full-4-full" with default Nomad job definition
2022/10/28 11:34:16 Update for job: "testnet-nodeset-validators-1-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:16 Update for job: "testnet-faucet", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:16 Update for job: "testnet-vegawallet", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:16 Update for job: "testnet-nodeset-full-2-full", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:16 Update for job: "testnet-nodeset-validators-0-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:16 Update for job: "testnet-nodeset-full-3-full", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:16 Update for job: "testnet-nodeset-full-4-full", jobStatus: dead, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:20 Update for job: "testnet-nodeset-validators-0-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:20 Update for job: "testnet-nodeset-full-3-full", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:20 Update for job: "testnet-nodeset-full-4-full", jobStatus: dead, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:20 Update for job: "testnet-nodeset-full-2-full", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:20 Update for job: "testnet-vegawallet", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:20 Update for job: "testnet-nodeset-validators-1-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:20 Update for job: "testnet-faucet", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:24 Update for job: "testnet-nodeset-validators-0-validator", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:24 Update for job: "testnet-nodeset-full-3-full", jobStatus: dead, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:24 Update for job: "testnet-nodeset-validators-1-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:24 Update for job: "testnet-nodeset-full-4-full", jobStatus: dead, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:24 Update for job: "testnet-nodeset-full-2-full", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:24 Update for job: "testnet-vegawallet", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:24 Update for job: "testnet-faucet", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:28 Update for job: "testnet-nodeset-validators-1-validator", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully
2022/10/28 11:34:28 Update for job: "testnet-nodeset-full-4-full", jobStatus: dead, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:32 Update for job: "testnet-nodeset-full-4-full", jobStatus: dead, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:36 Update for job: "testnet-nodeset-full-4-full", jobStatus: dead, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:40 Update for job: "testnet-nodeset-full-4-full", jobStatus: dead, deploymentStatus: "running", another: Deployment is running
2022/10/28 11:34:44 Job "testnet-nodeset-full-4-full" has timed out output: Task: "data-node-4", State: "dead", Events: "Received: Task received by client -> Task Setup: Building Task Directory -> Started: Task started by client -> Terminated: Exit Code: 255 -> Not Restarting: Policy allows no restarts -> Killing: Sent interrupt. Waiting 5s before force killing"
2022/10/28 11:34:44 Job "testnet-nodeset-full-4-full" has timed out output: Task: "vega-full-4", State: "dead", Events: "Received: Task received by client -> Task Setup: Building Task Directory -> Started: Task started by client -> Sibling Task Failed: Task's sibling \"data-node-4\" failed -> Killing: Sent interrupt. Waiting 5s before force killing -> Terminated: Exit Code: 130, Signal: 2 -> Killed: Task successfully killed"
2022/10/28 11:34:44 Job "testnet-nodeset-full-4-full" has timed out output: Task: "capsule-logscolletor", State: "dead", Events: "Received: Task received by client -> Task Setup: Building Task Directory -> Started: Task started by client -> Sibling Task Failed: Task's sibling \"data-node-4\" failed -> Killing: Sent interrupt. Waiting 5s before force killing -> Terminated: Exit Code: 1 -> Killed: Task successfully killed"

Logs from failed testnet-nodeset-full-4-full job:
-------- data-node-4.stderr-2022-10-28T11:34:13+01:00.log:
failed to check if data node has scheme or is empty: failed to check if node has exisiting schema:unable to connect to database: failed to connect to `host=localhost user=vega database=vega`: dial error (dial tcp [::1]:5432: connect: connection refused)

-------- data-node-4.stdout-2022-10-28T11:34:13+01:00.log:
2022-10-28T11:34:13.561+0100    INFO    cfgwatcher  config/watcher.go:90    config watcher started successfully {"config": "/Users/muthupandianvadivelu/muthu/vega/repos/networkdata/testnet/data/node4/config/data-node/config.toml"}
2022-10-28T11:34:13.562+0100    INFO    root    start/node_pre.go:55    vega is starting with pprof profile, this is not a recommended setting for production
2022-10-28T11:34:13.564+0100    INFO    root    start/node_pre.go:65    Starting Vega Datanode  {"version": "v0.60.0", "version-hash": "ec3c20d71b384a58f6f5a17a467f2636c0d859f7-modified"}
2022-10-28T11:34:13.564+0100    INFO    root    start/node_pre.go:85    Initializing Decentralized History
2022-10-28T11:34:13.709+0100    INFO    deHistory.service.store dehistory/service.go:44 starting decentralized history store with ipfs Peer Id:12D3KooWHUKGBMwuwSNMiJgmy4xMkDZAgPj9ReScoQc5a2EC13Ra
2022-10-28T11:34:13.721+0100    INFO    deHistory.service.store dehistory/service.go:44 Using chain id testnet as the swarm key
2022-10-28T11:34:14.737+0100    INFO    deHistory.service.store dehistory/service.go:44 Swarm is limited to private network of peers with the fingerprint b1fec4aa02bad74b55fc5be0a45bd8b8

-------- vega-full-4.stdout-2022-10-28T11:34:13+01:00.log:
2022-10-28T11:34:13.578+0100    INFO    root    node/node.go:95 starting vega   {"version": "v0.60.0", "commit-hash": "ec3c20d71b384a58f6f5a17a467f2636c0d859f7-modified"}
2022-10-28T11:34:13.721+0100    ERROR   broker  broker/socket_client.go:174 failed to connect, retrying {"error": "dial tcp 127.0.0.1:3045: connect: connection refused", "peer": "tcp://127.0.0.1:3045"}

2022/10/28 11:34:44 Trying to stop jobs: ganache-1, postgres-1, testnet-faucet, testnet-nodeset-full-2-full, testnet-nodeset-full-3-full, testnet-nodeset-full-4-full, testnet-nodeset-validators-0-validator, testnet-nodeset-validators-1-validator, testnet-vegawallet
2022/10/28 11:34:44 Job "testnet-vegawallet" stopped
2022/10/28 11:34:44 Job "ganache-1" stopped
2022/10/28 11:34:44 Job "testnet-faucet" stopped
2022/10/28 11:34:44 Job "testnet-nodeset-full-2-full" stopped
2022/10/28 11:34:44 Job "postgres-1" stopped
2022/10/28 11:34:44 Job "testnet-nodeset-full-4-full" stopped
2022/10/28 11:34:44 Job "testnet-nodeset-validators-1-validator" stopped
2022/10/28 11:34:44 Job "testnet-nodeset-validators-0-validator" stopped
2022/10/28 11:34:44 Job "testnet-nodeset-full-3-full" stopped
2022/10/28 11:34:44 Jobs have been stopped.
Error: failed to start network: failed to start network: failed to start vega network: failed to run node sets: failed to wait for node sets: failed to run testnet-nodeset-full-4-full job: starting deadline has been exceeded
make: *** [vegacapsule-start-network] Error 1
Muthus-MacBook-Pro:scripts muthupandianvadivelu$
MuthuVega commented 1 year ago

Tried this after all the changes for vegacapsule , system-tests and vega have been merged - but still unable to successfully start a network with multiple datanodes.

Jenkins Run - https://jenkins.ops.vega.xyz/blue/organizations/jenkins/common%2Fsystem-tests-wrapper/detail/system-tests-wrapper/20845/pipeline/353

System Test branch - feat_decentralised_datanode_1