vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
37 stars 22 forks source link

data-node crash right after network start #6654

Closed claudiumilea closed 1 year ago

claudiumilea commented 1 year ago

Problem encountered

data-node crashes immediately after network was started - it complains it does not find a file /Users/claudiumilea/repo/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/testnet-12-currentstatesnapshot/current_margin_levels but my directory /Users/claudiumilea/repo/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/ is empty

Observed behaviour

A clear and concise description of how the system is behaving.

Expected behaviour

A clear and concise description of what you expected to happen.

System response

Describe what the system response was, include the output from the command, automation, or else.

Steps to reproduce

Manual

Steps to reproduce the behaviour manually:

  1. cleanup network
  2. cleanup nomad
  3. start nomad
  4. start network
  5. check localhost:4646 to see the dead node

Automation

Link to automation and explanation on how to run it to reproduce the problem/bug

Evidence

Logs

this is the data-node-2 stderr: panic: failed to snapshot data:failed to copy table data:failed to execute copy copy (select * from current_margin_levels order by account_id) to '/Users/claudiumilea/repo/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/testnet-12-currentstatesnapshot/current_margin_levels': ERROR: could not open file "/Users/claudiumilea/repo/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/testnet-12-currentstatesnapshot/current_margin_levels" for writing: No such file or directory (SQLSTATE 58P01)

goroutine 1302 [running]: code.vegaprotocol.io/vega/datanode/dehistory/snapshot.(Service).CreateSnapshot.func5() /Users/claudiumilea/repo/vega/datanode/dehistory/snapshot/service_create_snapshot.go:87 +0x238 created by code.vegaprotocol.io/vega/datanode/dehistory/snapshot.(Service).CreateSnapshot /Users/claudiumilea/repo/vega/datanode/dehistory/snapshot/service_create_snapshot.go:81 +0x928

Additional context

(venv) claudiumilea@Claudius-Mac-Studio scripts % make cleanup
Check local environment: Start WORK_DIR=/Users/claudiumilea/repo SYSTEM_TEST_DIR=/Users/claudiumilea/repo/system-tests VEGA_DIR=/Users/claudiumilea/repo/vega DEVOPS_DIR=/Users/claudiumilea/repo/devops-infra VEGATOOLS_DIR=/Users/claudiumilea/repo/vegatools DOCKER_GOCACHE=/Users/claudiumilea/repo/system-tests/go/cache GOPATH=/Users/claudiumilea/go GOPRIVATE=code.vegaprotocol.io GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=arm64 GOPROXY= SYSTEM_TESTS_ENV=127.0.0.1 VALIDATOR_NODE_COUNT=2 NON_VALIDATOR_NODE_COUNT=1 SYSTEM_TESTS_PORTBASE=1000 SYSTEM_TESTS_DOCKER_IMAGE_TAG=local VEGA_CORE_IMAGE_TAG=local DATA_NODE_IMAGE_TAG=local TEST_FUNCTION= TEST_DIRECTORY=tests TEST_MARK= TEST_EXTRA_PYTEST_ARGS= UID=501 GID=20 VEGA_LOG_LEVEL=Info VEGACAPSULE_CONFIG_FILENAME=/Users/claudiumilea/repo/system-tests/vegacapsule/capsule_config.hcl TESTS_DIR=/Users/claudiumilea/repo/networkdata NETWORK_HOME_PATH=/Users/claudiumilea/repo/networkdata/testnet smartcontractsmount=-v /Users/claudiumilea/repo/networkdata/testnet/smartcontracts:/workspace/smartcontracts -e SMART_CONTRACTS_ADDRESSES=/workspace/smartcontracts/addresses.json MAINNET_TEST_CASE= === Check Successful === Check local environment: End Additional check steps for vegacapsule: Start VEGACAPSULE_DIR=/Users/claudiumilea/repo/vegacapsule VEGACAPSULE_NET_ADDR= GOBIN=/Users/claudiumilea/go/bin === Check successful === Additional check steps for vegacapsule: End Cleanup vegacapsule: Start No visor orphan is running No data-node orphan is running No tendermint orphan is running No vegawallet orphan is running No faucet orphan is running Cleanup vegacapsule: End (venv) claudiumilea@Claudius-Mac-Studio scripts % make start-network Check local environment: Start WORK_DIR=/Users/claudiumilea/repo SYSTEM_TEST_DIR=/Users/claudiumilea/repo/system-tests VEGA_DIR=/Users/claudiumilea/repo/vega DEVOPS_DIR=/Users/claudiumilea/repo/devops-infra VEGATOOLS_DIR=/Users/claudiumilea/repo/vegatools DOCKER_GOCACHE=/Users/claudiumilea/repo/system-tests/go/cache GOPATH=/Users/claudiumilea/go GOPRIVATE=code.vegaprotocol.io GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=arm64 GOPROXY= SYSTEM_TESTS_ENV=127.0.0.1 VALIDATOR_NODE_COUNT=2 NON_VALIDATOR_NODE_COUNT=1 SYSTEM_TESTS_PORTBASE=1000 SYSTEM_TESTS_DOCKER_IMAGE_TAG=local VEGA_CORE_IMAGE_TAG=local DATA_NODE_IMAGE_TAG=local TEST_FUNCTION= TEST_DIRECTORY=tests TEST_MARK= TEST_EXTRA_PYTEST_ARGS= UID=501 GID=20 VEGA_LOG_LEVEL=Info VEGACAPSULE_CONFIG_FILENAME=/Users/claudiumilea/repo/system-tests/vegacapsule/capsule_config.hcl TESTS_DIR=/Users/claudiumilea/repo/networkdata NETWORK_HOME_PATH=/Users/claudiumilea/repo/networkdata/testnet smartcontractsmount=-v /Users/claudiumilea/repo/networkdata/testnet/smartcontracts:/workspace/smartcontracts -e SMART_CONTRACTS_ADDRESSES=/workspace/smartcontracts/addresses.json MAINNET_TEST_CASE= === Check Successful === Check local environment: End Additional check steps for vegacapsule: Start VEGACAPSULE_DIR=/Users/claudiumilea/repo/vegacapsule VEGACAPSULE_NET_ADDR= GOBIN=/Users/claudiumilea/go/bin === Check successful === Additional check steps for vegacapsule: End 2022/10/31 19:44:06 generating network 2022/10/31 19:44:06 Initiating faucet with: /Users/claudiumilea/repo/networkdata/vega [faucet init --home /Users/claudiumilea/repo/networkdata/testnet/faucet --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/faucet/faucet-wallet-pass.txt --output json] 2022/10/31 19:44:07 Initiating Tendermint node "validator" with: /Users/claudiumilea/repo/networkdata/vega [tm init validator --home /Users/claudiumilea/repo/networkdata/testnet/tendermint/node0] 2022/10/31 19:44:07 Initiating Tendermint node "validator" with: /Users/claudiumilea/repo/networkdata/vega [tm init validator --home /Users/claudiumilea/repo/networkdata/testnet/tendermint/node1] 2022/10/31 19:44:07 Initiating Tendermint node "full" with: /Users/claudiumilea/repo/networkdata/vega [tm init full --home /Users/claudiumilea/repo/networkdata/testnet/tendermint/node2] 2022/10/31 19:44:07 I[2022-10-31|19:44:07.873] Generated private validator module=main keyFile=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node1/config/priv_validator_key.json stateFile=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node1/data/priv_validator_state.json I[2022-10-31|19:44:07.873] Generated node key module=main path=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node1/config/node_key.json I[2022-10-31|19:44:07.873] Generated genesis file module=main path=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node1/config/genesis.json

2022/10/31 19:44:07 I[2022-10-31|19:44:07.873] Generated private validator module=main keyFile=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node0/config/priv_validator_key.json stateFile=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node0/data/priv_validator_state.json I[2022-10-31|19:44:07.873] Generated node key module=main path=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node0/config/node_key.json I[2022-10-31|19:44:07.874] Generated genesis file module=main path=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node0/config/genesis.json

2022/10/31 19:44:07 I[2022-10-31|19:44:07.873] Generated private validator module=main keyFile=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node2/config/priv_validator_key.json stateFile=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node2/data/priv_validator_state.json I[2022-10-31|19:44:07.873] Generated node key module=main path=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node2/config/node_key.json I[2022-10-31|19:44:07.874] Generated genesis file module=main path=/Users/claudiumilea/repo/networkdata/testnet/tendermint/node2/config/genesis.json

2022/10/31 19:44:07 Initiating node "full" with: /Users/claudiumilea/repo/networkdata/vega [init --home /Users/claudiumilea/repo/networkdata/testnet/vega/node2 --nodewallet-passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node2/node-vega-wallet-pass.txt --no-tendermint --output json full] 2022/10/31 19:44:07 Initiating node "validator" with: /Users/claudiumilea/repo/networkdata/vega [init --home /Users/claudiumilea/repo/networkdata/testnet/vega/node1 --nodewallet-passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt --no-tendermint --output json validator] 2022/10/31 19:44:07 Initiating node "validator" with: /Users/claudiumilea/repo/networkdata/vega [init --home /Users/claudiumilea/repo/networkdata/testnet/vega/node0 --nodewallet-passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt --no-tendermint --output json validator] 2022/10/31 19:44:07 vega config initialized for node "full" with id 2, paths: "/Users/claudiumilea/repo/networkdata/testnet/vega/node2/config/node/config.toml" 2022/10/31 19:44:07 Initiating data node with: /Users/claudiumilea/repo/networkdata/vega [datanode init -f --home /Users/claudiumilea/repo/networkdata/testnet/data/node2 testnet] 2022/10/31 19:44:07 Creating vega wallet with: /Users/claudiumilea/repo/networkdata/vega [wallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node1 create --output json --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/vega-wallet-pass.txt --wallet created-wallet] 2022/10/31 19:44:07 Creating vega wallet with: /Users/claudiumilea/repo/networkdata/vega [wallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node0 create --output json --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/vega-wallet-pass.txt --wallet created-wallet] 2022/10/31 19:44:08 2022-10-31T19:44:08.034+0200 INFO root commands/init.go:72 configuration generated successfully {"path": "/Users/claudiumilea/repo/networkdata/testnet/data/node2/config/data-node/config.toml"}

2022/10/31 19:44:08 node wallet create out: &vega.createWalletOutput{Wallet:struct { Name string "json:\"name\""; FilePath string "json:\"filePath\""; RecoveryPhrase string "json:\"recoveryPhrase\"" }{Name:"created-wallet", FilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node0/data/wallets/created-wallet", RecoveryPhrase:"gaze awesome endorse size voyage empty isolate echo lawn snack add announce purchase convince boss bike message cherry mule swim uncover once series version"}, Key:struct { Public string "json:\"publicKey\"" }{Public:"175a496a253bbbe981f42496dec432a52f0520503ae245e07bc8801af5e3fbee"}} 2022/10/31 19:44:08 node wallet create out: &vega.createWalletOutput{Wallet:struct { Name string "json:\"name\""; FilePath string "json:\"filePath\""; RecoveryPhrase string "json:\"recoveryPhrase\"" }{Name:"created-wallet", FilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node1/data/wallets/created-wallet", RecoveryPhrase:"wife guilt leaf twenty riot dune viable original round half word obvious chicken hire rug code buddy usual kidney face short zebra true cook"}, Key:struct { Public string "json:\"publicKey\"" }{Public:"d007cdcb2f0f7f7ce76fa25deb143ce1b6c2ec2750a8910c275bd7fb9e8f293d"}} 2022/10/31 19:44:08 Importing node vega wallet with: /Users/claudiumilea/repo/networkdata/vega [nodewallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node1 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt import --output json --chain vega --wallet-passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/vega-wallet-pass.txt --wallet-path /Users/claudiumilea/repo/networkdata/testnet/vega/node1/data/wallets/created-wallet] 2022/10/31 19:44:08 Importing node vega wallet with: /Users/claudiumilea/repo/networkdata/vega [nodewallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node0 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt import --output json --chain vega --wallet-passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/vega-wallet-pass.txt --wallet-path /Users/claudiumilea/repo/networkdata/testnet/vega/node0/data/wallets/created-wallet] 2022/10/31 19:44:08 node wallet import out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node0/config/node/wallets.encrypted", TendermintPubkey:"", WalletFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node0/data/node/wallets/vega/vega.1667238248106701000"} 2022/10/31 19:44:08 Collecting vega node id with: /Users/claudiumilea/repo/networkdata/vega [wallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node0 info --output json --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/vega-wallet-pass.txt --wallet created-wallet] 2022/10/31 19:44:08 node wallet import out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node1/config/node/wallets.encrypted", TendermintPubkey:"", WalletFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node1/data/node/wallets/vega/vega.1667238248107189000"} 2022/10/31 19:44:08 Collecting vega node id with: /Users/claudiumilea/repo/networkdata/vega [wallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node1 info --output json --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/vega-wallet-pass.txt --wallet created-wallet] 2022/10/31 19:44:08 node wallet info out: &vega.walletInfoOutput{Name:"created-wallet", ID:"2f68e6c6469e9b0fd9b643594c3dfaae4493385619f366b482cb99f49da90206"} 2022/10/31 19:44:08 node wallet info out: &vega.walletInfoOutput{Name:"created-wallet", ID:"2f118ae246ee00241bcc1d7ab986161e1e37d937b1d3486cca80aa5b9345fa0a"} 2022/10/31 19:44:08 Generating node "ethereum" wallet with: /Users/claudiumilea/repo/networkdata/vega [nodewallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node1 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt generate --output json --chain ethereum --wallet-passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/ethereum-vega-wallet-pass.txt] 2022/10/31 19:44:08 Generating node "ethereum" wallet with: /Users/claudiumilea/repo/networkdata/vega [nodewallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node0 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt generate --output json --chain ethereum --wallet-passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/ethereum-vega-wallet-pass.txt] 2022/10/31 19:44:11 ethereum wallet out: &vega.generateNodeWalletOutput{Mnemonic:"", RegistryFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node0/config/node/wallets.encrypted", WalletFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node0/data/node/wallets/ethereum/UTC--2022-10-31T17-44-08.252804000Z--1bf661d7ca6c5c8acf98d1dc17e545d864810337"} 2022/10/31 19:44:11 ethereum wallet out: &vega.generateNodeWalletOutput{Mnemonic:"", RegistryFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node1/config/node/wallets.encrypted", WalletFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node1/data/node/wallets/ethereum/UTC--2022-10-31T17-44-08.253237000Z--c8a34b59e7d750cce345e72f2c480bc5b427c60d"} 2022/10/31 19:44:11 Importing tenderming wallet: [nodewallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node0 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt import --output json --chain tendermint --tendermint-home /Users/claudiumilea/repo/networkdata/testnet/tendermint/node0] 2022/10/31 19:44:11 Importing tenderming wallet: [nodewallet --home /Users/claudiumilea/repo/networkdata/testnet/vega/node1 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt import --output json --chain tendermint --tendermint-home /Users/claudiumilea/repo/networkdata/testnet/tendermint/node1] 2022/10/31 19:44:11 tendermint wallet out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node0/config/node/wallets.encrypted", TendermintPubkey:"1IeKVChqYGiace0wiGOeK8Wh3epLmQTc1T5VaDg5iKU=", WalletFilePath:""} 2022/10/31 19:44:11 vega config initialized for node "validator" with id 0, paths: "/Users/claudiumilea/repo/networkdata/testnet/vega/node0/config/node/config.toml" 2022/10/31 19:44:11 tendermint wallet out: &vega.importNodeWalletOutput{RegistryFilePath:"/Users/claudiumilea/repo/networkdata/testnet/vega/node1/config/node/wallets.encrypted", TendermintPubkey:"LaXhORKKk9suLWMc8wc/PkEWhNeoNGr6B1z+xA5OVvs=", WalletFilePath:""} 2022/10/31 19:44:11 vega config initialized for node "validator" with id 1, paths: "/Users/claudiumilea/repo/networkdata/testnet/vega/node1/config/node/config.toml" 2022/10/31 19:44:11 Overwriting Tendermint config for nodeset testnet-nodeset-validators-0-validator 2022/10/31 19:44:11 Overwriting Vega config for nodeset testnet-nodeset-validators-0-validator 2022/10/31 19:44:11 Overwriting Tendermint config for nodeset testnet-nodeset-validators-1-validator 2022/10/31 19:44:11 Overwriting Vega config for nodeset testnet-nodeset-validators-1-validator 2022/10/31 19:44:11 Overwriting Tendermint config for nodeset testnet-nodeset-full-2-full 2022/10/31 19:44:11 Overwriting Vega config for nodeset testnet-nodeset-full-2-full 2022/10/31 19:44:11 Overwriting Data Node config for nodeset testnet-nodeset-full-2-full 2022/10/31 19:44:11 Updating genesis with: /Users/claudiumilea/repo/networkdata/vega [genesis --home /Users/claudiumilea/repo/networkdata/testnet/vega/node0 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node0/node-vega-wallet-pass.txt update --tm-home /Users/claudiumilea/repo/networkdata/testnet/tendermint/node0 --dry-run] 2022/10/31 19:44:13 Updating genesis with: /Users/claudiumilea/repo/networkdata/vega [genesis --home /Users/claudiumilea/repo/networkdata/testnet/vega/node1 --passphrase-file /Users/claudiumilea/repo/networkdata/testnet/vega/node1/node-vega-wallet-pass.txt update --tm-home /Users/claudiumilea/repo/networkdata/testnet/tendermint/node1 --dry-run] 2022/10/31 19:44:15 Initiating wallet "wallet-1" with: [wallet init --output json --home /Users/claudiumilea/repo/networkdata/testnet/wallet] 2022/10/31 19:44:16 Importing network to wallet "wallet-1" with: [wallet network import --output json --home /Users/claudiumilea/repo/networkdata/testnet/wallet --from-file /Users/claudiumilea/repo/networkdata/testnet/wallet/config.toml] 2022/10/31 19:44:16 generating network success 2022/10/31 19:44:16 starting network 2022/10/31 19:44:20 Update for job: "ganache-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:20 Update for job: "postgres-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:24 Update for job: "ganache-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:24 Update for job: "postgres-1", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:28 Update for job: "ganache-1", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully 2022/10/31 19:44:28 Update for job: "postgres-1", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully 2022/10/31 19:44:28 adding node set "testnet-nodeset-validators-0-validator" with default Nomad job definition 2022/10/31 19:44:28 adding node set "testnet-nodeset-validators-1-validator" with default Nomad job definition 2022/10/31 19:44:28 adding node set "testnet-nodeset-full-2-full" with default Nomad job definition 2022/10/31 19:44:32 Update for job: "testnet-nodeset-validators-0-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:32 Update for job: "testnet-nodeset-full-2-full", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:32 Update for job: "testnet-nodeset-validators-1-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:32 Update for job: "testnet-faucet", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:32 Update for job: "testnet-vegawallet", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:36 Update for job: "testnet-nodeset-validators-0-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:36 Update for job: "testnet-nodeset-full-2-full", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:36 Update for job: "testnet-nodeset-validators-1-validator", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:36 Update for job: "testnet-faucet", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:36 Update for job: "testnet-vegawallet", jobStatus: running, deploymentStatus: "running", another: Deployment is running 2022/10/31 19:44:40 Update for job: "testnet-nodeset-validators-0-validator", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully 2022/10/31 19:44:40 Update for job: "testnet-nodeset-full-2-full", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully 2022/10/31 19:44:40 Update for job: "testnet-nodeset-validators-1-validator", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully 2022/10/31 19:44:40 Update for job: "testnet-faucet", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully 2022/10/31 19:44:40 Update for job: "testnet-vegawallet", jobStatus: running, deploymentStatus: "successful", another: Deployment completed successfully 2022/10/31 19:44:40 Network successfully started. 2022/10/31 19:44:40 printing exposed network addresses Job "ganache-1"

Definition of Done

ℹ️ Not every issue will need every item checked, however, every item on this list should be properly considered and actioned to meet the DoD.

Before Merging

After Merging

gordsport commented 1 year ago

Hi @claudiumilea,

Is this using the latest core version? The following issue has a similar error and has been fixed

RomanLeca commented 1 year ago

@gordsport , I can perfectly replicate this behaviour on my machine (latest develop core branch) and it is blocking for local setup.

stderr `panic: failed to snapshot data:failed to copy table data:failed to execute copy copy (select * from current_liquidity_provisions order by id) to '/Users/roman/Documents/work/vega/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/testnet-12-currentstatesnapshot/current_liquidity_provisions': ERROR: could not open file "/Users/roman/Documents/work/vega/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/testnet-12-currentstatesnapshot/current_liquidity_provisions" for writing: No such file or directory (SQLSTATE 58P01)

goroutine 1262 [running]: code.vegaprotocol.io/vega/datanode/dehistory/snapshot.(Service).CreateSnapshot.func5() /Users/roman/Documents/work/vega/vega/datanode/dehistory/snapshot/service_create_snapshot.go:87 +0x374 created by code.vegaprotocol.io/vega/datanode/dehistory/snapshot.(Service).CreateSnapshot /Users/roman/Documents/work/vega/vega/datanode/dehistory/snapshot/service_create_snapshot.go:81 +0xaed`

stdout 2022-11-01T12:16:32.730+0200 INFO cfgwatcher config/watcher.go:90 config watcher started successfully {"config": "/Users/roman/Documents/work/vega/networkdata/testnet/data/node2/config/data-node/config.toml"} 2022-11-01T12:16:32.730+0200 INFO root start/node_pre.go:55 vega is starting with pprof profile, this is not a recommended setting for production 2022-11-01T12:16:32.730+0200 INFO root start/node_pre.go:65 Starting Vega Datanode {"version": "v0.62.0+dev.46d1c0c2.modified", "version-hash": "46d1c0c29f82b9a2ca13a49bcaff9b6c06dab2b8-modified"} 2022-11-01T12:16:32.730+0200 INFO root start/node_pre.go:86 Initializing Decentralized History 2022-11-01T12:16:32.868+0200 INFO deHistory.service.store dehistory/service.go:44 starting decentralized history store with ipfs Peer Id:12D3KooWMcgDij5ZvWGWBMEv9cNAVB3QDisAvYHN2b6vrmu9G3WB 2022-11-01T12:16:32.869+0200 INFO deHistory.service.store dehistory/service.go:44 Using chain id testnet as the swarm key 2022-11-01T12:16:43.083+0200 INFO deHistory.service.store dehistory/service.go:44 Swarm is limited to private network of peers with the fingerprint b1fec4aa02bad74b55fc5be0a45bd8b8 2022-11-01T12:16:43.123+0200 INFO root start/node_pre.go:107 Attempting to connect to SQL stores... 2022-11-01T12:16:46.209+0200 INFO db migration v3@v3.6.1/migration.go:38 OK 0001_initial.sql 2022-11-01T12:16:46.213+0200 INFO db migration v3@v3.6.1/up.go:210 goose: no migrations to run. current version: 1 2022-11-01T12:16:46.214+0200 INFO root start/node_pre.go:128 Applying Data Retention Policies 2022-11-01T12:16:46.758+0200 INFO root start/node_pre.go:135 Enabling SQL stores 2022-11-01T12:16:46.842+0200 DEBUG connection_source sqlstore/market_data.go:109 Retrieving markets data from Postgres 2022-11-01T12:16:46.870+0200 INFO buffered-event-source start/node_pre.go:207 Starting buffered event source with a max buffered event count of %!f(int=100,000,000), and events per buffer file size 10000000 2022-11-01T12:16:46.872+0200 INFO root start/node.go:143 Vega data node startup complete 2022-11-01T12:16:46.873+0200 INFO root broker/socket_server.go:86 Starting broker socket server {"addr": "0.0.0.0", "port": 3025} 2022-11-01T12:16:46.873+0200 INFO api.grpc api/server.go:333 Starting gRPC based API {"addr": "0.0.0.0", "port": "3027"} 2022-11-01T12:16:46.877+0200 INFO gateway.gql graphql/server.go:147 Starting GraphQL based API {"addr": "0.0.0.0", "port": 3028} 2022-11-01T12:16:46.877+0200 WARN gateway.gql graphql/server.go:184 graphql playground enabled, this is not a recommended setting for production 2022-11-01T12:16:46.878+0200 WARN gateway.gql graphql/server.go:244 GraphQL server is not configured to use HTTPS, which is required for subscriptions to work. Please see README.md for help configuring 2022-11-01T12:16:46.877+0200 INFO gateway.restproxy rest/server.go:79 Starting REST<>GRPC based API {"addr": "0.0.0.0", "port": 3029} 2022-11-01T12:16:46.936+0200 INFO api.grpc api/server.go:516 Starting gRPC Web UI {"addr": "0.0.0.0", "port": 3026} 2022-11-01T12:16:47.695+0200 INFO root broker/socket_server.go:90 New broker connection event {"eventType": "Attaching", "id": 672043170, "address": "tcp://[::]:3025"} 2022-11-01T12:16:47.695+0200 INFO root broker/socket_server.go:90 New broker connection event {"eventType": "Attached", "id": 672043170, "address": "tcp://[::]:3025"} 2022-11-01T12:16:48.757+0200 INFO root broker/sqlstore_broker.go:175 first unprocessed block received, starting block processing 2022-11-01T12:16:48.884+0200 WARN root broker/sqlstore_broker.go:125 slow time update detected, time between checks 2s, block height: 1, total block processing time: 126.72175ms 2022-11-01T12:16:52.714+0200 INFO deHistory.snapshot snapshot/service.go:47 creating snapshot for {History Snapshot for Chain ID:testnet Height From:1 Height To:12} 2022-11-01T12:16:52.848+0200 INFO deHistory.snapshot snapshot/service_create_snapshot.go:83 copying all table data.... 2022-11-01T12:16:52.887+0200 ERROR deHistory.snapshot snapshot/service_create_snapshot.go:85 failed to snapshot data {"error": "failed to copy table data:failed to execute copy copy (select * from current_liquidity_provisions order by id) to '/Users/roman/Documents/work/vega/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/testnet-12-currentstatesnapshot/current_liquidity_provisions': ERROR: could not open file \"/Users/roman/Documents/work/vega/networkdata/testnet/data/node2/state/data-node/dehistory/snapshotscopyto/testnet-12-currentstatesnapshot/current_liquidity_provisions\" for writing: No such file or directory (SQLSTATE 58P01)"}

claudiumilea commented 1 year ago

fixed