Improve error message: verification of blockchain snark failed but it was our fault

firebeefaizul commented 1 year ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
[X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

Having error while I am hitting mina daemon using this command(mina daemon --peer-list-url https://storage.googleapis.com/mina-seed-lists/mainnet_seeds.txt).

Thanks,

Steps to Reproduce

mina daemon --peer-list-url https://storage.googleapis.com/mina-seed-lists/mainnet_seeds.txt

Expected Result

we are expect block verify.

Actual Result

Error Message 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault 2023-04-24 05:41:42 UTC [Error] verifier terminated unexpectedly

How frequently do you see this issue?

Rarely

What is the impact of this issue on your ability to run a node?

High

Status

Error Message 
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Warn] verification of blockchain snark failed but it was our fault
2023-04-24 05:41:42 UTC [Error] verifier terminated unexpectedly

Additional information

No response

vanarasiantonyswamy commented 1 year ago

can you kindly shed some light on the repro -

which image did you use - Debian Packages or docker
did you create ~/.mina-config or use existing one
is this first time you started the daemon or was it already running
was it against : https://github.com/MinaProtocol/mina/discussions/12946

This will help reproduce at our end and help further investigation

vanarasiantonyswamy commented 1 year ago

Hi Any updates on this issue with clear repro will help on our end to triage.

firebeefaizul commented 1 year ago

sorry for delay... Hai @vanarasiantonyswamy

Debian package i used. existing config yes, this is first time.

Now, also facing same issue

vanarasiantonyswamy commented 1 year ago

can you kindly attach the ~/.mina-config folder zip to this issue.

can you stop the daemon delete the ~/.mina-config start the daemon again and allow it to resync please keep posted here if still you hit the same issue.

firebeefaizul commented 1 year ago

we don't have any config, get the code from https://github.com/MinaProtocol/mina. kindly give a suggesion

vanarasiantonyswamy commented 1 year ago

i am not able to repro this issue. When mina daemon runs, it will create a ~/.mina-config folder and keep the logs in this folder. if you can attach it, it will ease us to investigate further.

vanarasiantonyswamy commented 1 year ago

I could repro the above using mainnet stable 1.3.1.2.

vanarasiantonyswamy commented 1 year ago

Detailed Repro steps: Install on ubuntu VM - mainnet stable sudo rm /etc/apt/sources.list.d/mina*.list echo "deb [trusted=yes] http://packages.o1test.net $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/mina.list sudo apt-get update sudo apt-get install -y mina-mainnet=1.3.1.2-25388a0

generate key cd ~ mkdir ~/keys chmod 700 ~/keys mina advanced generate-keypair --privkey-path /home/user/keys/my-wallet chmod 600 ~/keys/my-wallet

mkdir ~/.mina-config

create ~/.mina-env file MINA_PRIVKEY_PASS="your password used for creating the key" LOG_LEVEL=Info FILE_LOG_LEVEL=Debug EXTRA_FLAGS=" --block-producer-key /home/user/keys/my-wallet"

verify you see the same mina version as in the release doc by running mina version. chain id matches. mina daemon --peer-list-url https://storage.googleapis.com/mina-seed-lists/mainnet_seeds.txt

Check the logs shared in slack. [folder .mina-config] Daemon gets stuck in catchup and then restarts back to bootstrap on the terminal you will see [Warn] verification of blockchain snark failed but it was our fault

Document referred : https://github.com/MinaProtocol/mina/releases/tag/1.3.1.2 https://docs.minaprotocol.com/node-operators/generating-a-keypairhttps://docs.minaprotocol.com/node-operators/connecting-to-the-network

Expected: The Daemon should move to Synced status Actual: the daemon restarts after catchup. sometimes its stuck in bootstrap.

I did the above repro for Mainnet Beta Release 1.4.0beta2 i see that the daemon also gets stuck in Bootstrap

amc-ie commented 1 year ago

@vanarasiantonyswamy are you seeing the same error message when it is stuck in bootstrap?

vanarasiantonyswamy commented 1 year ago

yes when stuck in catchup i saw this error on terminal

MinaProtocol / mina