search

MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.97k stars 523 forks source link

Mina Daemon crashed #14542

Open Atikocak opened 7 months ago

Atikocak commented 7 months ago

Preliminary Checks

Description

Hello,

I'm actually trying to run my block producer node on "Berkeley Testnet Release 2.0.0rampup5". The latest release given here => https://github.com/MinaProtocol/mina/releases/tag/2.0.0rampup5

And here is the latest lines of crash report.

Nov 11 14:25:54 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:54.414332Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 559, characters 8-19"},"message":"Mina daemon is booting up; built with commit $commit on branch $branch","metadata":{"branch":"rampup","commit":"55b78189c46e1811b8bdb78864cfa95409aeb96a","commit_date":"2023-10-11T18:26:29+02:00","marlin_commit":"fd7c19d4c5dbb6826d1d5ce6bc24796a44087f70"}}
Nov 11 14:25:54 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:54.414913Z","level":"Info","source":{"module":"Mina_lib__Conf_dir","location":"File \"src/lib/mina_lib/conf_dir.ml\", line 70, characters 18-29"},"message":"Removing lockfile for terminated process","metadata":{"lockfile":"/root/.mina-config/.mina-lock","pid":1854}}
Nov 11 14:25:54 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:54.415056Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 567, characters 10-21"},"message":"Daemon will expire at $exp","metadata":{"exp":"2024-12-10 14:00:00-07:00"}}
Nov 11 14:25:54 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:54.415094Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 580, characters 8-19"},"message":"Booting may take several seconds, please wait","metadata":{}}
Nov 11 14:25:54 m23636.contaboserver.net mina[1975]: Using libp2p keypair private-key password from environment variable MINA_LIBP2P_PASS
Nov 11 14:25:54 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:54.634743Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 700, characters 12-23"},"message":"Reading configuration files $config_files","metadata":{"config_files":["/var/lib/coda/config_55b78189.json","/root/.mina-config/daemon.json"]}}
Nov 11 14:25:55 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:55.154128Z","level":"Warn","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 721, characters 24-35"},"message":"Could not read configuration from $config_file","metadata":{"config_file":"/root/.mina-config/daemon.json","error":{"commit_id":"55b78189c46e1811b8bdb78864cfa95409aeb96a","exn_name":"Async_kernel__Monitor.Error_","exn":["monitor.ml.Error",["Yojson.Json_error(\"Blank input data\")"],["Raised at Yojson.json_error in file \"common.ml\", line 5, characters 19-39","Called from Async_kernel__Deferred1.M.map.(fun) in file \"src/deferred1.ml\", line 17, characters 40-45","Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5","Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47","Caught by monitor try_with_or_error at file \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 815, characters 34-34"]]}}}
Nov 11 14:25:55 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:55.353093Z","level":"Info","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 840, characters 2-13"},"message":"Initializing with runtime configuration. Ledger name: $name","metadata":{"accounts_omitted":200000,"config":{"genesis":{"genesis_state_timestamp":"2023-10-17T16:01:01Z"},"ledger":{"name":"berkeley-ITN"}},"name":"berkeley-ITN"}}
Nov 11 14:25:55 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:55.353151Z","level":"Info","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 868, characters 8-19"},"message":"Using the compiled constraint constants","metadata":{}}
Nov 11 14:25:56 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:25:56.307493Z","level":"Info","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 916, characters 2-13"},"message":"Loaded genesis ledger from $ledger_file","metadata":{"ledger_file":"/root/.mina-config/genesis/genesis_ledger_accounts_3e16f7ed32d2507ceea1f11d6fc1443e82bce2d3b061218501c2c45fe047100f.tar.gz"}}
Nov 11 14:26:03 m23636.contaboserver.net mina[1975]: Using block producer keypair private-key password from environment variable MINA_PRIVKEY_PASS
Nov 11 14:26:03 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:26:03.908644Z","level":"Debug","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 1059, characters 14-26"},"message":"Long async job, $long_async_job seconds, $monitors, $o1trace","metadata":{"long_async_job":7.375592418,"monitors":[[{"name":"coda","here":[],"id":"33","has_seen_error":"false","is_detached":"true"},{"name":"","here":[],"id":"2","has_seen_error":"false","is_detached":"true"}],[{"name":"","here":[],"id":"2","has_seen_error":"false","is_detached":"true"}]],"most_recent_2_backtrace":"","o1trace":["mina"]}}
Nov 11 14:26:37 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:26:37.054225Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 1226, characters 10-21"},"message":"Daemon will use chain id 332c8cc05ba8de9efc23a011f57015d8c9ec96fac81d5d3f7a06969faf4bce92","metadata":{}}
Nov 11 14:26:37 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:26:37.054291Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 1227, characters 10-21"},"message":"Daemon running protocol version 3.1.0","metadata":{}}
Nov 11 14:26:37 m23636.contaboserver.net mina[1975]: Using uptime submitter keypair private-key password from environment variable UPTIME_PRIVKEY_PASS
Nov 11 14:26:37 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:26:37.054464Z","level":"Debug","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 1040, characters 14-26"},"message":"Long async cycle, $long_async_cycle seconds, $monitors, $o1trace","metadata":{"long_async_cycle":33.145957445,"monitors":[[{"name":"","here":[],"id":"1505","has_seen_error":"false","is_detached":"true"}]],"o1trace":["mina"]}}
Nov 11 14:26:37 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:26:37.054529Z","level":"Debug","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 1059, characters 14-26"},"message":"Long async job, $long_async_job seconds, $monitors, $o1trace","metadata":{"long_async_job":30.863257878,"monitors":[[{"name":"coda","here":[],"id":"33","has_seen_error":"false","is_detached":"true"},{"name":"","here":[],"id":"2","has_seen_error":"false","is_detached":"true"}],[{"name":"","here":[],"id":"2","has_seen_error":"false","is_detached":"true"}]],"most_recent_2_backtrace":"","o1trace":["mina"]}}
Nov 11 14:26:37 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:26:37.054582Z","level":"Debug","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 1059, characters 14-26"},"message":"Long async job, $long_async_job seconds, $monitors, $o1trace","metadata":{"long_async_job":2.282347915,"monitors":[[{"name":"coda","here":[],"id":"33","has_seen_error":"false","is_detached":"true"},{"name":"","here":[],"id":"2","has_seen_error":"false","is_detached":"true"}],[{"name":"","here":[],"id":"2","has_seen_error":"false","is_detached":"true"}]],"most_recent_2_backtrace":"","o1trace":["mina"]}}
Nov 11 14:26:37 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:26:37.271738Z","level":"Info","source":{"module":"Prover","location":"File \"src/lib/prover/prover.ml\", line 367, characters 2-13"},"message":"Starting a new prover process","metadata":{}}
Nov 11 14:27:10 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:10.878550Z","level":"Info","source":{"module":"Prover","location":"File \"src/lib/prover/prover.ml\", line 385, characters 2-13"},"message":"Daemon started process of kind $process_kind with pid $prover_pid","metadata":{"process_kind":"Prover","prover_pid":2035}}
Nov 11 14:27:10 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:10.878700Z","level":"Info","source":{"module":"Verifier__Prod","location":"File \"src/lib/verifier/prod.ml\", line 410, characters 4-15"},"message":"Starting a new verifier process","metadata":{}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.481613Z","level":"Info","source":{"module":"Verifier__Prod","location":"File \"src/lib/verifier/prod.ml\", line 450, characters 4-15"},"message":"Daemon started process of kind $process_kind with pid $verifier_pid","metadata":{"process_kind":"Verifier","verifier_pid":2083}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.481750Z","level":"Info","source":{"module":"Vrf_evaluator","location":"File \"src/lib/vrf_evaluator/vrf_evaluator.ml\", line 416, characters 2-13"},"message":"Starting a new vrf-evaluator process","metadata":{}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.592332Z","level":"Info","source":{"module":"Vrf_evaluator","location":"File \"src/lib/vrf_evaluator/vrf_evaluator.ml\", line 422, characters 2-13"},"message":"Daemon started process of kind $process_kind with pid $vrf_evaluator_pid","metadata":{"process_kind":"Vrf_evaluator","vrf_evaluator_pid":2142}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.592455Z","level":"Info","source":{"module":"Uptime_service__Uptime_snark_worker","location":"File \"src/lib/uptime_service/uptime_snark_worker.ml\", line 104, characters 2-13"},"message":"Starting a new uptime service SNARK worker process","metadata":{}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.718440Z","level":"Info","source":{"module":"Uptime_service__Uptime_snark_worker","location":"File \"src/lib/uptime_service/uptime_snark_worker.ml\", line 110, characters 2-13"},"message":"Daemon started process of kind $process_kind with pid $uptime_snark_worker_pid","metadata":{"process_kind":"Uptime_snark_worker","uptime_snark_worker_pid":2147}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.718507Z","level":"Info","source":{"module":"Exit_handlers","location":"File \"src/lib/exit_handlers/exit_handlers.ml\", line 28, characters 2-13"},"message":"Registering async shutdown handler: $description","metadata":{"description":"Close transition frontier, if exists"}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: Environment variable MINA_TIME_OFFSET not found, using default of 0
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.718606Z","level":"Debug","source":{"module":"Network_pool__Transaction_pool","location":"File \"src/lib/network_pool/transaction_pool.ml\", line 801, characters 17-29"},"message":"no frontier","metadata":{}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.718620Z","level":"Debug","source":{"module":"Network_pool__Network_pool_base","location":"File \"src/lib/network_pool/network_pool_base.ml\", line 160, characters 8-30"},"message":"transaction_pool $rate_limiter","metadata":{"rate_limiter":{"by_ip":[],"by_peer_id":[]}}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.718645Z","level":"Debug","source":{"module":"Network_pool__Network_pool_base","location":"File \"src/lib/network_pool/network_pool_base.ml\", line 160, characters 8-30"},"message":"snark_pool $rate_limiter","metadata":{"rate_limiter":{"by_ip":[],"by_peer_id":[]}}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.718658Z","level":"Debug","source":{"module":"Transition_handler__Block_sink","location":"File \"src/lib/transition_handler/block_sink.ml\", line 197, characters 6-18"},"message":"new_block $rate_limiter","metadata":{"rate_limiter":{"by_ip":[],"by_peer_id":[]}}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.718720Z","level":"Debug","source":{"module":"Uptime_service__Uptime_snark_worker","location":"File \"src/lib/uptime_service/uptime_snark_worker.ml\", line 133, characters 12-24"},"message":"Uptime SNARK worker stdout: $stdout","metadata":{"stdout":"{\"timestamp\":\"2023-11-11 13:27:41.718113Z\",\"level\":\"Info\",\"source\":{\"module\":\"Uptime_service__Uptime_snark_worker\",\"location\":\"File \\\"src/lib/uptime_service/uptime_snark_worker.ml\\\", line 82, characters 8-19\"},\"message\":\"Uptime SNARK worker started\",\"metadata\":{}}\n"}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.789940Z","level":"Debug","source":{"module":"Child_processes","location":"File \"src/lib/child_processes/child_processes.ml\", line 119, characters 6-18"},"message":"Found PID file for libp2p_helper /root/.mina-config/mina_net2/libp2p_helper.lock with contents 1954","metadata":{}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.789990Z","level":"Debug","source":{"module":"Child_processes","location":"File \"src/lib/child_processes/child_processes.ml\", line 124, characters 12-24"},"message":"Couldn't kill libp2p_helper with PID 1954, does not exist","metadata":{}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.790103Z","level":"Debug","source":{"module":"Child_processes","location":"File \"src/lib/child_processes/child_processes.ml\", line 217, characters 2-14"},"message":"Starting custom child process $name with args $args","metadata":{"args":[],"name":"libp2p_helper"}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.791116Z","level":"Info","source":{"module":"Child_processes","location":"File \"src/lib/child_processes/child_processes.ml\", line 239, characters 2-13"},"message":"Custom child process $name started with pid $pid","metadata":{"args":[],"name":"libp2p_helper","pid":2154}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.791531Z","level":"Info","source":{"module":"Gossip_net__Libp2p","location":"File \"src/lib/gossip_net/libp2p.ml\", line 266, characters 10-36"},"message":"libp2p peer ID this session is $peer_id","metadata":{"host":"161.97.171.36","peer_id":"12D3KooWBRzYmZ1i94WV4zWNNXDA6enGJp3qnSsR7nPtfeFFj54p","port":8302}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.791560Z","level":"Debug","source":{"module":"Gossip_net__Libp2p","location":"File \"src/lib/gossip_net/libp2p.ml\", line 269, characters 12-39"},"message":"(Re)initializing libp2p result","metadata":{"host":"161.97.171.36","peer_id":"12D3KooWBRzYmZ1i94WV4zWNNXDA6enGJp3qnSsR7nPtfeFFj54p","port":8302}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.791984Z","level":"Info","source":{"module":"Node_error_service","location":"File \"src/lib/node_error_service/node_error_service.ml\", line 111, characters 10-21"},"message":"Could not send error report: mina instance has not been created yet.","metadata":{"host":"161.97.171.36","peer_id":"12D3KooWBRzYmZ1i94WV4zWNNXDA6enGJp3qnSsR7nPtfeFFj54p","port":8302}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.792112Z","level":"Fatal","source":{"module":"Init__Mina_run","location":"File \"src/app/cli/src/init/mina_run.ml\", line 670, characters 2-26"},"message":"Unhandled top-level exception: $exn\nGenerating crash report","metadata":{"exn":{"commit_id":"55b78189c46e1811b8bdb78864cfa95409aeb96a","sexp":["monitor.ml.Error",["src/common.ml.Read_error","Option_code","16"],["Raised at Bin_prot__Common.raise_read_error in file \"src/common.ml\", line 74, characters 31-60","Called from Core_kernel__Option.bin_read_t in file \"src/option.ml\" (inlined), line 4, characters 0-50","Called from Core_kernel__Std_internal.bin_read_option in file \"src/std_internal.ml\" (inlined), line 137, characters 2-103","Called from Trust_system__Record.Stable.V1.bin_read_t in file \"src/lib/trust_system/record.ml\", line 9, characters 8-55","Called from Core_kernel__Binable0.of_bigstring in file \"src/binable0.ml\", line 169, characters 10-41","Called from Rocksdb__Serializable.Make.to_alist.(fun) in file \"src/lib/rocksdb/serializable.ml\", line 49, characters 10-51","Called from Base__List.count_map in file \"src/list.ml\", line 405, characters 13-17","Called from Trust_system__Peer_trust.Make0.peer_statuses.(fun) in file \"src/lib/trust_system/peer_trust.ml\", line 132, characters 8-23","Called from Gossip_net__Libp2p.Make.T.create_libp2p.(fun) in file \"src/lib/gossip_net/libp2p.ml\", line 311, characters 24-70","Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69","Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5","Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47","Caught by monitor coda"]],"backtrace":["Raised at Bin_prot__Common.raise_read_error in file \"src/common.ml\", line 74, characters 31-60","Called from Core_kernel__Option.bin_read_t in file \"src/option.ml\" (inlined), line 4, characters 0-50","Called from Core_kernel__Std_internal.bin_read_option in file \"src/std_internal.ml\" (inlined), line 137, characters 2-103","Called from Trust_system__Record.Stable.V1.bin_read_t in file \"src/lib/trust_system/record.ml\", line 9, characters 8-55","Called from Core_kernel__Binable0.of_bigstring in file \"src/binable0.ml\", line 169, characters 10-41","Called from Rocksdb__Serializable.Make.to_alist.(fun) in file \"src/lib/rocksdb/serializable.ml\", line 49, characters 10-51","Called from Base__List.count_map in file \"src/list.ml\", line 405, characters 13-17","Called from Trust_system__Peer_trust.Make0.peer_statuses.(fun) in file \"src/lib/trust_system/peer_trust.ml\", line 132, characters 8-23","Called from Gossip_net__Libp2p.Make.T.create_libp2p.(fun) in file \"src/lib/gossip_net/libp2p.ml\", line 311, characters 24-70","Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69","Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5","Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"]},"host":"161.97.171.36","peer_id":"12D3KooWBRzYmZ1i94WV4zWNNXDA6enGJp3qnSsR7nPtfeFFj54p","port":8302}}
Nov 11 14:27:41 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:41.800403Z","level":"Debug","source":{"module":"Init__Mina_run","location":"File \"src/app/cli/src/init/mina_run.ml\", line 99, characters 2-14"},"message":"Successfully wrote the visualization of the registered masks at location: /root/.mina-config/coda_crash_report_2023-11-11_13-27-41.793284/registered_masks.dot.","metadata":{"coda_run":"Logging state before program ends","host":"161.97.171.36","peer_id":"12D3KooWBRzYmZ1i94WV4zWNNXDA6enGJp3qnSsR7nPtfeFFj54p","port":8302}}
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]: {"timestamp":"2023-11-11 13:27:43.002353Z","level":"Warn","source":{"module":"Init__Mina_run","location":"File \"src/app/cli/src/init/mina_run.ml\", line 103, characters 6-17"},"message":"Shutdown before Coda instance was created, not saving a visualization","metadata":{"coda_run":"Logging state before program ends","host":"161.97.171.36","peer_id":"12D3KooWBRzYmZ1i94WV4zWNNXDA6enGJp3qnSsR7nPtfeFFj54p","port":8302}}
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]:   ☠  Mina Daemon crashed.
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]:    The Mina Protocol developers would like to know why!
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]:     Please:
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]:       Open an issue:
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]:         <https://github.com/MinaProtocol/mina/issues/new>
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]:       Briefly describe what you were doing and attach the crash report /root/.mina-config/coda_crash_report_2023-11-11_13-27-41.793284.tar.gz
Nov 11 14:27:43 m23636.contaboserver.net mina[1975]:
Nov 11 14:27:43 m23636.contaboserver.net systemd[663]: mina.service: Main process exited, code=exited, status=1/FAILURE
Nov 11 14:27:43 m23636.contaboserver.net systemd[663]: mina.service: Failed with result 'exit-code'.

I've successfully completed every steps to connect Main-Net and synced completely but i can't achieve same thing on Berkeley Test-Net.

Steps to Reproduce

I've completed the below steps carefully after installing the latest berkeley-testnet mina branch.

  1. Created the ~/.mina-env file.
  2. To produce blocks, added the required configuration:
PEER_LIST_URL=https://storage.googleapis.com/seed-lists/testworld-2-0_seeds.txt 
MINA_PRIVKEY_PASS="My_V3ry_S3cure_Password"
UPTIME_PRIVKEY_PASS="My_V3ry_S3cure_Password"
MINA_LIBP2P_PASS="My_V3ry_S3cure_Password"
RAYON_NUM_THREADS=6
LOG_LEVEL=Debug
FILE_LOG_LEVEL=Debug
EXTRA_FLAGS=" "

EXTRA_FLAGS parameters have been updated as on the 3rd step.

  1. Configured the block producer node with the following flags:
--log-json
--log-snark-work-gossip true
--internal-tracing
--insecure-rest-server
--config-directory <path to the mina configuration directory> 
--external-ip  <external server IP> 
--itn-keys  f1F38+W3zLcc45fGZcAf9gsZ7o9Rh3ckqZQw6yOJiS4=,6GmWmMYv5oPwQd2xr6YArmU1YXYCAxQAxKH7aYnBdrk=,ZJDkF9EZlhcAU1jyvP3m9GbkhfYa0yPV+UdAqSamr1Q=,NW2Vis7S5G1B9g2l9cKh3shy9qkI1lvhid38763vZDU=,Cg/8l+JleVH8yNwXkoLawbfLHD93Do4KbttyBS7m9hQ=
--itn-graphql-port 3089
--uptime-submitter-key  <uptime-keyfile path> (**same key as block producer key**) 
--uptime-url https://block-producers-uptime-itn.minaprotocol.tools/v1/submit 
--enable-peer-exchange  true
--libp2p-keypair <keyfile path> (**libp2p keypair pre-generated by the Node Operator**)
--log-precomputed-blocks true
--generate-genesis-proof  true
--block-producer-key <keyfile path> (**keys will be distributed prior to launch**)
--node-status-url https://nodestats-itn.minaprotocol.tools/submit/stats 
--node-error-url https://nodestats-itn.minaprotocol.tools/submit/stats 
--file-log-rotations 500

and the following environment variables:

RAYON_NUM_THREADS=6
UPTIME_PRIVKEY_PASS
MINA_LIBP2P_PASS
MINA_PRIVKEY_PASS

  1. Generated Libp2p Keypair into /root/keys/libp2p-key file mina libp2p generate-keypair /root/keys/my-wallet /root/keys/libp2p-key

  2. Monitor the mina process that's running in the background and auto-restarting.

Checked if mina had any trouble getting started:

systemctl --user status mina

Stop mina gracefully and stop automatically restarting the service:

systemctl --user stop mina

Manually restart the mina process:

systemctl --user restart mina

Look at logs that show the last 1000 lines, and follow from there:

journalctl --user-unit mina -n 1000 -f

Expected Result

Actual Result

How frequently do you see this issue?

Always

What is the impact of this issue on your ability to run a node?

Blocker

Status

Cannot having a daemon status since it is crashing.

Error: Unable to connect to Mina daemon.
- The daemon might not be running. See logs (in `~/.mina-config/mina.log`) for details under the host:127.0.0.1.
  Run `mina daemon -help` to see how to start daemon.
- If you just started the daemon, wait a minute for the RPC server to start.
- Alternatively, the daemon may not be running the RPC server on (127.0.0.1 8301).
  If so, add flag `---daemon-port` with correct port when running this command.

Additional information

Here you can find the detailed crash report which was logged. coda_crash_report_2023-11-11_13-27-41.793284.tar.gz => Log File

vfrsilva commented 7 months ago

@Atikocak , we are currently running the latest testworld release, rampup6. Could you please update your node and see you still face the same issue? https://github.com/MinaProtocol/mina/releases/tag/2.0.0rampup6

Atikocak commented 7 months ago

@Atikocak , we are currently running the latest testworld release, rampup6. Could you please update your node and see you still face the same issue? https://github.com/MinaProtocol/mina/releases/tag/2.0.0rampup6

Hello, I've updated the Mina Daemon to the latest release (rampup6) that you have given.

Here is my latest issue;

Nov 16 20:50:02 m23636.contaboserver.net systemd[687]: Started Mina Daemon Service.
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: {"timestamp":"2023-11-16 19:50:02.796837Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 559, characters 8-19"},"message":"Mina daemon is booting up; built with commit $commit on branch $branch","metadata":{"branch":"rampup","commit":"4061884b18137c1182c7fcfa80f52804008a2509","commit_date":"2023-10-20T16:15:24+02:00","marlin_commit":"fd7c19d4c5dbb6826d1d5ce6bc24796a44087f70"}}
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: {"timestamp":"2023-11-16 19:50:02.797488Z","level":"Info","source":{"module":"Mina_lib__Conf_dir","location":"File \"src/lib/mina_lib/conf_dir.ml\", line 21, characters 10-21"},"message":"Created daemon lockfile $lockfile","metadata":{"lockfile":"/root/.mina-config/.mina-lock"}}
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: {"timestamp":"2023-11-16 19:50:02.797534Z","level":"Info","source":{"module":"Exit_handlers","location":"File \"src/lib/exit_handlers/exit_handlers.ml\", line 28, characters 2-13"},"message":"Registering async shutdown handler: $description","metadata":{"description":"Remove daemon lockfile"}}
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: {"timestamp":"2023-11-16 19:50:02.797560Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 567, characters 10-21"},"message":"Daemon will expire at $exp","metadata":{"exp":"2024-12-10 14:00:00-07:00"}}
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: {"timestamp":"2023-11-16 19:50:02.797588Z","level":"Info","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 580, characters 8-19"},"message":"Booting may take several seconds, please wait","metadata":{}}
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: Using libp2p keypair private-key password from environment variable MINA_LIBP2P_PASS
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: {"timestamp":"2023-11-16 19:50:02.798018Z","level":"Info","source":{"module":"Exit_handlers","location":"File \"src/lib/exit_handlers/exit_handlers.ml\", line 31, characters 4-15"},"message":"Running async shutdown handler: $description","metadata":{"description":"Remove daemon lockfile"}}
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]: (monitor.ml.Error
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:  ("Yojson.Json_error(\"Line 1, bytes 0-33:\\nInvalid token 'CAESQDwCBkrmPROgwtEiqEwyCxgyk4zUn'\")")
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:  ("Raised at Yojson.json_error in file \"common.ml\", line 5, characters 19-39"
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:   "Called from Yojson.Safe.read_json in file \"lib/read.ml\" (inlined), line 1025, characters 3-39"
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:   "Called from Yojson.Safe.from_lexbuf in file \"lib/read.mll\", line 1109, characters 8-26"
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:   "Called from Yojson.Safe.from_string in file \"lib/read.mll\", line 1122, characters 6-26"
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:   "Called from Secrets__Secret_file.read.(fun) in file \"src/lib/secrets/secret_file.ml\", line 130, characters 31-70"
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:   "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69"
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:   "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
Nov 16 20:50:02 m23636.contaboserver.net mina[33755]:   "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"))
Nov 16 20:50:02 m23636.contaboserver.net systemd[687]: mina.service: Main process exited, code=exited, status=1/FAILURE
Nov 16 20:50:02 m23636.contaboserver.net systemd[687]: mina.service: Failed with result 'exit-code'.

OS: Ubuntu 20.04 (bullseye/sid)

vfrsilva commented 7 months ago

@Atikocak have you found a fix? Looking at the logs it seems a problem with an invalid Token. May we close the issue?