Mina daemon keeps restarting

[jun0tpyrc]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
• [X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

in mina.log, grep error only showing

{"timestamp":"2021-10-13 05:47:06.481544Z","level":"Debug","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 529, characters 10-22"},"message":"Error
reading $file: $error. Cleaning up the config directory $config_directory","metadata":{"config_directory":"/root/.mina-config","error":"(Sys_error\"/root/.mina-config/mina.version: No such file or directory\")","file":"/root/.mina-config/mina.version","pid":16}}
{"timestamp":"2021-10-13 05:47:06.511652Z","level":"Warn","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 678, characters 18-29"},"message":"Could n
ot read configuration from $config_file","metadata":{"config_file":"/root/.mina-config/daemon.json","error":{"commit_id":"5b35b27df8e137ddbe88874b4efdcf4bb2cb4ae9","exn_name":"Async_kernel__Monitor.Error_","exn":["monitor.
ml.Error",["Unix.Unix_error","No such file or directory","open","((filename /root/.mina-config/daemon.json) (mode (O_RDONLY O_CLOEXEC)) (perm 0o0))"],["Raised at file \"src/result.ml\", line 158, characters 17-26","Called
from file \"src/deferred1.ml\", line 17, characters 40-45","Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5","Called from file \"src/job_queue.ml\", line 171, characters 6-47","Caught by monitor try_with_or_error at file \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 754, characters 34-34"]]},"pid":16}}

Steps to Reproduce

run docker mina with approximately

  mina:
    image: "gcr.io/o1labs-192920/mina-daemon:1.2.0beta8-5b35b27-devnet"
    restart: unless-stopped
    stop_grace_period: 30s
    env_file:
      - daemon.env
    ulimits:
      nofile: "60000"
      nproc: "60000"
      memlock: "-1"
    volumes:
      - ../app_data/daemon/keys/:/keys:ro
      - ../app_data/daemon/config/:/root/.mina-config/
    command: daemon --insecure-rest-server --block-producer-key /keys/my-wallet --stop-time 1

Expected Result

keeps staying up and sync

Actual Result

keep restart

How frequently do you see this issue?

Always

What is the impact of this issue on your ability to run a node?

Blocker

Status

can't get it up and start to sync

Additional information

{"timestamp":"2021-10-13 05:51:32.914238Z","level":"Warn","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 678, characters 18-29"},"message":"Could not read configuration from $config_file","metadata":{"config_file":"/root/.mina-config/daemon.json","error":{"commit_id":"5b35b27df8e137ddbe88874b4efdcf4bb2cb4ae9","exn_name":"Async_kernel__Monitor.Error_","exn":["monitor.ml.Error",["Unix.Unix_error","No such file or directory","open","((filename /root/.mina-config/daemon.json) (mode (O_RDONLY O_CLOEXEC)) (perm 0o0))"],["Raised at file \"src/result.ml\", line 158, characters 17-26","Called from file \"src/deferred1.ml\", line 17, characters 40-45","Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5","Called from file \"src/job_queue.ml\", line 171, characters 6-47","Caught by monitor try_with_or_error at file \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 754, characters 34-34"]]},"pid":16}}

long line for {"timestamp":"2021-10-13 05:51:32.926439Z","level":"Info","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 781, characters 2-13"},"message":"Initializing with runtime configuration. Ledger name: $name"......

{"timestamp":"2021-10-13 05:51:32.942798Z","level":"Info","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 803, characters 8-19"},"message":"Using the compiled constraint constants","metadata":{"pid":16}}
{"timestamp":"2021-10-13 05:51:32.953453Z","level":"Trace","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 113, characters 8-20"},"message":"Ledger file $path does not exist","metadata":{"path":"/var/lib/coda/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:32.953633Z","level":"Trace","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 113, characters 8-20"},"message":"Ledger file $path does not exist","metadata":{"path":"/usr/local/var/coda/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:32.953726Z","level":"Trace","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 113, characters 8-20"},"message":"Ledger file $path does not exist","metadata":{"path":"/tmp/s3_cache_dir/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:32.953793Z","level":"Trace","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 113, characters 8-20"},"message":"Ledger file $path does not exist","metadata":{"path":"/tmp/coda_cache_dir/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:32.953826Z","level":"Trace","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 113, characters 8-20"},"message":"Ledger file $path does not exist","metadata":{"path":"/var/lib/coda/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:32.953894Z","level":"Trace","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 108, characters 8-20"},"message":"Found $ledger file at $path","metadata":{"ledger":"genesis_ledger","path":"/root/.mina-config/genesis/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:32.953909Z","level":"Trace","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 180, characters 4-16"},"message":"Loading $ledger from $path","metadata":{"ledger":"genesis_ledger","path":"/root/.mina-config/genesis/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:33.022380Z","level":"Info","source":{"module":"Genesis_ledger_helper","location":"File \"src/lib/genesis_ledger_helper/genesis_ledger_helper.ml\", line 850, characters 2-13"},"message":"Loaded genesis ledger from $ledger_file","metadata":{"ledger_file":"/root/.mina-config/genesis/genesis_ledger_accounts_07c4db3946017fb3bd5383864c78e30e6a5571d63c2d3ca3c9a822e7f97b4d87.tar.gz","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345468Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"--external-port","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345539Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"--rest-port","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345550Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"--client-port","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345558Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"snark-worker-fee","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345567Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"work-selection","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345575Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"work-reassignment-wait","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345583Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"log-snark-work-gossip","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345591Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"log-txn-pool-gossip","pid":16}}
{"timestamp":"2021-10-13 05:51:33.345599Z","level":"Trace","source":{"module":"Mina_cli_entrypoint","location":"File \"src/app/cli/src/cli_entrypoint/mina_cli_entrypoint.ml\", line 754, characters 12-24"},"message":"Key '$key' not found in the config file, using default","metadata":{"key":"log-block-creation","pid":16}}
{"timestamp":"2021-10-13 05:51:33.799785Z","level":"Info","source":{"module":"Secrets__Keypair_common","location":"File \"src/lib/secrets/keypair_common.ml\", line 76, characters 10-21"},"message":"Using block producer keypair private-key password from environment variable CODA_PRIVKEY_PASS","metadata":{"pid":16}}

[joseandro]

Hi @jun0tpyrc, could you check if your $config_file directory has the right permissions? After you fixed the permissions for the above, leave the daemon working for some hours and then check if synching worked.

[Mitsushima1]

@joseandro I met the same issue. Someone in the community helped me solve it.

The key is, the default for Docker Toolbox is tiny (like 2 GB) so give it way more and some decent CPUs. Might not be it, but that's the most common issue.