search

MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.97k stars 523 forks source link

Snark worker crash #15674

Open CryptoTech-Guy opened 1 month ago

CryptoTech-Guy commented 1 month ago

Preliminary Checks

Description

When starting snark worker it works for few minutes/hours until crashing. Mina node is synced. crash log:

{"timestamp":"2024-05-24 19:55:53.441311Z","level":"Info","source":{"module":"Snark_worker__Functor","location":"File \"src/lib/snark_worker/functor.ml\", line 202, characters 10-21"},"message":"SNARK work $work_ids received from $address. Starting proof generation","metadata":{"address":"127.0.0.1:8301","pid":323583,"process":"Snark Worker","work_ids":[748525845,514954334]}}
(monitor.ml.Error
 ( "Snarky.Checked_runner.Runtime_error(_, _, _, _)\
  \n\
  \nEncountered an error while evaluating the checked computation:\
  \n  (Failure\
  \n   \"Constraint unsatisfied (unreduced):\\\
  \n  \\n\\\
  \n  \\n\\\
  \n  \\n\\\
  \n  \\nConstraint:\\\
  \n  \\n(((basic(Square(Var 32163)(Add(Var 32241)(Constant 5))))(annotation())))\\\
  \n  \\nData:\\\
  \n  \\nSquare 18315271761819935291118228892551565974065550412924806371350372225761100842006 22895585251652956032213272723278463074111444215619683920291624983582870186504\")\
  \n\
  \nLabel stack trace:\
  \n\
  \n\
  \n\
  \nRaised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33\
  \nCalled from Snarky_backendless__Checked_runner.Make_checked.add_constraint in file \"src/lib/snarky/src/base/checked_runner.ml\", line 188, characters 8-329\
  \nCalled from Snarky_backendless__Checked_runner.Make.handle_error in file \"src/lib/snarky/src/base/checked_runner.ml\", line 345, characters 8-12\
  \n")
 ("Raised at Snarky_backendless__Checked_runner.Make.handle_error in file \"src/lib/snarky/src/base/checked_runner.ml\", line 352, characters 8-55"
  "Called from Snarky_backendless__Checked_runner.Make.run in file \"src/lib/snarky/src/base/checked_runner.ml\", line 439, characters 19-60"
  "Called from Snarky_backendless__Snark0.Run.Make_basic.run in file \"src/lib/snarky/src/base/snark0.ml\", line 1808, characters 22-47"
  "Called from Pickles__Wrap_main.wrap_main.main in file \"src/lib/pickles/wrap_main.ml\", line 376, characters 6-956"
  "Called from Snarky_backendless__Snark0.Run.Make_basic.handle in file \"src/lib/snarky/src/base/snark0.ml\", line 2461, characters 14-18"
  "Called from Snarky_backendless__Snark0.Run.Make_basic.Proof_system.mark_active in file \"src/lib/snarky/src/base/snark0.ml\", line 2370, characters 20-24"
  "Re-raised at Snarky_backendless__Snark0.Run.Make_basic.Proof_system.mark_active in file \"src/lib/snarky/src/base/snark0.ml\", line 2375, characters 10-19"
  "Called from Snarky_backendless__Snark0.Run.Make_basic.as_stateful in file \"src/lib/snarky/src/base/snark0.ml\", line 1814, characters 14-18"
  "Called from Snarky_backendless__Snark0.Make_basic.Checked.auxiliary_input in file \"src/lib/snarky/src/base/snark0.ml\", line 207, characters 13-27"
  "Called from Snarky_backendless__Snark0.Make_basic.Run.generate_witness_conv.(fun) in file \"src/lib/snarky/src/base/snark0.ml\", line 1132, characters 12-132"
  "Called from Pickles__Wrap.wrap.(fun) in file \"src/lib/pickles/wrap.ml\", line 318, characters 4-1023"
  "Called from Pickles.Make.compile.f.wrap.(fun) in file \"src/lib/pickles/pickles.ml\", line 740, characters 12-383"
  "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69"
  "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
  "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"))

What I already tried:

Steps to Reproduce

  1. Start snark worker
  2. Wait until it crash
  3. Check the logs

Expected Result

Worker should not crash

Actual Result

Worker crash after few minutes/hours

Daemon version

Commit 039296a260080ed02d0d0750d185921f030b6c9c on branch compatible

How frequently do you see this issue?

Frequently

What is the impact of this issue on your ability to run a node?

High

Status

Mina daemon status
-----------------------------------

Global number of accounts:                     227196
Block height:                                  356789
Max observed block height:                     356789
Max observed unvalidated block height:         356789
Local uptime:                                  1h17m23s
Ledger Merkle root:                            jwnJ6BhhYLMZLA97j2wHMmwP1dMYDhLtvZoJ5ktppBLrzMAbtGQ
Protocol state hash:                           3NKQQYfEnaZKrR9CPL9dkda61GEvdXeooYg7L579eUTB2e1thzRM
Chain id:                                      5f704cc0c82e0ed70e873f0893d7e06f148524e3f0bdae2afb02e7819a0c24d1
Git SHA-1:                                     039296a260080ed02d0d0750d185921f030b6c9c
Configuration directory:                       ----/.mina-config
Peers:                                         110
User_commands sent:                            0
SNARK worker:                                  None
SNARK work fee:                                1000000
Sync status:                                   Synced
Catchup status:
        To build breadcrumb:           0
        To initial validate:           0
        Finished:                      299
        To download:                   0
        Waiting for parent to finish:  0
        To verify:                     0

Block producers running:                       1 (-----)
Coinbase receiver:                             -----
Best tip consensus time:                       epoch=78, slot=2202
Best tip global slot (across all hard-forks):  559122
Next block will be produced in:                None this epoch… checking at in 10.286d (Generated from consensus at slot: 558578 slot-since-genesis: 558578)
Consensus time now:                            epoch=78, slot=2202
Consensus mechanism:                           proof_of_stake
Consensus configuration:
        Delta:                     0
        k:                         290
        Slots per epoch:           7140
        Slot duration:             3m
        Epoch duration:            14d21h
        Chain start timestamp:     2021-03-17 00:00:00.000000Z
        Acceptable network delay:  3m

Addresses and ports:
        External IP:    -----
        Bind IP:        0.0.0.0
        Libp2p PeerID:  -----
        Libp2p port:    -----
        Client port:    -----

Metrics:
        block_production_delay:             7 (0 0 0 0 0 0 0)
        transaction_pool_diff_received:     577
        transaction_pool_diff_broadcasted:  0
        transactions_added_to_pool:         88
        transaction_pool_size:              17

Additional information

No response

CryptoTech-Guy commented 1 month ago

I see one more error in the log: {"timestamp":"2024-05-25 08:05:44.986668Z","level":"Error","source":{"module":"Snark_worker__Functor","location":"File \"src/lib/snark_worker/functor.ml\", line 158, characters 8-20"},"message":"Error submitting work: ((rpc_error\n (Bin_io_exn\n ((location \"server-side rpc query un-bin-io'ing\")\n (exn\n (\"Zexe_backend_common.Curve.Make(BaseField)(ScalarField)(Params)(C).Affine.Stable.V1.Invalid_curve_point(_)\")))))\n (connection_description (\"Client connected via TCP\" (127.0.0.1 8301)))\n (rpc_tag submit_work) (rpc_version 1))","metadata":{"pid":326747,"process":"Snark Worker"}}

rbonichon commented 1 month ago

@CryptoTech-Guy Would you be able to share your OS/hardware configuration? You mention a commit hash: can you confirm you have compiled the node from source?

CryptoTech-Guy commented 3 weeks ago

@CryptoTech-Guy Would you be able to share your OS/hardware configuration? You mention a commit hash: can you confirm you have compiled the node from source?

HW is AMD Ryzen 9 5900, 64GB RAM, OS Ubuntu 20.04 I did not compile it from source, I install it from Ubuntu pkg (apt)

CryptoTech-Guy commented 3 weeks ago

I upgraded to v3 but still getting errors. I see now that also mina daemon crashed

Jun 07 14:49:03 systemd[1713]: Started Mina Daemon Service. Jun 07 14:49:04 mina[91982]: 2024-06-07 11:49:04 UTC [Info] Mina daemon is booting up; built with commit "93e02797f72abe2cbf8dfca6f2328e9c8cd76546" on branch "berkeley" Jun 07 14:49:04 mina[91982]: (monitor.ml.Error Jun 07 14:49:04 mina[91982]: ("Error processing lockfile" Jun 07 14:49:04 mina[91982]: (lockfile /home/.mina-config/.mina-lock) Jun 07 14:49:04 mina[91982]: (Unix.Unix_error "Operation not permitted" kill "")) Jun 07 14:49:04 mina[91982]: ("Raised at Base__Error.raise in file \"src/error.ml\" (inlined), line 8, characters 14-30" Jun 07 14:49:04 mina[91982]: "Called from Mina_lib__Conf_dir.check_and_set_lockfile.(fun) in file \"src/lib/mina_lib/conf_dir.ml\", line 80, characters 10-158" Jun 07 14:49:04 mina[91982]: "Called from Async_kernel__Deferred1.M.map.(fun) in file \"src/deferred1.ml\", line 17, characters 40-45" Jun 07 14:49:04 mina[91982]: "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5" Jun 07 14:49:04 mina[91982]: "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47")) Jun 07 14:49:04 systemd[1713]: mina.service: Main process exited, code=exited, status=1/FAILURE Jun 07 14:49:04 systemd[1713]: mina.service: Failed with result 'exit-code'.

CryptoTech-Guy commented 3 weeks ago

Mina daemon log:

Jun 07 17:38:10  mina[16386]: 2024-06-07 14:38:10 UTC [Error] Failed to generate SNARK work: $error
Jun 07 17:38:10  mina[16386]:         error: {
Jun 07 17:38:10  mina[16386]:   "commit_id": "93e02797f72abe2cbf8dfca6f2328e9c8cd76546",
Jun 07 17:38:10  mina[16386]:   "sexp": [
Jun 07 17:38:10  mina[16386]:     "monitor.ml.Error",
Jun 07 17:38:10  mina[16386]:     [
Jun 07 17:38:10  mina[16386]:       "Failure",
Jun 07 17:38:10  mina[16386]:       "Constraint unsatisfied (unreduced):\n\nFile \"src/lib/pickles/wrap_main.ml\", line 387, characters 23-30\nFile \"src/lib/pickles/wrap_main.ml\", line 163, characters 15-22\n\nConstraint:\n((basic(Square(Var 32338)(Add(Var 32384)(Constant 5))))(annotation()))\nData:\nSquare 5305232088626891484824847168525350821393798105357875927487058112039760819766 926817814767253181923115173008695618534792233331010679656225523188174116378"
Jun 07 17:38:10  mina[16386]:     ],
Jun 07 17:38:10  mina[16386]:     [
Jun 07 17:38:10  mina[16386]:       "Raised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Make_checked.add_constraint.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 216, characters 12-399",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.run in file \"src/lib/snarky/src/base/snark0.ml\", line 748, characters 26-51",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.as_stateful in file \"src/lib/snarky/src/base/snark0.ml\", line 754, characters 14-18",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Make_checked.exists.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 277, characters 22-47",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Checked_runner.Simple.map.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 54, characters 29-32",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.run in file \"src/lib/snarky/src/base/snark0.ml\", line 748, characters 26-51",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.with_label in file \"src/lib/snarky/src/base/snark0.ml\", line 1260, characters 14-18",
Jun 07 17:38:10  mina[16386]:       "Called from Pickles__Wrap_main.wrap_main.main.(fun) in file \"src/lib/pickles/wrap_main.ml\", line 387, characters 12-426",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.with_label in file \"src/lib/snarky/src/base/snark0.ml\", line 1260, characters 14-18",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.handle in file \"src/lib/snarky/src/base/snark0.ml\", line 1244, characters 14-18",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.mark_active in file \"src/lib/snarky/src/base/snark0.ml\", line 1174, characters 18-22",
Jun 07 17:38:10  mina[16386]:       "Re-raised at Snarky_backendless__Snark0.Run.Make_basic.mark_active in file \"src/lib/snarky/src/base/snark0.ml\", line 1179, characters 8-17",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.as_stateful in file \"src/lib/snarky/src/base/snark0.ml\", line 754, characters 14-18",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Runners.Make.auxiliary_input.(fun) in file \"src/lib/snarky/src/base/runners.ml\", line 82, characters 21-33",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Runners.Make.Run.generate_witness_conv.(fun) in file \"src/lib/snarky/src/base/runners.ml\", line 337, characters 12-98",
Jun 07 17:38:10  mina[16386]:       "Called from Snarky_backendless__Snark0.Run.Make_basic.finalize_is_running in file \"src/lib/snarky/src/base/snark0.ml\", line 1279, characters 14-18",
Jun 07 17:38:10  mina[16386]:       "Re-raised at Snarky_backendless__Snark0.Run.Make_basic.finalize_is_running in file \"src/lib/snarky/src/base/snark0.ml\", line 1285, characters 12-19",
Jun 07 17:38:10  mina[16386]:       "Called from Pickles__Wrap.wrap.(fun) in file \"src/lib/pickles/wrap.ml\", line 863, characters 4-1023",
Jun 07 17:38:10  mina[16386]:       "Called from Pickles__Compile.Make.compile.f.wrap.(fun) in file \"src/lib/pickles/compile.ml\", line 813, characters 12-448",
Jun 07 17:38:10  mina[16386]:       "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69",
Jun 07 17:38:10  mina[16386]:       "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5",
Jun 07 17:38:10  mina[16386]:       "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47",
Jun 07 17:38:10  mina[16386]:       "Caught by monitor at file \"src/lib/snark_worker/prod.ml\", line 262, characters 59-59"
Jun 07 17:38:10  mina[16386]:     ]
Jun 07 17:38:10  mina[16386]:   ]
Jun 07 17:38:10  mina[16386]: }
CryptoTech-Guy commented 3 weeks ago

Snark worker log:

{"timestamp":"2024-06-07 14:38:10.193523Z","level":"Error","source":{"module":"Snark_worker__Prod","location":"File \"src/lib/snark_worker/prod.ml\", line 82, characters 16-28"},"message":"SNARK worker failed: $error","metadata":{"error":{"commit_id":"93e02797f72abe2cbf8dfca6f2328e9c8cd76546","exn_name":"Async_kernel__Monitor.Error_","exn":["monitor.ml.Error",["Failure","Constraint unsatisfied (unreduced):\n\nFile \"src/lib/pickles/wrap_main.ml\", line 387, characters 23-30\nFile \"src/lib/pickles/wrap_main.ml\", line 163, characters 15-22\n\nConstraint:\n((basic(Square(Var 32338)(Add(Var 32384)(Constant 5))))(annotation()))\nData:\nSquare 5305232088626891484824847168525350821393798105357875927487058112039760819766 926817814767253181923115173008695618534792233331010679656225523188174116378"],["Raised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33","Called from Snarky_backendless__Checked_runner.Make_checked.add_constraint.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 216, characters 12-399","Called from Snarky_backendless__Snark0.Run.Make_basic.run in file \"src/lib/snarky/src/base/snark0.ml\", line 748, characters 26-51","Called from Snarky_backendless__Snark0.Run.Make_basic.as_stateful in file \"src/lib/snarky/src/base/snark0.ml\", line 754, characters 14-18","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 64, characters 25-28","Called from Snarky_backendless__Checked_runner.Make_checked.exists.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 277, characters 22-47","Called from Snarky_backendless__Checked_runner.Simple.map.(fun) in file \"src/lib/snarky/src/base/checked_runner.ml\", line 54, characters 29-32","Called from Snarky_backendless__Snark0.Run.Make_basic.run in file \"src/lib/snarky/src/base/snark0.ml\", line 748, characters 26-51","Called from Snarky_backendless__Snark0.Run.Make_basic.with_label in file \"src/lib/snarky/src/base/snark0.ml\", line 1260, characters 14-18","Called from Pickles__Wrap_main.wrap_main.main.(fun) in file \"src/lib/pickles/wrap_main.ml\", line 387, characters 12-426","Called from Snarky_backendless__Snark0.Run.Make_basic.with_label in file \"src/lib/snarky/src/base/snark0.ml\", line 1260, characters 14-18","Called from Snarky_backendless__Snark0.Run.Make_basic.handle in file \"src/lib/snarky/src/base/snark0.ml\", line 1244, characters 14-18","Called from Snarky_backendless__Snark0.Run.Make_basic.mark_active in file \"src/lib/snarky/src/base/snark0.ml\", line 1174, characters 18-22","Re-raised at Snarky_backendless__Snark0.Run.Make_basic.mark_active in file \"src/lib/snarky/src/base/snark0.ml\", line 1179, characters 8-17","Called from Snarky_backendless__Snark0.Run.Make_basic.as_stateful in file \"src/lib/snarky/src/base/snark0.ml\", line 754, characters 14-18","Called from Snarky_backendless__Runners.Make.auxiliary_input.(fun) in file \"src/lib/snarky/src/base/runners.ml\", line 82, characters 21-33","Called from Snarky_backendless__Runners.Make.Run.generate_witness_conv.(fun) in file \"src/lib/snarky/src/base/runners.ml\", line 337, characters 12-98","Called from Snarky_backendless__Snark0.Run.Make_basic.finalize_is_running in file \"src/lib/snarky/src/base/snark0.ml\", line 1279, characters 14-18","Re-raised at Snarky_backendless__Snark0.Run.Make_basic.finalize_is_running in file \"src/lib/snarky/src/base/snark0.ml\", line 1285, characters 12-19","Called from Pickles__Wrap.wrap.(fun) in file \"src/lib/pickles/wrap.ml\", line 863, characters 4-1023","Called from Pickles__Compile.Make.compile.f.wrap.(fun) in file \"src/lib/pickles/compile.ml\", line 813, characters 12-448","Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69","Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5","Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47","Caught by monitor at file \"src/lib/snark_worker/prod.ml\", line 262, characters 59-59"]]},"spec":"(Transition((source((first_pass_ledger 3455488146492394693706246096071744884936335233863499457299129554047975317763)(second_pass_ledger 18316015679542381448767982029391209763708740699521686920950643568637956377259)(pending_coinbase_stack((data 17565849295236487117005698014466407944766819630903075312334386758220362116827)(state((init 20073240908364842602708354313591733737302929896452351308022517722215162205924)(curr 6063073883737604696895778358391759381867758104506343306192323741602717654702)))))(local_state((stack_frame 0x0641662E94D68EC970D0AFC059D02729BBF4A2CD88C548CCD9FB1E26E570C66C)(call_stack 0x0000000000000000000000000000000000000000000000000000000000000000)(transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(full_transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(excess((magnitude 0)(sgn Pos)))(supply_increase((magnitude 0)(sgn Pos)))(ledger 0)(success true)(account_update_index 0)(failure_status_tbl())(will_succeed true)))))(target((first_pass_ledger 22001387713642492732407099378621009763425845576012734824245175384784809583685)(second_pass_ledger 18316015679542381448767982029391209763708740699521686920950643568637956377259)(pending_coinbase_stack((data 17565849295236487117005698014466407944766819630903075312334386758220362116827)(state((init 20073240908364842602708354313591733737302929896452351308022517722215162205924)(curr 6063073883737604696895778358391759381867758104506343306192323741602717654702)))))(local_state((stack_frame 0x0641662E94D68EC970D0AFC059D02729BBF4A2CD88C548CCD9FB1E26E570C66C)(call_stack 0x0000000000000000000000000000000000000000000000000000000000000000)(transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(full_transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(excess((magnitude 0)(sgn Pos)))(supply_increase((magnitude 0)(sgn Pos)))(ledger 0)(success true)(account_update_index 0)(failure_status_tbl())(will_succeed true)))))(connecting_ledger_left 19241988113224295758752373916470321176603390659616275891551494605550872869608)(connecting_ledger_right 19241988113224295758752373916470321176603390659616275891551494605550872869608)(supply_increase((magnitude 0)(sgn Pos)))(fee_excess((fee_token_l 0x0000000000000000000000000000000000000000000000000000000000000001)(fee_excess_l((magnitude 1000000)(sgn Pos)))(fee_token_r 0x0000000000000000000000000000000000000000000000000000000000000001)(fee_excess_r((magnitude 0)(sgn Pos)))))(sok_digest()))((transaction(Command(Signed_command((payload((common((fee 1000000)(fee_payer_pk B62qmPc8Ziq7txW48YPf4qtavcD5mcQVjEAGo9LEZD8DeaNsNthYLsz)(nonce 589939)(valid_until 4294967295)(memo\"\\001\\025payout_from_AURO_Epoch_78\\000\\000\\000\\000\\000\\000\\000\")))(body(Payment((receiver_pk B62qo28fRrLkdn4z14Rtvi2tpop3sLjMdnoTSp87XMSe87ZjHXFpN8G)(amount 225435759))))))(signer B62qmPc8Ziq7txW48YPf4qtavcD5mcQVjEAGo9LEZD8DeaNsNthYLsz)(signature(17259272645117820890421443718780430497537039968705795250884075761350711455142 3829229944808322149301070721507598896223659731448155249200630859207345071102))))))(first_pass_ledger((indexes(((B62qo28fRrLkdn4z14Rtvi2tpop3sLjMdnoTSp87XMSe87ZjHXFpN8G 0x0000000000000000000000000000000000000000000000000000000000000001)70734)((B62qmPc8Ziq7txW48YPf4qtavcD5mcQVjEAGo9LEZD8DeaNsNthYLsz 0x0000000000000000000000000000000000000000000000000000000000000001)23327)))(depth 35)(tree(Node 3455488146492394693706246096071744884936335233863499457299129554047975317763(Node 4896935538106231324733819157267400288379907479927895154775961162702924938724(Node 2286286316253801162265566793542773625848607871885533568401088801136409586232(Node 25066870158313315786464361894130572811004692531552816146005545686185430532273(Node 19368806288603994391309623612871594876104612851514908663908647791458205755582(Node 10688963220738473616407927612185507466806400547673912473874624133987889518635(Node 25612864987280731009520878946928509848306752138892401193279090431330384870469(Node 17433483145388764285620362948648803918684022020901316633315336251108591276292(Node 9488705224279148610829491292713267696348457053456842599930897540626501052566(Node 28172786354155210535318094805648420625852138031315303283717597307373335943189(Node 17097865329375583896732052913721839138538784429295186397170426351325693838651(Node 15638509174257098685893033142345609579996216097806452824851977868284689968627(Node 436051946394997705314014569619550470499263031851348162592933114012351064684(Node 9025903990275369309963826827532058586247160327037693218702925282833585645450(Node 20774154393992723625120781330570054497802259203104483438824774975705959592993(Node 25930230256438808421565347223163597762321369809124680752688717649081539675658(Node 11355043722846786430924804095165457672186697088398109085253006637138708863813(Node 26973198344060567721126806377171956752427827827206045683199147245682568050238(Node 26561192451430353660545503567139529888941730941207148787332708291184468466743(Node 28024556206822703874818160418415864567561836160110932843341731862120348133832(Node 16785603093170350645977938805015263828961280883220077491456887638955944224071(Hash 5706849398072510842534423564308001097976520917549140208838828818519908828110)(Node 8422389934693798737216197842874087372030395093090054934374855155809527578975(Node 13319754598336468228782017610494635297013632352328268211498641529174649637602(Hash 27553577135916178008042195952925360465943520255637590064404300754667219238259)(Node 21576643530345254709958873100586948143640250859313932909461868590676034762468(Hash 22978718094957128311769450305910814147951674289916820203247065143011111914178)(Node 14806959076636932450521135505474883749391424678535749473484955203793487119440(Node 5967157410432088763366005597791604573826671136629073075294590912551139641248(Hash 14899042638274887117136371219137258142930064486673869080677616691248414616648)(Node 7117313511410087407445882315849303121044982881267039232853320182439480516038(Hash 365858617048704353314004208034835053924773080764502431406289461105350871746)(Node 23900011589336057113816561776566551620179834400822353907920685706712991760722(Node 21195916010465654897389818035368843504474133991651920551710577510482642191388(Node 13544674441661357010021905494759838060206980874508709527271991040222496233667(Node 10574733760481548385910408807123770458853829018282456382960607025658808685548(Hash 23695332766093071154639712073849298059152258656200270134875859473724785946606)(Node 23263634602245753451063961655404136967242042685885505431120474684181945389733(Hash 16818864594653795750483594145484500530547799933695170324955063888951381345329)(Node 2496877453414619218031492616541634393216469046518036250332044180284429779599(Hash 26325152942652394623913485608126868359581505358067590149442314590684665654014)(Node 14650425063321498546216564153528642915933898111203203973319106886583536525652(Hash 1672251043519471094809642303791305637378627066391961277420668522602136460606)(Node 3680587171336054203133363858136756108408132310487450218198846966425552929263(Hash 18084489660654256092123642794019593206145259614177986389211515824370883116410)(Account((public_key B62qmPc8Ziq7txW48YPf4qtavcD5mcQVjEAGo9LEZD8DeaNsNthYLsz)(token_id 0x0000000000000000000000000000000000000000000000000000000000000001)(token_symbol\"\")(balance 398914379747273)(nonce 589939)(receipt_chain_hash 21154459158887456426057585321036141159996414770822398629443238991295076955633)(delegate(B62qq3TQ8AP7MFYPVtMx5tZGF3kWLJukfwG1A1RGvaBW1jfTPTkDBW6))(voting_for 0)(timing Untimed)(permissions((edit_state Signature)(access None)(send Signature)(receive None)(set_delegate Signature)(set_permissions Signature)(set_verification_key(Signature 2))(set_zkapp_uri Signature)(edit_action_state Signature)(set_token_symbol Signature)(increment_nonce Signature)(set_voting_for Signature)(set_timing Signature)))(zkapp()))))))))(Hash 14535326754444330769253632848688085426020079440611720935404738894148852020534))(Hash 4261258321617343100879041861929932910306827007384438561383824085360521405176))(Hash 9159231553430092791912033560640189858933046131408391789348846863071862232531))))(Hash 19480264859030129956909669973333920189195012519501437337557487202296617950700))))(Hash 3352669251860896491646515371885097392309067770502712469689504162094571208778)))(Hash 22495362432468903156352510638347767618253966135885698667377801366704087509493))(Node 14358588748488623681584367231170428782913060356674805362705265711375404763507(Node 970458254794803491043352978087886040644313246450778950355763789993242610350(Node 10994084330969181118637684700998887108180462359877289753460481224959044556200(Node 27783180164652626546922553299806267754504226311446337287587883586415830737843(Hash 6884246501993332621537983928870193524404266302369585203513875155889666815689)(Node 25451948983825429819237864915302964069884338826356049636345588513719773726159(Node 27272621692744730389738241238039744697524676861996556388976601834762107217994(Hash 27063384030603089987018917689034604475635173109190182354311324851332695958280)(Node 22257291420462258618079302175178885848975251601236481309800652252914236578943(Node 7092143313093963470258736526609117816161087842627378692458158295228838467707(Node 25659626105974670054097478422740322605348922137245768170946382819870868047384(Node 13069325747605839510577295852987749769160687593278678987057024686258909887737(Hash 28352207452189453189353545832932382907291889718397244734960756295298048940249)(Node 13121848677817956172028023264960745840507758039724114646046564538822736887564(Node 4329917239334573867082753679715083537478375366956888448147681408458554185485(Node 16831843343265513699155123311676738144180054197765746701377035103063470900980(Hash 18279440942993662136385793760814234117901749881152069179880102155545361950211)(Node 16507304703076226972976246300968928067269666571167334207561868938118163021169(Hash 21353556399778004327977345313711978313156505408608475883319723438597365429078)(Node 13921503896241710459348709170002996666378346747962299657899460340656573950701(Hash 17809711804492984618666802407034141827696293269501867085665203717000010976406)(Node 28137792879429315192088715526457743821452218677496898580993267492820485577228(Account((public_key B62qo28fRrLkdn4z14Rtvi2tpop3sLjMdnoTSp87XMSe87ZjHXFpN8G)(token_id 0x0000000000000000000000000000000000000000000000000000000000000001)(token_symbol\"\")(balance 31320255626)(nonce 2)(receipt_chain_hash 23734093168081738593308871271268144933218802735365964038842898715591237181064)(delegate(B62qq3TQ8AP7MFYPVtMx5tZGF3kWLJukfwG1A1RGvaBW1jfTPTkDBW6))(voting_for 0)(timing Untimed)(permissions((edit_state Signature)(access None)(send Signature)(receive None)(set_delegate Signature)(set_permissions Signature)(set_verification_key(Signature 2))(set_zkapp_uri Signature)(edit_action_state Signature)(set_token_symbol Signature)(increment_nonce Signature)(set_voting_for Signature)(set_timing Signature)))(zkapp())))(Hash 21996930768807861074810719227764672824448546951836655211029805515889521212936)))))(Hash 24382372086420606104749512286799767579781746907618372545279197195252924864013))(Hash 18762262597383229583313511039136346166637978635531308926444084704352893594719)))(Hash 14224038739115341779391867001076313053758091506213674708305308561127749196228))(Hash 14254480585922701628390603951969073939500771039554209331307391432259158881699))(Hash 12110275148567042542889589593901715185558097672934686002710990315883307589103)))(Hash 12131748105144610707838828106718351445432476122769011051556648145735105158738)))(Hash 3842176760837909683996947035617933604392831838938428904682068141542051523726))(Hash 7320035858491162888661779985899102124712753554398750178339971990640402787131))(Hash 20511975586073725159467276923276440696276150672494963380552445184831766361203)))(Hash 18658226864331329939082125381927751546172213525751576765355057657697199681303))(Hash 16918038186554675056000578097630897159236284751433663183439602696852905624415))(Hash 28627628212319805343454811139849901640500073315285842033614442274505756385536))(Hash 28758859610334920473530372700694516882691452974475767468086895285359096307274))(Hash 16430190523938219721147185711044261861887815192145019235086508794037748662978))(Hash 23300248870286894992003584685086043578048220924304538552357941351343423298077))(Hash 18721105361232651344875520416758751524020336778169872574984370086949138642085))(Hash 10611828583016686420873473112404506690775956092186634604097134580560948638514))(Hash 11448813184116204118725235537673043143764404790624366392625860187638781235615))(Hash 6507258883578376028376981545955062827139698232277887871775971553968101691249))(Hash 17493244980280937955941580434326677459751473714746227943153210535373918340114))(Hash 18464507392887829891264190376236139806757048001410567864732128380527868369455))(Hash 374677701015397178366310913911051069892586440922808798004746377118603064136))(Hash 7414597540944309778303590613882007538175921122373418819351437768289067050688))(Hash 4911425897386868865951704774190352922917918118795543476097437449334578129718))(Hash 2537948008614359414287158135558328051431310188498471600040371216889620767874))(Hash 11945594868601920248817415866479228952991063900530321216805456660202303767865))(Hash 7727454798157491451398259626232336317494512806516291520841511378379541110711)))))(second_pass_ledger((indexes(((B62qo28fRrLkdn4z14Rtvi2tpop3sLjMdnoTSp87XMSe87ZjHXFpN8G 0x0000000000000000000000000000000000000000000000000000000000000001)70734)((B62qmPc8Ziq7txW48YPf4qtavcD5mcQVjEAGo9LEZD8DeaNsNthYLsz 0x0000000000000000000000000000000000000000000000000000000000000001)23327)))(depth 35)(tree(Node 18316015679542381448767982029391209763708740699521686920950643568637956377259(Node 19301094463871395919891076247397129834307021306183864205256314851109081691746(Node 26736084314587316236418106356176056219469299420338281532731019428461026746191(Node 24476548929424803514918668845195810346177353124979075997637694433678540056469(Node 23649018806241740546987128607293014578809076773586682207194415264377890980140(Node 28641538808805401306429870713201227145525118451270919530423141374763735354839(Node 7125780272760695094256929585052351735624750204931593965833484740971458157776(Node 25107909570953668724119279167205243513951520000589639050667700829204679609273(Node 16569524826291027173894285814647179270405455732189676083719580315968449400651(Node 24360537268896132838636904526747695538385587585117488255170947353010565172164(Node 1460225627014809094559109062275753989757288558419404819447135919902088293759(Node 305512331863691097533322975396063590981460694581976253093858445071578531301(Node 18187216999760394199464279367267233210572902876536866777893477580280623921612(Node 21866760338071040445858358713584047028350850218253876040150132857230438111116(Node 18466939050842561289660176179768374800250161096741711540578185960496964196130(Node 1127760829098160220732948857268960246317627571466711879839573216109758045184(Node 14977524400223158026055557063085438118070890083559759560412369920783137508902(Node 8372759950244221834618616559323635682136194315735990329470012690602953678527(Node 8815184494157801533403742314657366787105363694639841967817837293310620393171(Node 5296110203289983072306991384356856376298816189872661622021044084729964782206(Node 5462356288626380051606402212832611984046470144890902972225137384149243789450(Hash 2058674651288216913540730131669809459531694003047904999110615467110015914430)(Node 18252505871430492579846795626614216747790723643062880855819685145652100200573(Node 5821280192106817147275118739737753598064803258731704279171995224285185620284(Hash 27553577135916178008042195952925360465943520255637590064404300754667219238259)(Node 2897146731037187929231268335277774027740723053669253806296299150317142222302(Hash 22978718094957128311769450305910814147951674289916820203247065143011111914178)(Node 18520655258964976306500504357441930017151511895141452245302651821147164548852(Node 5102356789423517404288517071767466985111717899942461702268938174328144658109(Hash 14899042638274887117136371219137258142930064486673869080677616691248414616648)(Node 12830349183744532048610739737679326710930663302680125453484746321528715351145(Hash 4215282699246713217037377918533153073854288070800508400360289256617598886316)(Node 5082650515416437143460206111192576206671744285200271115301850119520585599569(Node 16066044455671416227209541068185822718873185674365452210845508737260190033233(Node 12856827425426488291651282140698416249344970636021246924722772261886345068583(Node 14886800063113699708926974955088412486409400893363680313128610940069925668528(Hash 23695332766093071154639712073849298059152258656200270134875859473724785946606)(Node 9356003193471855169138915765683265704934627518783322246951887417228792596550(Hash 16818864594653795750483594145484500530547799933695170324955063888951381345329)(Node 12479653240641731089449776756311587652605918199864298962026019904697170545737(Hash 26325152942652394623913485608126868359581505358067590149442314590684665654014)(Node 21517709540063175921204536429930381441067559015735716152086723450561837725984(Hash 1672251043519471094809642303791305637378627066391961277420668522602136460606)(Node 19199660983841467270154834756427609880659244954650568841429508700018950524391(Hash 18084489660654256092123642794019593206145259614177986389211515824370883116410)(Account((public_key B62qmPc8Ziq7txW48YPf4qtavcD5mcQVjEAGo9LEZD8DeaNsNthYLsz)(token_id 0x0000000000000000000000000000000000000000000000000000000000000001)(token_symbol\"\")(balance 398912121897105)(nonce 589949)(receipt_chain_hash 4285649566248937594173715983353674443340403272438945788826717919428250711396)(delegate(B62qq3TQ8AP7MFYPVtMx5tZGF3kWLJukfwG1A1RGvaBW1jfTPTkDBW6))(voting_for 0)(timing Untimed)(permissions((edit_state Signature)(access None)(send Signature)(receive None)(set_delegate Signature)(set_permissions Signature)(set_verification_key(Signature 2))(set_zkapp_uri Signature)(edit_action_state Signature)(set_token_symbol Signature)(increment_nonce Signature)(set_voting_for Signature)(set_timing Signature)))(zkapp()))))))))(Hash 14535326754444330769253632848688085426020079440611720935404738894148852020534))(Hash 4261258321617343100879041861929932910306827007384438561383824085360521405176))(Hash 9159231553430092791912033560640189858933046131408391789348846863071862232531))))(Hash 19480264859030129956909669973333920189195012519501437337557487202296617950700))))(Hash 3352669251860896491646515371885097392309067770502712469689504162094571208778)))(Hash 2611148510681006771889919288378612970799587864183424401622084763881537316446))(Node 4562209947552993466440410485973687073329038101924532736463700587587239802484(Node 11171951286823206949638017054366714796646508037796783656599209691603251718709(Node 9771637314978003144667846382293765225216112679602150392105103338640670973740(Node 11132141435326830590267921160520624591599258591927036500795680731075040757948(Hash 6884246501993332621537983928870193524404266302369585203513875155889666815689)(Node 28196661259162071019320466867133933875909118689133914795371032512768650876526(Node 5784539071505376583203521018126631785324572645083373068365990978966430878251(Hash 27063384030603089987018917689034604475635173109190182354311324851332695958280)(Node 27789769415118207517066829763244876375852272619922981877456067772063241136566(Node 20902515337585513521770521496295206192825668916818669278341346318512201213878(Node 8668709236306929094819638623875099074296403601875994587176782837591245944142(Node 6469082402908866011401134431776109500895565365508003594852303700744978082533(Hash 28352207452189453189353545832932382907291889718397244734960756295298048940249)(Node 2428222343535777391895541328504925975634107739658007840182717623925146268961(Node 11745036219283532980465734764541201709658976167405130099580285593443679528093(Node 4501936566791854063191420973789949785989337229876217381270204128296093164688(Hash 18279440942993662136385793760814234117901749881152069179880102155545361950211)(Node 5778379158827563763633586314481476602630214971633891420653109193314489102136(Hash 21353556399778004327977345313711978313156505408608475883319723438597365429078)(Node 26695736579971701651839911612071530807156909254640087386795935592823170255650(Hash 17809711804492984618666802407034141827696293269501867085665203717000010976406)(Node 18949356327031078711579728370331566058022771931951551611064325315123486294943(Account((public_key B62qo28fRrLkdn4z14Rtvi2tpop3sLjMdnoTSp87XMSe87ZjHXFpN8G)(token_id 0x0000000000000000000000000000000000000000000000000000000000000001)(token_symbol\"\")(balance 31545691385)(nonce 2)(receipt_chain_hash 23734093168081738593308871271268144933218802735365964038842898715591237181064)(delegate(B62qq3TQ8AP7MFYPVtMx5tZGF3kWLJukfwG1A1RGvaBW1jfTPTkDBW6))(voting_for 0)(timing Untimed)(permissions((edit_state Signature)(access None)(send Signature)(receive None)(set_delegate Signature)(set_permissions Signature)(set_verification_key(Signature 2))(set_zkapp_uri Signature)(edit_action_state Signature)(set_token_symbol Signature)(increment_nonce Signature)(set_voting_for Signature)(set_timing Signature)))(zkapp())))(Hash 21996930768807861074810719227764672824448546951836655211029805515889521212936)))))(Hash 24382372086420606104749512286799767579781746907618372545279197195252924864013))(Hash 18762262597383229583313511039136346166637978635531308926444084704352893594719)))(Hash 14224038739115341779391867001076313053758091506213674708305308561127749196228))(Hash 14254480585922701628390603951969073939500771039554209331307391432259158881699))(Hash 12110275148567042542889589593901715185558097672934686002710990315883307589103)))(Hash 12131748105144610707838828106718351445432476122769011051556648145735105158738)))(Hash 11039029092715754259265938924236709578778415959683684162056922913263184535703))(Hash 7320035858491162888661779985899102124712753554398750178339971990640402787131))(Hash 10260664064756411785195119332322902700763594167852118893906655461838902457895)))(Hash 26340242493929579439649470450352541512475234935042576108451892014299913870748))(Hash 16918038186554675056000578097630897159236284751433663183439602696852905624415))(Hash 28627628212319805343454811139849901640500073315285842033614442274505756385536))(Hash 28758859610334920473530372700694516882691452974475767468086895285359096307274))(Hash 16430190523938219721147185711044261861887815192145019235086508794037748662978))(Hash 23300248870286894992003584685086043578048220924304538552357941351343423298077))(Hash 18721105361232651344875520416758751524020336778169872574984370086949138642085))(Hash 10611828583016686420873473112404506690775956092186634604097134580560948638514))(Hash 11448813184116204118725235537673043143764404790624366392625860187638781235615))(Hash 6507258883578376028376981545955062827139698232277887871775971553968101691249))(Hash 17493244980280937955941580434326677459751473714746227943153210535373918340114))(Hash 18464507392887829891264190376236139806757048001410567864732128380527868369455))(Hash 374677701015397178366310913911051069892586440922808798004746377118603064136))(Hash 7414597540944309778303590613882007538175921122373418819351437768289067050688))(Hash 4911425897386868865951704774190352922917918118795543476097437449334578129718))(Hash 2537948008614359414287158135558328051431310188498471600040371216889620767874))(Hash 11945594868601920248817415866479228952991063900530321216805456660202303767865))(Hash 7727454798157491451398259626232336317494512806516291520841511378379541110711)))))(protocol_state_body((genesis_state_hash 23979920091195673795386525806121605315652663595695491169052082412294004666370)(blockchain_state((staged_ledger_hash((non_snark((ledger_hash 24916416925117821545353053321539334910728808378795587946890647990902157490944)(aux_hash\"\\180O/\\195\\015|J\\030%\\239\\147\\230\\001\\176\\163\\149D\\253&%\\174\\200\\135CB\\136\\229\\2502\\176\\000o\")(pending_coinbase_aux\"6\\001\\157X\\226M\\200\\221\\158ohz9\\254J\\157S\\006od\\230\\158X\\202mJt\\206\\014\\169\\020\\252\")))(pending_coinbase_hash 3787826106010502851973333482523143647593991524883096114583468860463649431404)))(genesis_ledger_hash 12413918321261948093075717299811383170543964450947772018040699593761744163621)(ledger_proof_statement((source((first_pass_ledger 19962008427452077250704318669373944486712501776019264437857808148135150271735)(second_pass_ledger 21588028521116667546192170826295540433806757525772269644292008112780133700029)(pending_coinbase_stack((data 13478948633790621346997153068092516261975764161208078295837519850718904039733)(state((init 14920167974010284033041337397061778130938334350141806113860588160704327879896)(curr 14920167974010284033041337397061778130938334350141806113860588160704327879896)))))(local_state((stack_frame 0x0641662E94D68EC970D0AFC059D02729BBF4A2CD88C548CCD9FB1E26E570C66C)(call_stack 0x0000000000000000000000000000000000000000000000000000000000000000)(transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(full_transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(excess((magnitude 0)(sgn Pos)))(supply_increase((magnitude 0)(sgn Pos)))(ledger 0)(success true)(account_update_index 0)(failure_status_tbl())(will_succeed true)))))(target((first_pass_ledger 7305140938659159611687770387707938248903933671563137265859965660963974414370)(second_pass_ledger 24199875689016803830721837642214061170747301604917620456437155049436990949514)(pending_coinbase_stack((data 4065160939727929136815980054181479730254127167892904199669098124358689814619)(state((init 14920167974010284033041337397061778130938334350141806113860588160704327879896)(curr 158090936961392235656491792025400101322487951138763310262619177391390222875)))))(local_state((stack_frame 0x0641662E94D68EC970D0AFC059D02729BBF4A2CD88C548CCD9FB1E26E570C66C)(call_stack 0x0000000000000000000000000000000000000000000000000000000000000000)(transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(full_transaction_commitment 0x0000000000000000000000000000000000000000000000000000000000000000)(excess((magnitude 0)(sgn Pos)))(supply_increase((magnitude 0)(sgn Pos)))(ledger 0)(success true)(account_update_index 0)(failure_status_tbl())(will_succeed true)))))(connecting_ledger_left 21588028521116667546192170826295540433806757525772269644292008112780133700029)(connecting_ledger_right 24199875689016803830721837642214061170747301604917620456437155049436990949514)(supply_increase((magnitude 720000000000)(sgn Pos)))(fee_excess((fee_token_l 0x0000000000000000000000000000000000000000000000000000000000000001)(fee_excess_l((magnitude 0)(sgn Pos)))(fee_token_r 0x0000000000000000000000000000000000000000000000000000000000000001)(fee_excess_r((magnitude 0)(sgn Pos)))))(sok_digest())))(timestamp 1717769340000)(body_reference\"\\255\\249\\178A\\210w\\151\\1920\\209A\\227\\211\\137\\183^\\224\\137'B\\200\\005\\162,\\132qz_\\224\\206\\244,\")))(consensus_state((blockchain_length 360324)(epoch_count 0)(min_window_density 77)(sub_window_densities(4 3 3 6 5 2 5 2 3 4 4))(last_vrf_output\"A]!D!\\225\\002\\149\\2545Vv\\136\\142\\195\\168{\\218\\211\\r\\139O\\207\\178\\137\\152\\246a\\193\\025\\002\\000\")(total_currency 1166529845840039233)(curr_global_slot_since_hard_fork((slot_number 1243)(slots_per_epoch 7140)))(global_slot_since_genesis 565723)(staking_epoch_data((ledger((hash 25431058814419771714686537214404403273571190473338438022204105901709705798376)(total_currency 1160334549840039233)))(seed 7191167798072417993396333679872646868560249781785384306800555326929635458378)(start_checkpoint 0)(lock_checkpoint 0)(epoch_length 1)))(next_epoch_data((ledger((hash 28196617333454140620749984326461012661939979311141486486800050252918189790798)(total_currency 1165736008840039233)))(seed 17542468023974611502814222213454160502904435958791408928176121637379598407580)(start_checkpoint 0)(lock_checkpoint 9974608775939045007838385825535614698337536747926469836237002756656535867215)(epoch_length 721)))(has_ancestor_in_same_checkpoint_window true)(block_stake_winner B62qkYUUe89b1uSCx6RMEwQJemutGWTKhtsZGCCfS4Noic94jkqft6U)(block_creator B62qpge4uMq4Vv5Rvc8Gw9qSquUYd6xoW1pz7HQkMSHm6h1o7pvLPAN)(coinbase_receiver B62qospDjUj43x2yMKiNehojWWRUsE1wpdUDVpfxH8V3n5Y1QgJKFfw)(supercharge_coinbase true)))(constants((k 290)(slots_per_epoch 7140)(slots_per_sub_window 7)(grace_period_slots 2160)(delta 0)(genesis_state_timestamp 1717545600000)))))(init_stack((data 17565849295236487117005698014466407944766819630903075312334386758220362116827)(state((init 20073240908364842602708354313591733737302929896452351308022517722215162205924)(curr 14785813412484954199694713920038880861741329799285332709067474088350791489331)))))(status Applied)(block_global_slot 565725)))"}}
{"timestamp":"2024-06-07 14:38:10.194731Z","level":"Error","source":{"module":"Snark_worker__Functor","location":"File \"src/lib/snark_worker/functor.ml\", line 250, characters 8-20"},"message":"Error performing work: (monitor.ml.Error\\n (Failure\\n   \"Constraint unsatisfied (unreduced):\\\\n  \\n\\\\n  \\nFile \\\"src/lib/pickles/wrap_main.ml\\\", line 387, characters 23-30\\\\n  \\nFile \\\"src/lib/pickles/wrap_main.ml\\\", line 163, characters 15-22\\\\n  \\n\\\\n  \\nConstraint:\\\\n  \\n((basic(Square(Var 32338)(Add(Var 32384)(Constant 5))))(annotation()))\\\\n  \\nData:\\","metadata":{"process":"Snark Worker"}}
CryptoTech-Guy commented 1 week ago 
{"timestamp":"2024-06-21 12:36:54.288744Z","level":"Error","source":{"module":"Snark_worker__Functor","location":"File \"src/lib/snark_worker/functor.ml\", line 247, characters 8-20"},"message":"Error performing work: (monitor.ml.Error\n (Failure\n   \"Constraint unsatisfied (unreduced):\\\n  \\n\\\n  \\nFile \\\"src/lib/pickles/wrap_main.ml\\\", line 163, characters 15-22\\\n  \\n\\\n  \\nConstraint:\\\n  \\n((basic(Square(Var 32196)(Add(Var 32266)(Constant 5))))(annotation()))\\\n  \\nData:\\\n  \\nSquare 16906482924653793927536364603695301351486215707822952978132240975189951891446 5025303310426993405745505323125469870711387606385066393004629422983355678793\")\n (\"Raised at Stdlib.failwith in file \\\"stdlib.ml\\\", line 29, characters 17-33\"\n  \"Called from Snarky_backendless__Checked_runner.Make_checked.add_constraint.(fun) in file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 216, characters 12-399\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.run in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 748, characters 26-51\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.as_stateful in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 754, characters 14-18\"\n  \"Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 64, characters 25-28\"\n  \"Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 64, characters 25-28\"\n  \"Called from Snarky_backendless__Checked_runner.Simple.bind.(fun) in file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 64, characters 25-28\"\n  \"Called from Snarky_backendless__Checked_runner.Make_checked.exists.(fun) in file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 277, characters 22-47\"\n  \"Called from Snarky_backendless__Checked_runner.Simple.map.(fun) in file \\\"src/lib/snarky/src/base/checked_runner.ml\\\", line 54, characters 29-32\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.run in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 748, characters 26-51\"\n  \"Called from Pickles__Wrap_main.wrap_main.main.(fun) in file \\\"src/lib/pickles/wrap_main.ml\\\", line 359, characters 10-1023\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.with_label in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 1260, characters 14-18\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.handle in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 1244, characters 14-18\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.mark_active in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 1174, characters 18-22\"\n  \"Re-raised at Snarky_backendless__Snark0.Run.Make_basic.mark_active in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 1179, characters 8-17\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.as_stateful in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 754, characters 14-18\"\n  \"Called from Snarky_backendless__Runners.Make.auxiliary_input.(fun) in file \\\"src/lib/snarky/src/base/runners.ml\\\", line 82, characters 21-33\"\n  \"Called from Snarky_backendless__Runners.Make.Run.generate_witness_conv.(fun) in file \\\"src/lib/snarky/src/base/runners.ml\\\", line 337, characters 12-98\"\n  \"Called from Snarky_backendless__Snark0.Run.Make_basic.finalize_is_running in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 1279, characters 14-18\"\n  \"Re-raised at Snarky_backendless__Snark0.Run.Make_basic.finalize_is_running in file \\\"src/lib/snarky/src/base/snark0.ml\\\", line 1285, characters 12-19\"\n  \"Called from Pickles__Wrap.wrap.(fun) in file \\\"src/lib/pickles/wrap.ml\\\", line 863, characters 4-1023\"\n  \"Called from Pickles__Compile.Make.compile.f.wrap.(fun) in file \\\"src/lib/pickles/compile.ml\\\", line 813, characters 12-448\"\n  \"Called from Async_kernel__Deferred0.bind.(fun) in file \\\"src/deferred0.ml\\\", line 54, characters 64-69\"\n  \"Called from Async_kernel__Job_queue.run_job in file \\\"src/job_queue.ml\\\" (inlined), line 128, characters 2-5\"\n  \"Called from Async_kernel__Job_queue.run_jobs in file \\\"src/job_queue.ml\\\", line 169, characters 6-47\"\n  \"Caught by monitor at file \\\"src/lib/snark_worker/prod.ml\\\", line 262, characters 59-59\"))","metadata":{"process":"Snark Worker"}}
rbonichon commented 1 week ago

@CryptoTech-Guy Thanks for your report(s) and patience. So far we haven't been able to replicate your issue. We could even prove the data related to the first bug report. Thus, so far, it has been quite head-scratching to understand what's happening.

Besides, this error doesn't seem related to the snark errors your other logs are reporting.