Expose pending snark work query and proof submission in CLI/GraphQL

[deepthiskumar]

PR:

• https://github.com/MinaProtocol/mina/pull/16244
• https://github.com/MinaProtocol/mina/pull/16260

[svv232]

This should already be possible via the "pendingSnarkWork" graphql end point

https://github.com/MinaProtocol/mina/blob/develop/src/lib/mina_graphql/mina_graphql.ml#L2201

[svv232]

I think the original idea for this was to query for specific ranges of work.

[deepthiskumar]

The changes are intended to decouple daemon and snark worker process to allow for custom setups.

Snark worker:

• Stand alone snark worker process that can be spawned when required
• Takes statement, witness, prover fees and prover public key as arguments
• Connects to a mina daemon node to submit generated proof

Snark coordinator/Daemon:

• Graphql command to retrieve pending snark work (includes proofs in the pool with fee, in the case prover wants to ignore or outbid)
• GraphQL command to accept generated proofs

External service/script (To be implemented by node operators)

• Queries pending work from a node
• Determines the order of execution
• Spawns stand alone snark work by passing the required arguments
• Maintain progress state to re-request proof in case of failures

[jrwashburn]

Will the current coordinator, built into the daemon, and the current worker (mina internal snark-worker) still operate as-is?

I like that this would allow for third party coordinators to be built, I don't like that this sounds like it will be required to have any distribution of work. I also don't like the closeness implied by the proposal (coordinator service spawns a snark worker.)

I think the persistent workers that can register with / get work from a coordinator is a better way to manage distributed work, vs. spawning a worker for a specific job. There should be a published/supported protocol from worker to coordinator that can be implemented by anyone.

The value of a coordinator is to coordinate work across many independent, distributed machines - we should not lose that capability, or need to invent a new protocol and add a shim on the remote end to spawn workers one-by-one. I don't think we want to deal with connection setup delays or add extra work to the third party coordinator to also have to deal with creating their own protocol and method for distributing work.

I think there should be a clearly defined protocol from daemon to coordinator and from coordinator to worker. It should be supported by both ocaml and rust nodes, and can be independently implemented in private or open source coordinators alongside the "base" implementation that daemon developers ship.

[deepthiskumar]

@jrwashburn the current coordinator-worker setup will remain as is. This is to expose necessary tools/endpoints for a custom setup

[deepthiskumar]

There should be a published/supported protocol from worker to coordinator that can be implemented by anyone.

I think this solution is trying to achieve this to some extent? a new graphql endpoint that returns a sequence of work that is required by BPs in the same order that required to add transactions, another endpoint that accepts snark work and then standalone snark worker process that consumes the spec and produces snarks. The main dependency on coordinator is to get the sequence of work that BPs will need and this can be requested using the proposed graphql endpoint.