Add alternative method to prove txs without pre-loaded table circuits

[Nashtare]

This PR:

• Allows to write/load only upper circuits when serializing/deserializing the prover state (i.e. by skipping all the STARK table circuits).
• Adds an alternative method to prove txns (prove_root_light()) by passing an additional path to a folder containing all different table circuits. After generating the initial STARK, the prover will read only the necessary circuit files, deserializes them and process to recursion. The files are expected to be stored at initial_path/TableName_size.ser.
• Also removes the expand methods, which were only added previously for easier testing but ended up never being really useful.

cc @cpubot: For zero-bin / eth-tx-proof, we would:

• still be generating the initial prover state as usual.
• instead of writing the entire state in a single file, we would write the upper circuits to a single file, which would be the default prover state (and sufficient for aggreg/block proofs).
• for each individual table, write them to a dedicated file within the same folder, with the format above.
• pass the serializers and the folder containing serialized tables when calling generate_txn_proof, with new signature below:

Important note: when writing the individual tables to file, we MUST prepend the byte payload by a byte indicating the position of this table in the initial range specified when constructing the prover state. Say we specified ranges 16..28 for the Arithmetic table, we should write first 3 in the file Arithmetic_19.ser before writing the actual serialized circuit. This is necessary to be able to prove transactions recursively.

pub fn generate_txn_proof(
    p_state: &ProverState,
    preprocessed_tables_path: &Path,
    start_info: TxnProofGenIR,
    gate_serializer: &dyn GateSerializer<GoldilocksField, 2>,
    generator_serializer: &dyn WitnessGeneratorSerializer<GoldilocksField, 2>,
    abort_signal: Option<Arc<AtomicBool>>,
) -> ProofGenResult<GeneratedTxnProof> {

Please let me know if there's anything that you'd like changed. I can also help with the integration in the decoder and zero-bin.

closes #1394

[cpubot]

I love the overall concept! However, I'm not particularly fond of coupling the innards of plonky2 to performing disk IO and serialization. In other words, I think it would be cleaner to expose an API that is agnostic to the underlying methodology of yielding the pre-processed tables. What do you think about adjusting the API such that it expects a reference to the tables, rather than being given a path and a pair of serializers?

[Nashtare]

@cpubot I wasn't super fond of this either, but not sure on how to handle it best. With your suggestion, we would then have the logic related to disk IO and serialization in the decoder?

Note that a reference to the tables isn't possible without computing the initial STARK proof, which will tell us which preprocessed tables we need

[cpubot]

@cpubot I wasn't super fond of this either, but not sure on how to handle it best. With your suggestion, we would then have the logic related to disk IO and serialization in the decoder?

I think it'd be preferable to have these steps performed in the layer that is composing the various proving functions — the logic that is calling the protocol decoder and plonky2 — which would be zero-bin in this case.

[sonarcloud[bot]]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[Nashtare]

I've updated the logic, let me know if that looks better now.

As for

which would be zero-bin in this case.

I think my concern was that we now have to to txn proofs in three steps, generate the initial STARK proof, load the necessary circuits for this one, and use the new method to finish (prove_root_after_initial_stark in my latest commit). This was initially wrapped and handled by plonky-block-proof-gen (proof-protocol-decoder now) hence why I'm not sure I see how this would cleanly be done within zero-bin, but I guess that's not a big issue.