[Testnet] Deploy prover server on cloud

[thomas-nguy]

Investigate how to build and run the prover server

Reference https://docs.google.com/document/d/1xoB126DpqH0DTZU16FAEiKQvfk_qo6Rha74lsDnKqIA/edit

The document explain how to run it but it does not seems to be connected to the zk server

[thomas-nguy]

Note: Need to figure out how does the prover server generate proof from witnesses

it seems that the prover server is not included in the local setup https://github.com/matter-labs/local-setup/blob/main/docker-compose.yml

[JayT106]

We will need the server to run a witness generator for generating prover jobs, it will require the user to do export ZKSYNC_LOCAL_SETUP=false to launch it. The comment says it suggests running it without the node. https://github.com/matter-labs/zksync-era/blob/main/core/bin/zksync_core/src/lib.rs#L400

So my might run it solely with zk server --components witness_generator

[JayT106]

For the prover, from the default config settings we can see they only let the CPU prover runs 2 types of circuits (total is 19), guessing to run a circuit is a pretty heavy operation. Therefore in the config file, it divides the prover into several groups. If we want to try to run it with one CPU prover, we could simply set
PROVER_GROUP_GROUP_0_CIRCUIT_IDS=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18 and PROVER_NON_GPU_NUMBER_OF_SETUP_SLOTS=19 But it will need huge RAM to be able to run it.

the formula is comment in the code:

    // Number of setup-keys kept in memory without swapping
    // number_of_setup_slots = (R-C*A-4)/S
    // R is available ram
    // C is the number of parallel synth
    // A is the size of Assembly that is 12gb
    // S is the size of the Setup that is 20gb
    // constant 4 is for the data copy with GPU

So I guess we will need a 512RAM aws instance to be able to run a single CPU prover.

[thomas-nguy]

To be able to feed the prover with the assembly, we need to "circuit_synthetizer"

The flow is the following

• the "zk_server" (witness generator component) generates witnesses for each block and store in DB. Multiple witnesses are generated, corresponding to group ids.

• the "circuit_synthetizer" fetches the witness for specific group Id, synthetized it to assembly and send it to the "prover_server" for a specific group id

Note1: The "circuit_synthetizer" seems to fetch using gcp for now instead of DB (maybe a way to change the behavior through config)

Note2: We need to run multiple "circuit_synthetizer" and "prover_server" for each group ids

• the "prover_server" receives the assembly and generates the proof and save it in DB

• the "zk_server" picks the proof and send it to the contract

[JayT106]

to run circuit_synthesizer, we can use this command

zk f cargo +nightly run --release --bin zksync_circuit_synthesizer

[JayT106]

To be able to feed the prover with the assembly, we need to "circuit_synthetizer"

The flow is the following

• the "zk_server" (witness generator component) generates witnesses for each block and store in DB. Multiple witnesses are generated, corresponding to group ids.
• the "circuit_synthetizer" fetches the witness for specific group Id, synthetized it to assembly and send it to the "prover_server" for a specific group id

Note1: The "circuit_synthetizer" seems to fetch using gcp for now instead of DB (maybe a way to change the behavior through config)

I think the circuit_synthetizer fetches data from the ConnectionPool connected to the database_url, which the default PATH will be DATABASE_URL=postgres://postgres@localhost/zksync_local

Note2: We need to run multiple "circuit_synthetizer" and "prover_server" for each group ids

• the "prover_server" receives the assembly and generates the proof and save it in DB
• the "zk_server" picks the proof and send it to the contract

[thomas-nguy]

I think it uses two things

the DB for synchronizing the jobs (as you mention connectionPool uses database_url

and an object store which use the following config

OBJECT_STORE_BUCKET_BASE_URL=base_url
OBJECT_STORE_MODE=FileBacked
OBJECT_STORE_FILE_BACKED_BASE_PATH=artifacts
OBJECT_STORE_GCS_CREDENTIAL_FILE_PATH=/path/to/gcs_credentials.json
OBJECT_STORE_MAX_RETRIES=5

FileBacked means it is probably stored locally but since we are going to run the prover + synthetizer in another machine than the zkserver (+witness generator) it may not works

[JayT106]

I think it uses two things

the DB for synchronizing the jobs (as you mention connectionPool uses database_url

and an object store which use the following config

OBJECT_STORE_BUCKET_BASE_URL=base_url
OBJECT_STORE_MODE=FileBacked
OBJECT_STORE_FILE_BACKED_BASE_PATH=artifacts
OBJECT_STORE_GCS_CREDENTIAL_FILE_PATH=/path/to/gcs_credentials.json
OBJECT_STORE_MAX_RETRIES=5

FileBacked means it is probably stored locally but since we are going to run the prover + synthetizer in another machine than the zkserver (+witness generator) it may not works

We may need to implement the objectStore for Amazon S3 if we stick to Amazon cloud infra. https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/rust_dev_preview/s3#code-examples

[thomas-nguy]

we can add the implementation of aws object store there

https://github.com/matter-labs/zksync-era/tree/main/core/lib/object_store/src

It has to impl the interface ObjectStore

[JayT106]

working on S3 ObjectStore implementation https://github.com/cronos-labs/zksync-era/tree/object-store-s3-wip

[JayT106]

Will deploy new prover instead of the current prover setup https://github.com/cronos-labs/zksync-era/issues/27