Script to pre-generated genesis ledger from a list of accounts and reference that in the config file

[deepthiskumar]

Config file passed as daemon argument consists of genesis ledger as a list of accounts. These accounts can grow (mainnet now has over 200k accounts). This can take up significant boot-up time during which the daemon is unresponsive. The config file has options to take the merkle root or a name of the ledger instead. With these options, the daemon will look for files with a specific name locally as well as in an s3 bucket that's currently hardcoded.

The task for this issue is to test the two functionalities (using Hash and Name) in runtime config. The ledger can be generated by the daemon using a list of accounts first. Or preferably, one could create an app to do ledger generation using account list (just extracting the same piece of code the daemon uses) instead of running the daemon. I believe, https://github.com/MinaProtocol/mina/blob/berkeley/src/app/runtime_genesis_ledger/runtime_genesis_ledger.ml does this but needs to be confirmed.

Update: Script: takes config file with ledgers (incluuding both the staking ledgers) with account list as input and generate as output the corresponding tars and a config file that refers to these files (instead of list of accounts)

[michal0mina]

@deepthiskumar when you are back -- can you add a bit more details here?

[joaosreis]

I've looked into this. My findings are the following:

• We can use src/app/runtime_genesis_ledger/runtime_genesis_ledger.ml to produce the tar file from a config JSON with accounts.
• We can check the utility log for the Merkle root hash and use it in a config file instead of specifying the accounts. Example:

  {
  "genesis": {
  "genesis_state_timestamp": "2023-10-17T16:01:01Z"
  },
  "ledger": {
  "hash": "3NKm9dYH5TtyjQXccBBD7WQNezQnNuShCU6396CrpvwGeh7udhos"
  }
  }

• Using this config, the node will search locally for the tar file corresponding to that hash (the one generated by the utility/node). If the file is not found locally, it will search for it in the S3 bucket.
• I tested loading the genesis ledger from an existing local file on an ITN node, working as expected. I didn't test if loading from S3 works as I would need access to the S3 bucket to send the test tar file.
• Testing with a genesis ledger with over 200k accounts, it took approximately 0.5s to load the genesis ledger from the tar file (based on the timestamp diff between the log messages stating that the ledger file was found and that the ledger was loaded from the file. The tar file itself has a size of ~58M.

Let me know if you need me to look into something else or give more details. @deepthiskumar @michal0mina

[joaosreis]

I now have a script to automate the process of using a config file with the genesis ledger accounts as well as both staking ledger accounts and producing the tar files for each of these ledgers and a new config file referencing the Merkle root hash of these ledgers instead of the accounts.

However, I noticed that the runtime config doesn't allow specifying the hash of the ledger for the staking ledgers, it only allows specifying accounts, so this needs to be changed. The logic used to load the staking ledger is the same as the genesis one, so this should be simply a matter of adding the possibility of specifying the hash on both staking ledgers in the config file.

Today I will create a PR with both of these changes.