search

cartesi / machine-emulator

The off-chain implementation of the Cartesi Machine
GNU Lesser General Public License v3.0
67 stars 34 forks source link

Optimize root hash computation while loading and saving snapshots #258

Open edubart opened 4 months ago

edubart commented 4 months ago

Context

The emulator is starting to be used to create "reader nodes" in production for some dapps where it advances and saves a machine snapshot, then when an inspect comes the saved machine snapshot is loaded and an inspect request is served. The problem is that saving/loading could take minutes (for example when running in a cheap cloud service with just 2 CPU cores and 4GB of ram).

We could optimize all this loading/saving to be faster. The culprit here is root hash computation, most of time is spent there.

Possible solutions

One solution is to save the entire merkle tree state to a file in the snapshot, so not all merkle nodes have to be recomputed after loading. And also save checksum (like sha1sum) of all snapshot files to not lose integrity check, it should be faster to check a sha1sum of a few files rather than a merkle tree hash. Maybe even CRC64 could be enough, I think we don't need a very secure and heavy hash to just check snapshot file integrity.

edubart commented 4 months ago

This problem is also becoming more relevant when loading snapshots in the validator node on cheap cloud services that have just 1 CPU and 1GB ram, to the point loading a snapshot takes so long that the validator is unable to initialize properly due to timeouts and advances/inpects doesn't work.