Exploring an Enhanced Contract Storage System (Perfs)

[moul]

Background:

Our current contract execution process works well, but we believe there's room for performance improvement, particularly in storage. The current key-value storage model, which combines RAM and Disk with lazy loading, presents potential opportunities for enhancement.

It's important to clarify that we're not proposing to replace how we store blocks; we're focusing on optimizing the representation of the state, which can be reconstructed by replaying the blocks. Think of this storage as a cache, and it's worth considering customizing it. In fact, there could even be competing strategies, as long as they maintain determinism and block compatibility.

Explorations:

Overlay File Systems: One intriguing idea is to explore file systems that operate similarly to overlay systems, like aufs, overlayfs, zfs, or btrfs. In this model, each new commit generates a new overlay with a Merkle proof. Eventually, we would only need to inspect the current state, taking advantage of kernel and file system features for caching and optimization.

Potential Performance Enhancements:

• Batch Loading: Fetching related data in larger groups to reduce retrieval times.
• Data Compression: Using efficient compression methods to save space and speed up operations.
• Smart Caching: Storing frequently used data in memory for faster access
• Parallel Processing: Handling multiple data operations simultaneously for faster access and updates.
• Data Partitioning: Splitting data into manageable parts for more balanced and efficient storage.
• Optimized Indexing: Fine-tuning database indexing to speed up data retrieval, especially in large datasets.

Share Your Insights:

If you have experience with Flamegraphs (I'll generate and share myself later it not yet done) or similar profiling tools, please share your results. Your insights can help us identify areas for potential improvement.

[ajnavarro]

I would add to the list of potential performance enhancements data linear read. This process should be I/O bounded, and to make sure we can get the most significant performance gain, we could store data on disk in a way that we usually read it linearly, not randomly. I did some research about that and how to keep trees on disk to boost performance on writes and reads.

Edit: some storage experiment I created for that specific use case in the past: https://github.com/ajnavarro/super-blockstore/