feat: iavl v2

[kocubinski]

See also: https://hackmd.io/rXq7coSyTFmhWIa4mifBYQ

For a complete end to end test, this version of code is running on an osmosis node syncing on mainnet, some metrics are viewable at this Grafana. Sync started from block 13946655 after an export of application.db to an iavl v2 SQLite database with the code in ./migrate in this PR.

Sync rates as fast as ~10 blocks per second are observed, most of the spikiness in commit rate is spent waiting on blocks from consensus, but I don't have a way to show this in telemetry. AFAIK this is about 2x from iavl v1 with osmosis, I don't see evidence that the osmosis build is I/O bound on disk either as benchmarks put max throughput 10-20x iavl v1.

This configuration is using 45G of disk storage, down from 177G. Checkpoints are taken every 5k blocks, with a WAL maintained on each commit. Pruning occurs every 900 blocks with the last 5k blocks kept, therefore the last checkpoint always contains a fully materialized tree.

[tac0turtle]

is this ready for review?

[coderabbitai[bot]]

Walkthrough

The update enhances the iavl package with significant improvements in version control, database management, and performance optimization. It introduces new functionalities such as command-line tools for database operations, integration with SQLite for snapshot management, advanced metrics tracking, and enhanced tree manipulation capabilities. These changes aim to provide more efficient and robust tools for managing tree structures and their data, emphasizing scalability and user experience.

Changes

File(s)	Change Summary
v2/.gitignore	Excludes vendor files, test databases, profiling data, IDE-specific files, and Go workspace files.
v2/.golangci.yml	Introduces configuration for tests, linters, and issue reporting limits.
v2/cmd/... (multiple files)	Adds CLI functionality for database operations like data generation, snapshots, rollback, and scanning.
v2/export.go	Introduces functionality for exporting tree structures with traversal order options.
v2/internal/encoding.go, encoding_test.go	Adds encoding/decoding functionality for varint length-prefixed byte slices and related tests.
v2/iterator.go	Implements iterators for traversing tree structures and leaf nodes.
v2/metrics/metrics.go	Introduces metrics tracking for pools, trees, and database operations.
v2/migrate/... (multiple files)	Adds functionality for migrating database versions and managing readonly stores.
v2/multitree.go	Introduces MultiTree for managing multiple tree instances.
v2/node.go, v2/pool.go	Enhances node management within trees and introduces a node pool for memory management.
v2/range.go	Adds a VersionRange struct for managing and querying version ranges.
v2/sh/rollback.sh	Provides a bash script for database rollback operations.
v2/snapshot.go, v2/sqlite... (multiple files)	Introduces snapshot management and SQLite integration for efficient data storage and retrieval.
v2/testutil/util.go	Adds utility functions for configuring tree build options for testing.
v2/tree.go, v2/visualize.go	Enhances tree management and introduces visualization of tree structures.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): > Note: Auto-reply has been disabled for this repository by the repository owner. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged. - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger a review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

[cool-develope]

Rollback feature is missed (DeleteVersionsFrom in v1). @kocubinski, I think MultiTree has the potential advantages in terms of deferred pruning and also would be easy to manage the flow, for example resolving the conflicts between export/import and pruning

[cool-develope]

how do we test v2 ?

[tac0turtle]

gonna merge this as john is working iavlv2 cleann