[indexer]: backend database alternative

[popcnt1]

Background:

the mainnet data set is huge, in presnet, is about 0.3B items, and cost 180GB space for sqlite. Will sqlite works well with these size of data and continue supprts high qps?

[popcnt1]

For sqlite:

p.s.

in rebuild process, sqlite handles well enough(cost 18000s for 0.3B items import): rebuild.log

Test Result(After rebuild):

1. bench1

rooch indexer bench -d ~/moe/rooch_data/indexer_rebuild/ -n main
2024-08-15T13:15:15.869524Z  INFO rooch::commands::indexer::commands::bench: indexer bench started
thread 'main' panicked at crates/rooch/src/commands/indexer/commands/bench.rs:47:10:
called `Result::unwrap()` on an `Err` value: Indexer async read SQLiteDB error: `deadline has elapsed`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

disk iostat:

util: 100% 6k-12k read/s, 300MB/s read

2. bench2

For indexer built from 10M utxo + 3M inscription(16000029 items，8.5G space):

2024-08-15T20:09:13.320622Z  INFO rooch::commands::indexer::commands::bench: indexer bench started
\2024-08-15T20:09:16.654604Z  INFO rooch::commands::indexer::commands::bench: bench start for tx_order: 0, state_index: 10000140. init query cost: 3.333354632s
query: 8, avg_duration(ms): 2309.25
query: 8, avg_duration(ms): 2309.38
query: 8, avg_duration(ms): 2307.54
query: 8, avg_duration(ms): 2307.81
query: 8, avg_duration(ms): 2307.22
query: 8, avg_duration(ms): 2307.92
query: 8, avg_duration(ms): 2308.43
query: 8, avg_duration(ms): 2308.53
query: 8, avg_duration(ms): 2308.67
query: 8, avg_duration(ms): 2308.97
query: 8, avg_duration(ms): 2308.99
total query: 88, avg_duration(ms): 2308.99,
null