stacks-network / stacks-core

The Stacks blockchain implementation
https://docs.stacks.co
GNU General Public License v3.0
3.01k stars 667 forks source link

`stacks-inspect`: add `index-range` option to `replay-blocks` #4885

Closed kantai closed 3 months ago

kantai commented 3 months ago

Description

This adds the index-range option to replay-blocks. This is useful for selecting a uniformly random slice of blocks for replay (because ordering by index block hash is indistinguishable from a pseudorandom function).

Note: As part of this PR, I also considered making the replay-blocks command alter the SQLite transaction behavior from IMMEDIATE to DEFERRED. The goal was to allow multiple processes to share a chainstate directory during replay-blocks. Unfortunately, this didn't really help, because replay-blocks does perform writes in its transactions: it just rolls back those transactions. This means that SQLite will try to obtain a write lock even in DEFERRED operation.

However, one workable strategy for avoiding storage overhead when trying to run multiple process with replay-blocks is to copy the whole chainstate directory except for chainstate/vm/clarity/marf.sqlite.blobs, and then simply symlink that file for each instance of the replay-blocks processes. That file isn't a database, so it doesn't lock on transactions (and this is safe because none of the replay-blocks processes actually write to that file). This file is also ~75% of the chainstate's disk size.

jcnelson commented 3 months ago

@kantai what if you used BEGIN CONCURRENT instead of BEGIN DEFERRED?

From the docs:

Usually, SQLite allows at most one writer to proceed concurrently. The BEGIN CONCURRENT enhancement allows multiple writers to process write transactions simultanously if the database is in "wal" or "wal2" mode, although the system still serializes COMMIT commands.

When a write-transaction is opened with "BEGIN CONCURRENT", actually locking the database is deferred until a COMMIT is executed. This means that any number of transactions started with BEGIN CONCURRENT may proceed concurrently. The system uses optimistic page-level-locking to prevent conflicting concurrent transactions from being committed.

kantai commented 3 months ago

I thought about that @jcnelson, but I think BEGIN CONCURRENT is only available in a branch of sqlite (https://github.com/sqlite/sqlite/tree/begin-concurrent).