Closed kantai closed 3 months ago
@kantai what if you used BEGIN CONCURRENT
instead of BEGIN DEFERRED
?
From the docs:
Usually, SQLite allows at most one writer to proceed concurrently. The BEGIN CONCURRENT enhancement allows multiple writers to process write transactions simultanously if the database is in "wal" or "wal2" mode, although the system still serializes COMMIT commands.
When a write-transaction is opened with "BEGIN CONCURRENT", actually locking the database is deferred until a COMMIT is executed. This means that any number of transactions started with BEGIN CONCURRENT may proceed concurrently. The system uses optimistic page-level-locking to prevent conflicting concurrent transactions from being committed.
I thought about that @jcnelson, but I think BEGIN CONCURRENT
is only available in a branch of sqlite (https://github.com/sqlite/sqlite/tree/begin-concurrent).
Description
This adds the
index-range
option toreplay-blocks
. This is useful for selecting a uniformly random slice of blocks for replay (because ordering by index block hash is indistinguishable from a pseudorandom function).Note: As part of this PR, I also considered making the
replay-blocks
command alter the SQLite transaction behavior from IMMEDIATE to DEFERRED. The goal was to allow multiple processes to share a chainstate directory during replay-blocks. Unfortunately, this didn't really help, becausereplay-blocks
does perform writes in its transactions: it just rolls back those transactions. This means that SQLite will try to obtain a write lock even in DEFERRED operation.However, one workable strategy for avoiding storage overhead when trying to run multiple process with
replay-blocks
is to copy the whole chainstate directory except forchainstate/vm/clarity/marf.sqlite.blobs
, and then simply symlink that file for each instance of the replay-blocks processes. That file isn't a database, so it doesn't lock on transactions (and this is safe because none of the replay-blocks processes actually write to that file). This file is also ~75% of the chainstate's disk size.