mitchmindtree
commented
1 month ago OK after our chat and doing some more thinking I think I've found an approach with a pretty nice balance.
The aim is to setup the freedom for experimenting with parallel checking of sequential chunks of solutions, but with control over the upper bound on worst-case performance. I've also tried to do so in a way that doesn't conflate the role of the node DB with the builder's solution sequencing process while keeping the implementation pretty minimal/simple.
state::Mutations
The Transaction type has been removed in favour of a Mutations map that allows for querying the state of a key at any point within some sequence of proposed solutions. It is similar to the node DB's mutations table, but avoids the need to calculate the block hash, recalculate solution hashes, or insert unrelated data like solution_hash or data_index as we do in node_db::insert_block.
state::View
Basically the same as the node validation's State type - it wraps a node DB connection pool and a Arc<Mutations> instance and provides a view into either the pre or post state for a given solution index.
Parallel Checking
This enables optimistic parallel checking of "chunks" of sequential solutions at a time, where the chunk size defaults to num_cpus::get(). By limiting the number of solutions checked in parallel at a time, we reduce the number of times some solutions must be re-checked in the case that an earlier solution was found to be invalid.
The chunk size is configurable, so that we can tune this trade-off depending on the ratio of invalid to valid solutions that we observe in practise. Setting it to 1 matches the original sequential behaviour.
essential-builder-dbtweaksThis adds a new
solution_failuretable to the builder DB, along with query and deletion. This table is purely for tracking failures so that we're able to give feedback to solution submitters. We just store some fixed number up to a certain limit, and delete the oldest logs when the limit is exceeded.A
ConnectionPoolhas been added for thebuilder_dbwith shorthands for common builder DB interactions.essential-builderimplementationThis implementation is a simple FIFO block building approach. The builder queries a large chunk of solutions from the DB, and attempts to apply them all in the order in which they were submitted. All successful solutions are applied to the block, while all failed solutions have their failures recorded to the new
solution_failuretable.All solutions that were attempted, whether successful or not, are deleted from the pool. In the follow-up API work, we should be able to expose a
solution-outcomerequest that returns the latest inclusions and failures, ordered by how recent they are. Submitters can use this to determine whether or not they wish to resubmit their solution.To-Do
essential-builder.Follow-up
Closes #3 Closes #5