impose higher than necessary cognitive load to account for particular invariants (like the error-prone juggling of RocksDB iterator handles #1397) when modifying or composing with new needs (like filtering a stream before picking a random element: https://github.com/helium/erlang-libp2p/pull/437).
This PR canonicalizes a stream abstraction adaptable to any of those needs, implements the common stream processing patterns and adapts them to RocksDB, testing at every step.
Reservations
I'm not crazy about the data_ prefix;
it should ideally live outside of core, so it can be used by libs deeper in the dependency tree, like erlang-libp2p.
Something like erlang-hel (standing for Helium Library) can be a good home.
TL;DR
This is a prerequisite to #1334 which also adds optimal random sampling of RocksDB and other reusable stream operations.
Summary
data_stream:t/1
, initially used in streaming blocks from ledger snapshot;foreach
fold
sample
: implementing an optimal reservoir sampling algorithm - can be used in all cases where we need to pick a random record from RocksDB;pmap_to_bag
: parallel maps a stream into a list with arbitrary re-order - this was used to parallelizebuild_hash_chain
for a ~6x speedup in #1334map
filter
(can be used in https://github.com/helium/erlang-libp2p/pull/437, see:blockchain_rocks_SUITE:t_sample_filtered/1
)append
(allowing chaining N streams into one)blockchain_rocks
), offering all of the above operations.Pitch
Our data access patterns have the same broad shape - stream processing; and the same specific patterns, including, but not limited to:
While we've accumulated adhoc solutions, they:
This PR canonicalizes a stream abstraction adaptable to any of those needs, implements the common stream processing patterns and adapts them to RocksDB, testing at every step.
Reservations
data_
prefix;core
, so it can be used by libs deeper in the dependency tree, likeerlang-libp2p
.Something like
erlang-hel
(standing for Helium Library) can be a good home.