Closed abourget closed 3 months ago
@abourget how would you feel about an "in-place" migration for all existing add
stores, where we check the first byte for "a" or "s", and if it is not there, treat thee existing snapshot as an "a" and save the next snapshot accordingly?
@abourget or do you think it would be best to simply have a new store update policy called AddOrSet ?
what about SetSum
? add
is a bit unfortunate.. set:
and sum:
prefixes because they will be visible in StoreDeltas arriving in some modules' inputs.. so a quick separator and a readable term would be good
The set+sum
store cannot be parallelize since it is not an associativity
operation.
We know that substreams requires associativity of operations to work, since it can "group" the work into shards, i.e. (a+b)+(c+d) must be equal to (a+b+c) + (d) given a set of operations on a store that allow "adds" and "sets": add(2) add(5) set(0) add(6)
The first grouping gives 13 = (+2, +5) + (set0, +6) = +7 +6 = 13
But this grouping gives 6 (+2 +5 set0) + (+6) = 0 + 6 = 6
(+2, +5) + (set0, +6)
-> 6
, not 13
, because each time we hit a set
, we erase whatever was accumulated from the left
example substreams posted here: https://github.com/streamingfast/substreams-examples
PropellerHeads need this.
Definition of done done:
substreams-rs
has been updated and published, to provide the new primitivesBy having a single byte prefix in an
add
store, we could know if someone hasset
a key, and then the merge operation could either SET ofadd
with the previous segment's keys.Example flow:
This would help certain folks trying to use some set (when the liquidity of something is published in absolute terms), and track with greater precision all the adds and removals that are tracked in relative terms (amounts of burns, mints, etc..)
This feature would not be so complex to build, it would add an
updatePolicy
here and there.. but all those functions are very simple.