streamingfast / substreams

Powerful Blockchain streaming data engine, based on StreamingFast Firehose technology.

Apache License 2.0

159 stars 45 forks source link

Implement the substreams logic to read block indexes and apply block filters #412

Closed sduchesneau closed 4 months ago

sduchesneau commented 6 months ago

Dependancies

code to convert a blockFilter query to an array of arrays of strings: https://github.com/streamingfast/substreams/issues/409
https://github.com/streamingfast/substreams/issues/411

TODO

On tier2, when one or more modules of a given stage reference a blockIndex module (through the blockFilter attribute),

[x] load it in memory
[x] extract the list of blocks of interest by matching the roaring bitmap to the query string (converted to an array of arrays, with implementation here https://github.com/streamingfast/substreams/issues/409 )
[x] Give the index provider to the filesource (this may need some tweaking in bstream) so that the filesource skips blocks that we don't want.

On tier1, when a module references a blockindex module:

[x] apply the filter query to the output of that index module before deciding if we call the module or skip it...

sduchesneau commented 4 months ago

Shipped in https://github.com/streamingfast/substreams/releases/tag/v1.6.0