search

streamingfast / substreams

Powerful Blockchain streaming data engine, based on StreamingFast Firehose technology.
Apache License 2.0
164 stars 45 forks source link

Huge substreams overload tier 2 nodes #458

Open matthewdarwin opened 7 months ago

matthewdarwin commented 7 months ago

Some substreams are huge (large number of stores). Some substreams are tiny (no stores).

Trying to set the maximum number of current substreams to run on a single container is hard.

How about a new feature for substreams where you can put a multiplier on the complexity of the substream to better balance the load. We have a substream with 7 stores in a single stage. This gobbles a lot of ram and crashes the tier2 nodes. If we could say every "store" is an extra 0.5, then this substream would cost 1 + (7 x 0.5) = 4.5 instead of just 1 in the context of evaluating the capacity in substreams-tier2-max-concurrent-requests. The "0.5" would be configurable.

matthewdarwin commented 6 months ago

Also some chains have small blocks, other chains have large blocks. There is probably some impact related to that.