streamingfast / substreams

Powerful Blockchain streaming data engine, based on StreamingFast Firehose technology.
Apache License 2.0
160 stars 45 forks source link

Improve protocol versioning between tier1 and tier2 #468

Open matthewdarwin opened 4 months ago

matthewdarwin commented 4 months ago

In the substreams 1.6.0 release notes (https://github.com/streamingfast/substreams/releases/tag/v1.6.0), it says "Upgrading to v1.6.0 will require changing the tier1 and tier2 versions concurrently, as the internal protocol has changed."

If you don't upgrade them both together the tier1 gets a fatal message from the tier2 and then sends fatal (do not retry) error back to the client.

The block stream encountered a substreams fatal error and will not retry: rpc error: code = InvalidArgument desc = validate request: validate tier2 request: stop block 0 should be higher than start block 0

This is an enhancement to have tier1 and tier2 actually negotiate a proper version to avoid this from happening.

Tier1 should sent tier2 a version number in the connect message. If tier2 doesn't understand that version, it should send a non-fatal error back to tier1 (like the "tier2 is too busy now" message). Tier1 will try other teir2 nodes in the cluster presumably.

This is to prevent a single mis-configured tier2 node causing the whole substream to terminate.

Prometheus counter on tier1 can record how many times it gets bad version from tier2.