Substreams module's `initialBlock` is not considered as the chain's genesis block

[maoueh]

On arb-one.streamingfast.io:443, if you try a substreams.spkg for which the initialBlock is 0 (and you don't specify any range which means full chain), then you are faced with a weird error message from the server:

2023-08-30T23:19:36.051-0400 ERRO (sink-postgres) substreams encountered a retryable error {"error": "rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = start block 22207818 is after stop block 70000"}

This happens as the tier2 makes a Firehose request and the error comes from there. And since we do 10 parallel request or more, the stop block varies as it's a race to the first one that succeed in forward its error.

There is 2 sides to this issue IMO:

• The initialBlock: 0 in my opinion should always be considered as the first streamable block of the chain
• Validation is missing when an initialBlock != 0 && initialBlock < firstStreamableBlock in which case the error should be clear
  • Side missing, validated that the gRPC error code going out is InvalidArgument and not rpc error: code = Unknown desc = rpc error: code = InvalidArgument (which is an Unknown error whose message is a stringified gRPC InvalidArgument error so consumer see Unknown error but it should have seen InvalidArgument)

[colindickson]

https://github.com/streamingfast/substreams/pull/296

[maoueh]

Re-opening because it seems the RPC is still rejecting such requests which should be accepted as chain's genesis head block.

Not supporting that breaks graph-node support for generic Substreams as on boot it request "genesis" block by making a request for block 0. This gives graph-node knowledge of the genesis block of the chain then comparer among provider on the same chain to avoid any misconfiguration (or worst the chain's rebooting).

See https://github.com/graphprotocol/graph-node/issues/4769#issuecomment-1781286362 for details there.

Reproduction steps:

# In https://github.com/streamingfast/substreams-sink-noop/tree/develop/substreams-head-tracker
make run START_BLOCK=0 ENDPOINT="mainnet.sol.streamingfast.io:443"
...

Error: rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = invalid start block 0, must be >= 1690600 (the first streamable block of the chain)

Firehose works properly with such request:

grpcurl -H "Authorization: Bearer $SF_API_TOKEN" -d '{"start_block_num":0}' mainnet.sol.streamingfast.io:443 sf.firehose.v2.Stream/Blocks | jq -r .block.slot | head -n1
1690600

[colindickson]

fixed with https://github.com/streamingfast/substreams/commit/d2f4c22f6021cec14da03b8cebcfedc8105fa69d

[maoueh]

Re-opening, it appears in production mode the first scheduled block is stucked, can be tested on Solana:

• With 0 as start block, first job stay stucked: substreams run -e mainnet.sol.streamingfast.io:443 https://github.com/graphprotocol/graph-node/raw/master/substreams/substreams-head-tracker/substreams-head-tracker-v1.0.0.spkg --production-mode -s 0 map_blocks
• Before the first job boundary, it's always broken too: - substreams run -e mainnet.sol.streamingfast.io:443 https://github.com/graphprotocol/graph-node/raw/master/substreams/substreams-head-tracker/substreams-head-tracker-v1.0.0.spkg --production-mode -s 1690999 map_blocks
• Only when starting on the "next" job boundary it works: substreams run -e mainnet.sol.streamingfast.io:443 https://github.com/graphprotocol/graph-node/raw/master/substreams/substreams-head-tracker/substreams-head-tracker-v1.0.0.spkg --production-mode -s 1691000 map_blocks this streams data right away.

[colindickson]

This is an issue with Solana's data integrity situation at the moment. The merged blocks are currently in a weird state. If you test the same scenario on Near, which has a high start block as well, you can see it starts streaming right away as expected:

substreams run -e mainnet.near.streamingfast.io:443 https://github.com/graphprotocol/graph-node/raw/master/substreams/substreams-head-tracker/substreams-head-tracker-v1.0.0.spkg --production-mode -s 0 map_blocks