Substreams initial block later than chain's head leads to weird error

[maoueh]

I trie to run a substreams.yaml whose initialBlock was 22207818 but on mainnet.eth.streamingfast.io:443 by mistake. The initial block is later than Ethereum Mainnet chain's head.

This seemed to work but when bundling happened, an execution error was received:

2023-08-30T23:21:47.590-0400 ERRO (sink-postgres) substreams encountered a retryable error {"error": "rpc error: code = Unknown desc = rpc error: code = Internal desc = error during init_stores_and_backprocess: run_parallel_process failed: parallel processing run: stores didn't sync up properly, expected store \"store_block_meta_end\" to be at block 18031600 but was at 0: load store \"store_block_meta_end\": load full store store_block_meta_end at 0018031600-0022207818.kv: opening file: not found"}
2023-08-30T23:21:47.590-0400 INFO (sink-postgres) sleeping before re-connecting {"sleep": "551.129931ms"}

It appears we don't properly handle the case where the initialBlock of a module is in the future. We should probably simply reject.

[colindickson]

tried with latest substreams develop against our unstable endpoint and i am not able to reproduce. @maoueh can you try with your substreams to reproduce this way?

[maoueh]

Yep going to try sure

[maoueh]

Nope, same error. Here my try out steps.

1. Edit https://github.com/streamingfast/substreams-eth-block-meta so that initialBlock: 22207818 is there under kind of every module
2. In substreams-sink-postgres, run ./up.sh -c
3. In substreams-sink-postgres, run

   SUBSTREAMS_MANIFEST="`pwd`/../substreams-eth-block-meta" SUBSTREAMS_ENDPOINT="api-unstable.streamingfast.io:443" ./devel/eth-block-meta/start.sh -c

[colindickson]

ok yes, i can reproduce now even without the sink:

1. editing the eth-block-meta substreams as you described
2. building the .spkg
3. running substreams run -e mainnet.eth.streamingfast.io:443 ./substreams.spkg db_out

[colindickson]

issue fixed with https://github.com/streamingfast/substreams/commit/9240d8f141899e272e3446e382da8c06c86352e7

[maoueh]

It might not be fully resolved, see https://discord.com/channels/666749063386890256/1146717627952152589/1153622551755497492.

We need to investigate that again.

[AngeloCa]

I am having an issue when trying to stream my .spkg with an initialBlock close to the head of the chain (around 100-200 blocks).

It doesn't work in both production and dev mode. I am requesting it as follow

substreams run -e mainnet.eth.streamingfast.io:443 my-package.spkg map_out --start-block 18274900 --stop-block +1

Here start is equal to the initial block in the manifest and is 100 blocks before the head of the chain.

Here is a Trace Id that gives the error : aa9c9377821670e7609086758c722d50

[AngeloCa]

I did a couple of tests:

During my first test, I used initialBlock = 18275900 and it crashed without any Error Message, it only said Cliend Error.

Then I waited a bit to try with initial block = 18276000 and I got a different error message:

Error: rpc error: code = Unknown desc = rpc error: code = Internal desc = error during init_stores_and_backprocess: run_parallel_process failed: parallel processing run: stores didn't sync up properly, expected store "store_first_token_transfer_block" to be at block 18275900 but was at 0: load store "store_first_token_transfer_block": load full store store_first_token_transfer_block at 0018275900-0018276000.kv: opening file: not found.

I don't know how the files are handled by the server but the boudaries of this file 0018275900-0018276000.kv are the ones from my first test that crashed and the others from the second test.