sduchesneau
commented
1 year ago Hi,
Tier-2 write its processed files to the substreams-state-store-url Tier-1 reads its processed files from substreams-state-store-url The system has no way of predicting if those two URLs point to the same storage (ex: nfs mount point, local files, etc.), so no easy lookup would fix this.
In production mode, the "LinearReader" polls for the files that it wants "forever" instead of being informed by the scheduler about what files "are waiting to be processed". This would need to be modified heavily to fix your specific issue. This is not something that we will do in the near future, as it does not bring much value for the effort involved (outside of catching a misconfiguration) -- Unless we find a good reason to go that road.
Closing as "won't fix".
Meanwhile, I recommend that you run this simple command (in develop mode) to validate configuration changes on your tiers:
substreams run -e {endpoint} https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.4.1/substreams-eth-block-meta-v0.4.1.spkg kv_out -s 2 -t 3This will fail within a few seconds if the state stores are misconfigured.
We had an issue with tier-2 firehose misconfiguration (wrong
substream-state-store-urlflag), so if you stream a module in production mode, you would eventually get an error:That's great, however, the problem is that the RPC error will only show up after all blocks are processed, so it may take many hours before we reach the head block, and you see that error. Before that, you just get progress messages as if everything's going well.
Would be great if you could see that error sooner.