Improve error message when streaming blocks that are too old

[0237h]

Problem

When streaming blocks that are before the common-first-streamable-block: 265580000, no explicit error message indicates the issue. Instead, the gRPC channel freezes for a while and eventually shuts down with Stream removed errors.

---

Do you think it's possible to detect from the request if the block range or block number is too far back and return a more explicit error message ?

[fschoell]

can you post the full error message here you are getting back? Need to investigate where this comes from

[0237h]

It actually works for older blocks (I've been able to get data as far back as #180000000) but as you go further back, it takes more and more time for processing so I guess it's more of a timeout issue rather than an inability to service blocks.

Here's what the error looks like:

(.venv) user@dev-eosnation:~/Documents/eos-blockchain-data$ python pyfirehose 170000000 170000001 -g eos_firehose_v2 -p firehose.default.eos_block_processor -l logs/test_bug.log                                                                                                                                         
2022-12-07 10:05:44:T+184 [*] Getting JWT token...
2022-12-07 10:05:45:T+193 [*] Got JWT token (cached) [SUCCESS]
2022-12-07 10:05:45:T+194 [*] Streaming 1 blocks on EOS chain (running 1 workers)...
2022-12-07 10:05:45:T+195 [*] Getting JWT token...
2022-12-07 10:05:45:T+196 [*] Got JWT token (cached) [SUCCESS]
E1207 10:11:07.086865328    4768 hpack_parser.cc:1228]       Error parsing metadata: error=invalid value key=content-type value=text/html
2022-12-07 10:11:07:T+322275 [ERROR] [Task-02] Failed to process block number #170000000: <AioRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:216.66.68.24:9001 {grpc_message:"Stream removed", grpc_status:2, created_time:"2022-12-07T10:11:07.087042174-05:00"}"

More from the log file:

2022-12-07 10:05:45:T+197 [DEBUG] [Task-02] Starting streaming blocks from #170000000 to #170000001...
2022-12-07 10:11:07:T+322274 [DEBUG] Failed to receive any message from Core
2022-12-07 10:11:07:T+322275 [ERROR] [Task-02] Failed to process block number #170000000: <AioRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Stream removed"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:216.66.68.24:9001 {grpc_message:"Stream removed", grpc_status:2, created_time:"2022-12-07T10:11:07.087042174-05:00"}"
>
2022-12-07 10:11:07:T+322275 [DEBUG] [Task-05] Starting streaming blocks from #170000000 to #170000001...
2022-12-07 10:16:08:T+623329 [DEBUG] Failed to receive any message from Core
2022-12-07 10:16:08:T+623329 [ERROR] [Task-05] Failed to process block number #170000000: <AioRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Stream removed"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:216.66.68.24:9001 {grpc_message:"Stream removed", grpc_status:2, created_time:"2022-12-07T10:16:08.141667316-05:00"}"
>
2022-12-07 10:16:08:T+623330 [DEBUG] [Task-08] Starting streaming blocks from #170000000 to #170000001...

2022-12-07 10:11:07:T+322274 [DEBUG] Failed to receive any message from Core this does seems to me like a timeout issue.

[matthewdarwin]

The old blocks are still being processed. There are "random" holes in the chain until all that gets filled in.

[fschoell]

I'm closing this issue as this is not the responsibility of firehose-antelope. This error handling happens in the firehose and substreams apps or in the bstream library. It seems like the Streamingfast team is aware about this error being confusing.