Big Block Issue Optimization

[yoozo]

Description

Background: Currently, if a large block is encountered, the SDK may fail to index properly. For example, Kusama: 25845986.

It behaves differently under HTTP and WS, and here’s the information we know so far:

• When using WS, it may cause an out-of-memory (OOM) error because the block is too large (at this point, we cannot do anything, and the process will be blocked), potentially leading to a 1006 connection closed error.
• When using HTTP, it may fail due to too many concurrent requests, resulting in many HTTP request failures.

Issues:

• How to detect large blocks?
• WS cannot retrieve large block results properly, causing OOM.
• HTTP cannot handle too many concurrent requests for large blocks, leading to timeouts and connection closures.

Optimizations:

• Implement an RPC response size detection mechanism. If the response exceeds the set size, record this block in the metadata, indicating that the current block height is a large block, which may warrant increasing the timeout.
• Add logs during the processing of large blocks to notify that the process is still running normally.

[stwiname]

Some more information.

Example block: https://kusama.subscan.io/block/25845986 https://polkadot.js.org/apps/#/explorer/query/0xa5d5bddd0e6a478a54535f8988113d93d70f9e1f41371bc33449afbd0f9c15a6

Logs:

2024-11-19 11:35:57        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch events at block 0x8d35c22763788ec8ad467699b75e735f856fca1a3c0dacc5b53ad34f126fa4d3
2024-11-19 11:35:57        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19 11:35:57        RPC-CORE: getBlockHash(blockNumber?: BlockNumber): BlockHash:: No response received from RPC endpoint in 60s
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch events at block 0x0f1780830c6f485c2c9dc14d6e476746fe59b8f57d830d723b8cc072e585bc99
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch events at block 0x24032d6e4ed3bd87cf10c27f3ea7073908aa42ace034382b94025837cc0e2a79
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461156
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461161
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461147
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461146
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461166
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461158
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461169
2024-11-19T11:35:57.918Z <fetch-#1> ERROR failed to fetch BlockHash 23461165

[wangminqi]

Having this issue too. not only Kusama, Polkadot too. Around this period of blocks have ~5000 extrinsic per block. Sample polkadot block: 23527798