Block syncer synchronization performance analysis

[rabbitz]

Environment:

• mode: mainnet
• date: 2024-01-18

Summary:

Upon the Omiga mainnet's launch, a surge in transaction volume per block (800+) led to an escalated delay in block syncer synchronization. The average synchronization time for each block is currently 2 minutes.

Analysis:

1. Through New Relic, abnormal node requests were initially identified. The number of node requests per block synchronization is around 1.3K.

Through simulating mainnet block synchronization locally, it was observed that there is frequent querying of node requests when retrieving the cell_type of output cells. This is due to the necessity of requesting the current network's chain_type each time.

https://github.com/nervosnetwork/ckb-explorer/blob/d8c479b241792b839c8fcc6627b5c69917ee919b/app/models/ckb_sync/new_node_data_processor.rb#L621

https://github.com/nervosnetwork/ckb-explorer/blob/1eff9711fd51e80c6c3a7a3ae2950c190a05fd0a/app/utils/ckb_utils.rb#L407-L420

https://github.com/nervosnetwork/ckb-explorer/blob/97a924409a401e3f14b82b21afdf802dd0e55286/app/models/ckb_sync/api.rb#L24-L32

• Possible Solition: By utilizing ENV["CKB_NET_MODE"], we can reduce the number of requests, thereby enhancing traversal speed.

2. Through New Relic, it was identified that a portion of the process still incurs significant latency. However, specific details regarding the method call chain are currently unavailable.

• Possible Solition: By employing ApplicationRecord.benchmark, we can retrieve the execution duration for each method and inspect the code responsible for the longest processing time.

https://github.com/nervosnetwork/ckb-explorer/blob/d8c479b241792b839c8fcc6627b5c69917ee919b/app/models/ckb_sync/new_node_data_processor.rb#L80-L90

3. CellDependency processing took a considerable amount of time, accounting for around 90% of the synchronization time. Therefore, this processing has been shifted to Sidekiq for asynchronous handling.

https://github.com/nervosnetwork/ckb-explorer/blob/d8c479b241792b839c8fcc6627b5c69917ee919b/app/models/ckb_sync/new_node_data_processor.rb#L122-L124

4. Operations have replaced Redis with AWS services to enhance performance.

Problems:

1. Some queries still involve excessive query counts. We have considered using caching, but it may impact data retrieval. Currently, no optimization has been implemented.

https://github.com/nervosnetwork/ckb-explorer/blob/d8c479b241792b839c8fcc6627b5c69917ee919b/app/models/ckb_sync/new_node_data_processor.rb#L1239

https://github.com/nervosnetwork/ckb-explorer/blob/d8c479b241792b839c8fcc6627b5c69917ee919b/app/models/ckb_sync/new_node_data_processor.rb#L896

2. When transaction synchronization experiences a sharp increase, the asynchronous processing of CellDependency is still slow. Further optimization is required.

https://github.com/nervosnetwork/ckb-explorer/blob/7c06249dba3c8507269c21fdf0582567da1acc2f/app/models/deployed_cell.rb#L52

3. Due to architectural constraints, the current block synchronization is processed sequentially according to the block order, preventing the initiation of multiple containers for parallel synchronization.

https://github.com/nervosnetwork/ckb-explorer/blob/d8c479b241792b839c8fcc6627b5c69917ee919b/app/models/ckb_sync/new_node_data_processor.rb#L5

[rabbitz]

@quake The aforementioned represents the solutions we previously implemented to address encountered issues. Kindly review the remaining aspects that require optimization. Thank you.