search

skalenetwork / skaled

Running more than 20 production blockchains, SKALED is Ethereum-compatible, high performance C++ Proof-of-Stake client, tools and libraries. Uses SKALE consensus as a blockchain consensus core. Includes dynamic Oracle. Implements file storage and retrieval as an EVM extension.
https://skale.network
GNU General Public License v3.0
84 stars 40 forks source link

Skaled couldn't send serialized blocks on random node on Skale chain. #1610

Open oleksandrSydorenkoJ opened 1 year ago

oleksandrSydorenkoJ commented 1 year ago

After load and recoverability tests, skaled on one of nodes printed in logs "CatchupServerAgent" warning continuously for 10 seconds and repeat for every block.

Veriosn skalenetwork/schain:3.17.0-beta.6

Preconditions Active schain medium type Connected blockscout Load tests with block rotations: 200 users 10 MTM_bulk 8Kb data

Steps to reproduce

  1. Run load test on 16 endpoints
  2. Stop manually 4 skaled for 4 hours
  3. After 4 hours delay - restart skaled and stop the load
  4. wait until 4 nodes will sync with other nodes
  5. check skaled logs on all nodes

Expected state Skaled on all nodes should work without warnings or errors

Actual state: Random node has been stuck and cannot send blocks to other nodes because couldn't send serialized binary

Logs: skaled_couldnt_send_serialized_blocks.txt

oleksandrSydorenkoJ commented 1 year ago

UPD. may be fixed by restarting of skaled container

kladkogex commented 1 year ago

Will take a look at this for 2.4 - these things sometimes happen usually when the receiving node is underloaded