search

paritytech / polkadot

Polkadot Node Implementation
GNU General Public License v3.0
7.15k stars 1.58k forks source link

Many Kusama nodes stuck with a large number of transactions in queue #3149

Open bjweaver opened 3 years ago

bjweaver commented 3 years ago

Client ver: 0.9.3 Network: Kusama

One of our nodes, and many in telemetry are running very large transactions in queue backlogs

Nodes in this state will run very high CPU loads.

Restarting the polkadot service will usually clear the transaction backlog and lower the CPU usage.

Nodes should be able to clear this backlog on their own without a service restart.

AlexZhenWang commented 3 years ago

Same issue here. I checked the log, there are many Error importing block xxx: Err(Other(ClientImport("Invalid op │ │ eration in the pending changes tree: Tried to import or finalize node that is an ancestor of a previously finalized node"))) errors. Seems like the nodes were busy reverting blocks, which took lots of CPU resources.

I added --in-peers and --out-peers to use fewer peers, and found the CPU usage became lower.

Swader commented 3 years ago

This was due to an NFT promotion on RMRK and is over now. Restart your nodes and it should be fine now.

bkchr commented 3 years ago

This was due to an NFT promotion on RMRK and is over now. Restart your nodes and it should be fine now.

I want to have this again after https://github.com/paritytech/substrate/pull/8963 :P

bjweaver commented 3 years ago

Restarting is a work around, but this is a bug that will need to be resolved as more and more projects increase the transaction load on Kusama and Polkadot.

Swader commented 3 years ago

Sure, it's being fixed in https://github.com/paritytech/substrate/pull/8963