search

BuildOnViction / victionchain

The Efficient Blockchain Powered By Proof Of Stake Voting Consensus
https://viction.xyz
GNU Lesser General Public License v3.0
168 stars 83 forks source link

Nodes stop syncing blocks when processing the debug_* method #464

Open Mummyhap opened 1 month ago

Mummyhap commented 1 month ago

The nodes we are running stop syncing blocks when processing the debug method, causing them to only returning timeout errors, like 'RPC exec write tcp 127.0.0.1:21661->127.0.0.1:37036: i/o timeout', which then leads to all other RPC requests timing out as well. Our charts show that the program spends most of its time in the "trie.(Trie).tryGet" function. Currently, the performance debug* is unable to meet Nansen's requirements. The data processing speed cannot keep up with the block height growth. We hope your tech team can optimize the client as soon as possible. This is the node running command:

/node/archive/tomo/bin/tomo --networkid 88 --syncmode full --gcmode archive --announce-txs --datadir /node/archive/tomo/geth/ --tomox.datadir /node/archive/tomo/geth/tomox --txpool.globalslots 1000000 --txpool.globalqueue 3000000 --rpc --rpccorsdomain '*' --rpcvhosts '*' --rpcapi 'eth,net,web3,debug,trace,txpool,tomox' --wsapi 'eth,net,web3,debug,trace,txpool,tomox' --rpcaddr 0.0.0.0 --ws --wsaddr 0.0.0.0 --wsorigins '*' --port 21660 --rpcport 21661 --wsport 21662 --gasprice 250000000 --bootnodes enode://fd3da177f9492a39d*****3ee25ebffc3a@*.*.*.*:30303,enode://97f0ca95a653e3c44*****543f89b2425700a68bc8876@*.*.*.*:30301,enode://b72927f349f3a27b*****8dd77ad85a869032dea@*.*.*.*:30301,enode://c8f2f0643****8b8df351@*.*.*.*:30301 --unlock 0x407638f566802d698117973817d867e3e9604e06 --identity blockpi --mine --store-reward --keystore /node/archive/tomo/geth/keystore --password /node/archive/tomo/geth/passwd --ethstats blockpi:getty-site-pablo-auger-room-sos-blair-shin-whiz-delhi@stats.tomochain.com:443

NghiaNDD98 commented 3 weeks ago

To investigate, you may

tungng98 commented 2 weeks ago

Hello, after some weeks running nodes to stress tests Viction with Debug API. I observed that Viction doesn't have many mechanism to balance the number of requests it can process. Debug API is very resource intensive, mostly on CPU and Disk I/O.

I used haproxy to load balancing and limit maximum connection that a node should handle, and a good value maxconn in my case is from 7 to 10 depend on the hardware of the VM. When I tried to set it higher, 12 for example, block sync can't catchup with the blockchain tip. The delay keeps increasing until the node cannot sync more blocks.