search

skalenetwork / skaled

Running more than 20 production blockchains, SKALED is Ethereum-compatible, high performance C++ Proof-of-Stake client, tools and libraries. Uses SKALE consensus as a blockchain consensus core. Includes dynamic Oracle. Implements file storage and retrieval as an EVM extension.
https://skale.network
GNU General Public License v3.0
88 stars 41 forks source link

Performance on the indexer node decreases as the number of archive databases increases. #2017

Open oleksandrSydorenkoJ opened 1 month ago

oleksandrSydorenkoJ commented 1 month ago

Describe the bug The indexer node saves all blocks from the Skale chain to archive db. All databases stay open during skaled work, which may lead to significant resources usage and skaled hangs.

Preconditions Active schain medium type with connected Indexer node Indexr node should have at least 300 million transactions of token transfer Load tests:

Skale chain on QA-net: skale_schain_massive-long-sirius

Versions skalenetwork/schain:3.19.3-beta.0

To Reproduce

  1. Run load tests and wait for 24 hours
  2. Send 5 JSON-RPC requests "eth_getBlockByNumber("latest", false)" Example call
  3. Check the latest block on the indexer node
  4. Check the response time

Expected behavior Skaled on indexer node should be fully synced with core nodes during the load No performance warnings - skaled on indexer node should operate JSON-RPC calls less than in 1 second

Amount of open databases in skaled

schains/massive-long-sirius/28e07f34# find blocks_and_extras/ -mindepth 1 -maxdepth 1 -type d | wc -l
2080

All databases in Snapshots directory 

schains/massive-long-sirius# find snapshots/*  -mindepth 1 -maxdepth 3 -type d | wc -l
6290
2024-10-07 15:00:05.365725   Performance warning: 2.821943 seconds execution time for eth_getBlockByNumber call with id=540 when called from origin http://localhost:47992 through server with index=0
[2024-10-07 15:00:06.169] [8:main] [info] 2255325:BLOCK_COMMITED: PRPSR:15:BID: 2255326:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:4149600e:BLOCK_TXS:17:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:33976:PRT:0:BTA:57610440204:BSA:0:TPS:0:LWT:0:LRT:0:LWC:91:LRC:78:STAMP:1728220748.912
2024-10-07 15:00:06.117266   Block sealed #2255325 (#1666580d…)
2024-10-07 15:00:06.117323   Block stats:BN:2255324:BTS:1728220744:TXS:2487:HDRS:12:LOGS:13766:SENGS:1:TXRS:13726:BLCKS:3:ACCS:199:BQS:1:BDS:72:TSS:0:UTX:0:VTX:0:CMM:10640662:KDS:0
2024-10-07 15:00:06.121137   Block 2255325 DB usage is 1151224. Piece DB usage is 122382995 bytes
2024-10-07 15:00:06.132590   SWT:1475:BFT:1605
2024-10-07 15:00:06.132603   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
2024-10-07 15:00:06.286485   Block sealed #2255326 (#0a245d4f…)
2024-10-07 15:00:06.286508   Block stats:BN:2255325:BTS:1728220745:TXS:51:HDRS:12:LOGS:12971:SENGS:1:TXRS:12931:BLCKS:3:ACCS:199:BQS:1:BDS:73:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:06.286643   Block 2255326 DB usage is 24668. Piece DB usage is 122407663 bytes
2024-10-07 15:00:06.294115   SWT:14:BFT:161
2024-10-07 15:00:06.294128   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:06.321] [8:main] [info] 2255326:BLOCK_COMMITED: PRPSR:1:BID: 2255327:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:4e5297a9:BLOCK_TXS:646:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:34089:PRT:0:BTA:55752038912:BSA:0:TPS:0:LWT:0:LRT:0:LWC:94:LRC:80:STAMP:1728220750.138
2024-10-07 15:00:07.326347   Block sealed #2255327 (#8f14aa5d…)
2024-10-07 15:00:07.326402   Block stats:BN:2255326:BTS:1728220748:TXS:1938:HDRS:12:LOGS:14246:SENGS:1:TXRS:14206:BLCKS:3:ACCS:199:BQS:1:BDS:74:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:07.328746   Block 2255327 DB usage is 897219. Piece DB usage is 123304882 bytes
2024-10-07 15:00:07.338275   SWT:903:BFT:1044
2024-10-07 15:00:07.338293   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:07.368] [8:main] [info] 2255327:BLOCK_COMMITED: PRPSR:1:BID: 2255328:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:22f9b13a:BLOCK_TXS:278:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:34718:PRT:0:BTA:54009787729:BSA:0:TPS:0:LWT:0:LRT:0:LWC:97:LRC:82:STAMP:1728220753.66
2024-10-07 15:00:07.647972   Block sealed #2255328 (#fc0676be…)
2024-10-07 15:00:07.648004   Block stats:BN:2255327:BTS:1728220750:TXS:834:HDRS:12:LOGS:14156:SENGS:1:TXRS:14116:BLCKS:3:ACCS:199:BQS:1:BDS:75:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:07.657688   Block 2255328 DB usage is 386443. Piece DB usage is 123691325 bytes
2024-10-07 15:00:07.665557   SWT:188:BFT:327
2024-10-07 15:00:07.665574   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:07.694] [8:main] [info] 2255328:BLOCK_COMMITED: PRPSR:3:BID: 2255329:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:ff24cd47:BLOCK_TXS:634:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:34889:PRT:0:BTA:52373127505:BSA:0:TPS:0:LWT:0:LRT:0:LWC:100:LRC:84:STAMP:1728220754.447
2024-10-07 15:00:08.680928   Block sealed #2255329 (#4bb73a10…)
2024-10-07 15:00:08.680962   Block stats:BN:2255328:BTS:1728220753:TXS:1902:HDRS:12:LOGS:15146:SENGS:1:TXRS:15106:BLCKS:3:ACCS:199:BQS:1:BDS:76:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:08.684135   Block 2255329 DB usage is 880569. Piece DB usage is 124571894 bytes
2024-10-07 15:00:08.694418   SWT:891:BFT:1028
2024-10-07 15:00:08.694437   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:08.740] [8:main] [info] 2255329:BLOCK_COMMITED: PRPSR:3:BID: 2255330:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:e33669e3:BLOCK_TXS:3:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:35432:PRT:0:BTA:50832741432:BSA:0:TPS:0:LWT:0:LRT:0:LWC:103:LRC:86:STAMP:1728220756.703
2024-10-07 15:00:08.868479   Block sealed #2255330 (#1e4b0963…)
2024-10-07 15:00:08.868520   Block stats:BN:2255329:BTS:1728220754:TXS:9:HDRS:12:LOGS:14518:SENGS:1:TXRS:14478:BLCKS:3:ACCS:199:BQS:1:BDS:77:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:08.868653   Block 2255330 DB usage is 5278. Piece DB usage is 124577172 bytes
2024-10-07 15:00:08.876631   SWT:11:BFT:182
2024-10-07 15:00:08.876643   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:08.909] [8:main] [info] 2255330:BLOCK_COMMITED: PRPSR:5:BID: 2255331:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:f96fd20a:BLOCK_TXS:914:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:35528:PRT:0:BTA:49380377396:BSA:0:TPS:0:LWT:0:LRT:0:LWC:106:LRC:88:STAMP:1728220757.852
2024-10-07 15:00:09.865421   Performance warning: 4.211493 seconds execution time for eth_getBlockByNumber call with id=541 when called from origin http://localhost:47992 through server with index=0
2024-10-07 15:00:10.726525   Block sealed #2255331 (#9e6193ce…)
2024-10-07 15:00:10.726565   Block stats:BN:2255330:BTS:1728220756:TXS:2742:HDRS:12:LOGS:16343:SENGS:1:TXRS:16303:BLCKS:3:ACCS:199:BQS:1:BDS:80:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:10.729571   Block 2255331 DB usage is 1269201. Piece DB usage is 125846373 bytes
2024-10-07 15:00:10.739439   SWT:1710:BFT:1862
2024-10-07 15:00:10.739455   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:10.769] [8:main] [info] 2255331:BLOCK_COMMITED: PRPSR:5:BID: 2255332:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:b8321dd7:BLOCK_TXS:56:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36177:PRT:0:BTA:48008700298:BSA:0:TPS:0:LWT:0:LRT:0:LWC:109:LRC:91:STAMP:1728220760.849
2024-10-07 15:00:10.895286   Block sealed #2255332 (#22f45fa4…)
2024-10-07 15:00:10.895306   Block stats:BN:2255331:BTS:1728220757:TXS:168:HDRS:12:LOGS:15541:SENGS:1:TXRS:15501:BLCKS:3:ACCS:199:BQS:1:BDS:81:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:10.895551   Block 2255332 DB usage is 78683. Piece DB usage is 125925056 bytes
2024-10-07 15:00:10.901784   SWT:24:BFT:162
2024-10-07 15:00:10.901796   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:10.919] [8:main] [info] 2255332:BLOCK_COMMITED: PRPSR:6:BID: 2255333:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:6c7df1ed:BLOCK_TXS:575:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36189:PRT:0:BTA:46711167862:BSA:0:TPS:0:LWT:0:LRT:0:LWC:112:LRC:94:STAMP:1728220762.587
2024-10-07 15:00:11.730274   Block sealed #2255333 (#5e7c25eb…)
2024-10-07 15:00:11.730350   Block stats:BN:2255332:BTS:1728220760:TXS:1725:HDRS:12:LOGS:16635:SENGS:1:TXRS:16595:BLCKS:3:ACCS:199:BQS:1:BDS:82:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:11.732960   Block 2255333 DB usage is 798674. Piece DB usage is 126723730 bytes
2024-10-07 15:00:11.746174   SWT:721:BFT:844
2024-10-07 15:00:11.746196   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:11.778] [8:main] [info] 2255333:BLOCK_COMMITED: PRPSR:7:BID: 2255334:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:ec804c9a:BLOCK_TXS:242:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36658:PRT:0:BTA:45481926624:BSA:0:TPS:0:LWT:0:LRT:0:LWC:115:LRC:96:STAMP:1728220764.558
2024-10-07 15:00:12.026958   Block sealed #2255334 (#a3e5f65f…)
2024-10-07 15:00:12.026997   Block stats:BN:2255333:BTS:1728220762:TXS:726:HDRS:12:LOGS:524:SENGS:1:TXRS:484:BLCKS:3:ACCS:199:BQS:1:BDS:34:TSS:0:UTX:0:VTX:0:CMM:6837:KDS:0
2024-10-07 15:00:12.027856   Block 2255334 DB usage is 336487. Piece DB usage is 127060217 bytes
2024-10-07 15:00:12.035013   SWT:137:BFT:288
2024-10-07 15:00:12.035028   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:12.065] [8:main] [info] 2255334:BLOCK_COMMITED: PRPSR:8:BID: 2255335:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:0eb6be56:BLOCK_TXS:658:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36823:PRT:0:BTA:44315723385:BSA:0:TPS:0:LWT:0:LRT:0:LWC:118:LRC:98:STAMP:1728220766.400
[2024-10-07 15:00:12.065] [8:main] [info] 2255334:LEVELDB_MEM_STATS:BLOCKS:2288841:PROPS:16416:DAPS:16416:OMS:16416:PHS:16416:PVS:16416:BSS:16416:IMS:16416:RMS:16416:PCS:16416:IIN:16416:DAS:16416
[2024-10-07 15:00:12.066] [8:main] [info] 2255334:RUSAGE:USER_CPU:24:240603:PROCESS_RSS:1102492:SWAPPED_MEM:0:HEAP_MEM:1065628592:
2024-10-07 15:00:12.938309   Performance warning: 2.053201 seconds execution time for eth_getBlockByNumber call with id=543 when called from origin http://localhost:47992 through server with index=0
PolinaKiporenko commented 1 month ago

Could be divided in two parts:

  1. skaled blocks DB
  2. consensus blocks DB