search

AntelopeIO / spring

C++ implementation of the Antelope protocol with Savanna consensus
Other
10 stars 5 forks source link

api get_info has no response randomly #991

Open lcgogo opened 1 month ago

lcgogo commented 1 month ago

After monitoring the log, I find the cpu metric (like cpu: 37350 us) grows very high (x5-10 than normal) when the get_info have no response.

I am sure the server has enough resource (8c64g) and disk

Is there any config to reduce this issue? I use spring 1.0.2 and starts from Oct 25th.

{"log":"\u001b[0minfo  2024-10-30T03:39:00.960 nodeos    controller.cpp:3641           log_applied          ] Received block 9ff6bb40da8b45f6... #402036080 @ 2024-10-30T03:39:01.000 signed by binancestake [trxs: 17, lib: 402036078, net: 2840, cpu: 9220 us, elapsed: 10436 us, applying time: 11460 us, latency: -39 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:00.960810687Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:01.421 nodeos    controller.cpp:3641           log_applied          ] Received block c9c527564f04562f... #402036081 @ 2024-10-30T03:39:01.500 signed by binancestake [trxs: 15, lib: 402036079, net: 2128, cpu: 9249 us, elapsed: 10360 us, applying time: 11230 us, latency: -78 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:01.421994665Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:01.878 nodeos    controller.cpp:3641           log_applied          ] Received block 7da2afd22d121e80... #402036082 @ 2024-10-30T03:39:02.000 signed by binancestake [trxs: 9, lib: 402036080, net: 1232, cpu: 5462 us, elapsed: 4815 us, applying time: 5373 us, latency: -121 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:01.87857747Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:02.343 nodeos    controller.cpp:3641           log_applied          ] Received block 736a1832aca7622f... #402036083 @ 2024-10-30T03:39:02.500 signed by binancestake [trxs: 13, lib: 402036081, net: 1856, cpu: 8702 us, elapsed: 8663 us, applying time: 9374 us, latency: -156 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:02.343972213Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:02.869 nodeos    controller.cpp:3641           log_applied          ] Received block 22f40b624bd1866d... #402036084 @ 2024-10-30T03:39:03.000 signed by binancestake [trxs: 17, lib: 402036082, net: 4104, cpu: 37350 us, elapsed: 55873 us, applying time: 56647 us, latency: -130 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:02.869749013Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:03.409 nodeos    controller.cpp:3641           log_applied          ] Received block f7c6fcd9de87665a... #402036085 @ 2024-10-30T03:39:03.500 signed by binancestake [trxs: 11, lib: 402036083, net: 1656, cpu: 93535 us, elapsed: 142344 us, applying time: 143175 us, latency: -90 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:03.410102133Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:03.841 nodeos    controller.cpp:3641           log_applied          ] Received block 96aaf37a914a3213... #402036086 @ 2024-10-30T03:39:04.000 signed by binancestake [trxs: 15, lib: 402036084, net: 2128, cpu: 84438 us, elapsed: 125085 us, applying time: 126080 us, latency: -158 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:03.841681292Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:04.218 nodeos    controller.cpp:3641           log_applied          ] Received block 2eb4e71cb9a11f57... #402036087 @ 2024-10-30T03:39:04.500 signed by binancestake [trxs: 10, lib: 402036085, net: 1256, cpu: 22636 us, elapsed: 28446 us, applying time: 29183 us, latency: -281 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:04.218753219Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:04.729 nodeos    controller.cpp:3641           log_applied          ] Received block c356a16cba4de394... #402036088 @ 2024-10-30T03:39:05.000 signed by binancestake [trxs: 12, lib: 402036086, net: 1744, cpu: 33986 us, elapsed: 56743 us, applying time: 57305 us, latency: -270 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:04.729409404Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:05.165 nodeos    controller.cpp:3641           log_applied          ] Received block 6dd8e858b7492344... #402036089 @ 2024-10-30T03:39:05.500 signed by binancestake [trxs: 9, lib: 402036087, net: 1208, cpu: 42544 us, elapsed: 60583 us, applying time: 61182 us, latency: -334 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:05.165421856Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:06.153 nodeos    controller.cpp:3641           log_applied          ] Received block eac22bb6379b105e... #402036090 @ 2024-10-30T03:39:06.000 signed by bp.daobox [trxs: 7, lib: 402036088, net: 880, cpu: 1811 us, elapsed: 2807 us, applying time: 3315 us, latency: 153 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:06.154146281Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:13.166 nodeos    controller.cpp:3641           log_applied          ] Received block 702f732c879a1587... #402036091 @ 2024-10-30T03:39:06.500 signed by bp.daobox [trxs: 9, lib: 402036089, net: 1232, cpu: 63714 us, elapsed: 6615375 us, applying time: 6619880 us, latency: 6666 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:13.16703147Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:17.163 nodeos    controller.cpp:3641           log_applied          ] Received block beec0b55c895a332... #402036092 @ 2024-10-30T03:39:07.000 signed by bp.daobox [trxs: 6, lib: 402036090, net: 840, cpu: 33644 us, elapsed: 3993749 us, applying time: 3996450 us, latency: 10163 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:17.163581626Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:20.958 nodeos    controller.cpp:3641           log_applied          ] Received block 5d9dba202fef2077... #402036093 @ 2024-10-30T03:39:07.500 signed by bp.daobox [trxs: 5, lib: 402036091, net: 928, cpu: 32364 us, elapsed: 3792047 us, applying time: 3794877 us, latency: 13458 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:20.958382072Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:25.038 nodeos    controller.cpp:3641           log_applied          ] Received block 209823b1f423a51d... #402036094 @ 2024-10-30T03:39:08.000 signed by bp.daobox [trxs: 8, lib: 402036092, net: 377640, cpu: 39682 us, elapsed: 4076699 us, applying time: 4079770 us, latency: 17038 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:25.038200196Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:28.740 nodeos    controller.cpp:3641           log_applied          ] Received block 5735896513e264a8... #402036095 @ 2024-10-30T03:39:08.500 signed by bp.daobox [trxs: 11, lib: 402036093, net: 498888, cpu: 35864 us, elapsed: 3699324 us, applying time: 3702741 us, latency: 20240 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:28.740977475Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:33.398 nodeos    controller.cpp:3641           log_applied          ] Received block ddb6dd4a39512153... #402036096 @ 2024-10-30T03:39:09.000 signed by bp.daobox [trxs: 12, lib: 402036094, net: 363752, cpu: 53781 us, elapsed: 4652920 us, applying time: 4657762 us, latency: 24398 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:33.398749439Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:34.397 nodeos    controller.cpp:3641           log_applied          ] Received block 9972018a515361ca... #402036097 @ 2024-10-30T03:39:09.500 signed by bp.daobox [trxs: 11, lib: 402036095, net: 325216, cpu: 14051 us, elapsed: 996949 us, applying time: 998912 us, latency: 24897 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:34.397713458Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:35.829 nodeos    controller.cpp:3641           log_applied          ] Received block 06c486b7c6ef01c7... #402036098 @ 2024-10-30T03:39:10.000 signed by bp.daobox [trxs: 5, lib: 402036096, net: 688, cpu: 13227 us, elapsed: 1430345 us, applying time: 1431853 us, latency: 25829 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:35.82962522Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:36.456 nodeos    controller.cpp:3641           log_applied          ] Received block f7f151ed4dcfd118... #402036099 @ 2024-10-30T03:39:10.500 signed by bp.daobox [trxs: 7, lib: 402036097, net: 263408, cpu: 8571 us, elapsed: 625527 us, applying time: 626891 us, latency: 25956 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:36.456514222Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:37.099 nodeos    controller.cpp:3641           log_applied          ] Received block 5f7282fe871e2a59... #402036100 @ 2024-10-30T03:39:11.000 signed by bp.daobox [trxs: 10, lib: 402036098, net: 656792, cpu: 10043 us, elapsed: 641673 us, applying time: 643405 us, latency: 26099 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:37.099969621Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:38.496 nodeos    controller.cpp:3641           log_applied          ] Received block e2c6b686d2ccb20d... #402036101 @ 2024-10-30T03:39:11.500 signed by bp.daobox [trxs: 14, lib: 402036099, net: 657904, cpu: 17488 us, elapsed: 1394223 us, applying time: 1396825 us, latency: 26996 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:38.496831989Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:38.504 nodeos    controller.cpp:3641           log_applied          ] Received block cead1b2932239c27... #402036102 @ 2024-10-30T03:39:12.000 signed by bp.defi [trxs: 9, lib: 402036100, net: 1184, cpu: 5401 us, elapsed: 6812 us, applying time: 7295 us, latency: 26504 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:38.504127409Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:39.263 nodeos    controller.cpp:3641           log_applied          ] Received block 75d1bc65a45aedfa... #402036103 @ 2024-10-30T03:39:12.500 signed by bp.defi [trxs: 11, lib: 402036101, net: 380816, cpu: 12973 us, elapsed: 757121 us, applying time: 758946 us, latency: 26763 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:39.263160004Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:39.412 nodeos    controller.cpp:3641           log_applied          ] Received block 237952fe532d2f97... #402036104 @ 2024-10-30T03:39:13.000 signed by bp.defi [trxs: 9, lib: 402036102, net: 1656, cpu: 5740 us, elapsed: 148238 us, applying time: 149316 us, latency: 26412 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:39.412590009Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:39.425 nodeos    controller.cpp:3641           log_applied          ] Received block 74ad709d50e89c43... #402036105 @ 2024-10-30T03:39:13.500 signed by bp.defi [trxs: 6, lib: 402036103, net: 1160, cpu: 4637 us, elapsed: 10630 us, applying time: 11490 us, latency: 25925 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:39.425136544Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:39.507 nodeos    controller.cpp:3641           log_applied          ] Received block da9c16763066103a... #402036106 @ 2024-10-30T03:39:14.000 signed by bp.defi [trxs: 12, lib: 402036104, net: 877392, cpu: 28339 us, elapsed: 81167 us, applying time: 82610 us, latency: 25507 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:39.507774275Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:39.653 nodeos    controller.cpp:3641           log_applied          ] Received block 266d3d36492c7f6a... #402036107 @ 2024-10-30T03:39:14.500 signed by bp.defi [trxs: 14, lib: 402036105, net: 664232, cpu: 39022 us, elapsed: 144353 us, applying time: 145933 us, latency: 25153 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:39.653761349Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:39.872 nodeos    controller.cpp:3641           log_applied          ] Received block 96f5fc10152800c4... #402036108 @ 2024-10-30T03:39:15.000 signed by bp.defi [trxs: 19, lib: 402036106, net: 1046104, cpu: 71841 us, elapsed: 216669 us, applying time: 218695 us, latency: 24872 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:39.872504496Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:39.977 nodeos    controller.cpp:3641           log_applied          ] Received block 4b283b21103d398c... #402036109 @ 2024-10-30T03:39:15.500 signed by bp.defi [trxs: 10, lib: 402036106, net: 534920, cpu: 34330 us, elapsed: 104038 us, applying time: 105473 us, latency: 24477 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:39.978003616Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.085 nodeos    controller.cpp:3641           log_applied          ] Received block e738a3a1cfc680b4... #402036110 @ 2024-10-30T03:39:16.000 signed by bp.defi [trxs: 6, lib: 402036107, net: 880, cpu: 33664 us, elapsed: 107255 us, applying time: 107875 us, latency: 24085 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.085920939Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.253 nodeos    controller.cpp:3641           log_applied          ] Received block 39f4160abc1451b1... #402036111 @ 2024-10-30T03:39:16.500 signed by bp.defi [trxs: 12, lib: 402036109, net: 1848, cpu: 59444 us, elapsed: 167174 us, applying time: 168073 us, latency: 23753 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.25404775Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.313 nodeos    controller.cpp:3641           log_applied          ] Received block 661de5aad3d0fa93... #402036112 @ 2024-10-30T03:39:17.000 signed by bp.defi [trxs: 6, lib: 402036110, net: 936, cpu: 21553 us, elapsed: 58824 us, applying time: 59384 us, latency: 23313 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.313453474Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.314 nodeos    controller.cpp:3641           log_applied          ] Received block f69b41302f4170fa... #402036113 @ 2024-10-30T03:39:17.500 signed by bp.defi [trxs: 4, lib: 402036111, net: 1016, cpu: 1162 us, elapsed: 1019 us, applying time: 1510 us, latency: 22814 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.314961121Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.342 nodeos    controller.cpp:3641           log_applied          ] Received block 805e3fa2687ebcf2... #402036114 @ 2024-10-30T03:39:18.000 signed by eoscannonchn [trxs: 13, lib: 402036112, net: 4704, cpu: 18764 us, elapsed: 26320 us, applying time: 27155 us, latency: 22342 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.342178158Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.345 nodeos    controller.cpp:3641           log_applied          ] Received block 0c72b1a3cc89f116... #402036115 @ 2024-10-30T03:39:18.500 signed by eoscannonchn [trxs: 4, lib: 402036113, net: 480, cpu: 2269 us, elapsed: 2629 us, applying time: 3048 us, latency: 21845 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.345227363Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.347 nodeos    controller.cpp:3641           log_applied          ] Received block 373a64dec1a8b5a2... #402036116 @ 2024-10-30T03:39:19.000 signed by eoscannonchn [trxs: 6, lib: 402036114, net: 2632, cpu: 3281 us, elapsed: 1753 us, applying time: 2167 us, latency: 21347 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.347438972Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.434 nodeos    controller.cpp:3641           log_applied          ] Received block 979cc377b048c0ba... #402036117 @ 2024-10-30T03:39:19.500 signed by eoscannonchn [trxs: 8, lib: 402036115, net: 3872, cpu: 3742 us, elapsed: 86690 us, applying time: 87324 us, latency: 20934 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.434823288Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.437 nodeos    controller.cpp:3641           log_applied          ] Received block 0156dee6a916a005... #402036118 @ 2024-10-30T03:39:20.000 signed by eoscannonchn [trxs: 5, lib: 402036116, net: 608, cpu: 4158 us, elapsed: 2620 us, applying time: 3028 us, latency: 20437 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.437862522Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.448 nodeos    controller.cpp:3641           log_applied          ] Received block 50db0badd0d5dbce... #402036119 @ 2024-10-30T03:39:20.500 signed by eoscannonchn [trxs: 6, lib: 402036117, net: 2416, cpu: 6407 us, elapsed: 9741 us, applying time: 10331 us, latency: 19948 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.448233728Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.451 nodeos    controller.cpp:3641           log_applied          ] Received block a89d6183944da676... #402036120 @ 2024-10-30T03:39:21.000 signed by eoscannonchn [trxs: 4, lib: 402036118, net: 480, cpu: 2229 us, elapsed: 2658 us, applying time: 2993 us, latency: 19451 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.451239742Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.454 nodeos    controller.cpp:3641           log_applied          ] Received block 2b8c462bdf187dfc... #402036121 @ 2024-10-30T03:39:21.500 signed by eoscannonchn [trxs: 6, lib: 402036119, net: 736, cpu: 3035 us, elapsed: 2812 us, applying time: 3202 us, latency: 18954 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.454477429Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.495 nodeos    controller.cpp:3641           log_applied          ] Received block 2483789840d52eb9... #402036122 @ 2024-10-30T03:39:22.000 signed by eoscannonchn [trxs: 7, lib: 402036120, net: 1296, cpu: 6327 us, elapsed: 40924 us, applying time: 41415 us, latency: 18495 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.495949762Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.502 nodeos    controller.cpp:3641           log_applied          ] Received block 47293713767211f3... #402036123 @ 2024-10-30T03:39:22.500 signed by eoscannonchn [trxs: 6, lib: 402036121, net: 888, cpu: 5499 us, elapsed: 5774 us, applying time: 6263 us, latency: 18002 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.502213964Z"}
{"log":"\u001b[0minfo  2024-10-30T03:39:40.505 nodeos    controller.cpp:3641           log_applied          ] Received block d737f82fd56e8ddc... #402036124 @ 2024-10-30T03:39:23.000 signed by eoscannonchn [trxs: 7, lib: 402036122, net: 880, cpu: 4414 us, elapsed: 2893 us, applying time: 3324 us, latency: 17505 ms]\u001b[0m\r\n","stream":"stdout","time":"2024-10-30T03:39:40.505578955Z"}
heifner commented 1 month ago

When syncing get_info can take minutes to respond. This has been greatly improved in 1.0.3 (not yet released) and improved even more in 1.1.0.

kiselev-sv commented 1 month ago

I have many errors such nodeos controller.cpp:3641 log_applied ] Received block 53b00922ab0b86b9... #402110449 @ 2024-10-30T14:01:44.500 signed by eossupportbp [trxs: 7, lib: 402110447, net: 1360, cpu: 2414 us, elapsed: 751 us, applying time: 1478 us, latency: -157 ms the node is started from snapshot v8 and synced

trace api plugin configured

I call trace_api/get_block

/v1/trace_api/get_block '{"block_num":401763564}' with different blocks

heifner commented 1 month ago

Please try again with https://github.com/AntelopeIO/spring/releases/tag/v1.0.3

lcgogo commented 2 weeks ago

Please try again with https://github.com/AntelopeIO/spring/releases/tag/v1.0.3

Tried 1.0.3，better than 1.0.2, but still met about 10-20 times in one day with per min curl test.

heifner commented 2 weeks ago

Can you provide the nodeos log at a time when there ia a delay. Also please provide config.ini and command line args of nodeos.

lcgogo commented 1 day ago

Can you provide the nodeos log at a time when there ia a delay. Also please provide config.ini and command line args of nodeos.

cat config.ini
# Specify the Access-Control-Allow-Origin to be returned on each request (eosio::http_plugin)
access-control-allow-origin = *

# The name supplied to identify this node amongst the peers. (eosio::net_plugin)
agent-name = "Cobo EOS"

# Enable block production, even if the chain is stale. (eosio::producer_plugin)
enable-stale-production = true

# Limits the maximum age (in seconds) of the DPOS Irreversible Block for a chain this node will produce blocks on (use negative value to indicate unlimited) (eosio::producer_plugin)
max-irreversible-block-age = -1

# ID of producer controlled by this node (e.g. inita; may specify multiple times) (eosio::producer_plugin)
producer-name = coboeos

# Maximum size (in MB) of the chain state database (eosio::chain_plugin)
#chain-state-db-size-mb = 8192
chain-state-db-size-mb = 131072

# Limits the maximum time (in milliseconds) that is allowed a pushed transaction's code to execute before being considered invalid (eosio::producer_plugin)
max-transaction-time = 60000

# Increase allowed abi serialization time for slow test machines #4505
# https://github.com/EOSIO/eos/pull/4505
abi-serializer-max-time-ms = 5000

# The local IP and port to listen for incoming http connections; set blank to disable. (eosio::http_plugin)
http-server-address = 0.0.0.0:8888

http-max-response-time-ms = 1000

# number of blocks to retrieve in a chunk from any individual peer during synchronization (eosio::net_plugin)
sync-fetch-span = 100

# Can be 'any' or 'producers' or 'specified' or 'none'. If 'specified', peer-key must be specified at least once. If only 'producers', peer-key is not required. 'producers' and 'specified' may be combined. (eosio::net_plugin)
allowed-connection = any

# Maximum number of clients from which connections are accepted, use 0 for no limit (eosio::net_plugin)
max-clients = 150

http-threads = 20

# Key=Value pairs in the form <public-key>=<provider-spec>
# Where:
#    <public-key>       is a string form of a vaild EOSIO public key
#
#    <provider-spec>    is a string in the form <provider-type>:<data>
#
#    <provider-type>    is KEY, KEOSD, or SE
#
#    KEY:<data>         is a string form of a valid EOSIO private key which maps to the provided public key
#
#    KEOSD:<data>       is the URL where keosd is available and the approptiate wallet(s) are unlocked
#
#  (eosio::producer_plugin)
signature-provider = EOS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV=KEY:5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3

# Use to indicate that the RPC responses will not use ABIs.
# Failure to specify this option when there are no trace-rpc-abi configuations will result in an Error.
# This option is mutually exclusive with trace-rpc-api (eosio::trace_api_plugin)
trace-no-abis = true

# Plugin(s) to enable, may be specified multiple times
plugin = eosio::producer_plugin
plugin = eosio::producer_api_plugin
plugin = eosio::chain_plugin
plugin = eosio::chain_api_plugin
plugin = eosio::http_plugin
#plugin = eosio::state_history_plugin
plugin = eosio::net_plugin
plugin = eosio::net_api_plugin
plugin = eosio::trace_api_plugin

# https://validate.eosnation.io/eos/reports/config.html
# bp4dex223dao: US, United States
p2p-peer-address = p2p.dex223.io:9876

# cryptolions1: DE, Germany-Finland
p2p-peer-address = p2p.eos.cryptolions.io:9876

# eosamsterdam: DE, eos01
p2p-peer-address = mainnet.eosamsterdam.net:9876

# eosauthority: GB, London
p2p-peer-address = node869-mainnet.eosauthority.com:9393

# eosnationftw: CA, Canada
p2p-peer-address = eos.seed.eosnation.io:9876

# eosriobrazil: BR, Rio de Janeiro
p2p-peer-address = p2p.mainnet.eosrio.io:9876

# ivote4eosusa: US, Greenville,SC,USA
p2p-peer-address = eos.p2p.eosusa.io:9882
lcgogo commented 1 day ago

As my monitor, when the disk tps reach 15000 or more (by command sar -d 4 -h), the response is slower.

Now, I use AWS EBS io1 which has 400GiB size and 20000 iops.

sar -d 4 -h | grep -E 'tps|nvme3n'
Average:          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
06:11:37      4795.25     97.9M      0.0k      0.0k     20.9k      2.71      0.57    100.0% nvme3n1
06:11:41      3272.75     65.8M      0.0k      0.0k     20.6k      1.80      0.55     97.5% nvme3n1
06:11:45      3190.50    102.7M      2.3M      0.0k     33.7k      2.07      0.65     99.3% nvme3n1
06:11:49     19219.25    101.8M     68.7M      0.0k      9.1k     13.47      0.70    100.0% nvme3n1
06:11:53     18737.75     73.0M     70.3M      0.0k      7.8k     13.13      0.70    100.0% nvme3n1
06:11:57     17381.50     29.9M     71.5M      0.0k      6.0k     12.34      0.71    100.0% nvme3n1
06:12:01     16995.50     12.7M     70.1M      0.0k      5.0k     12.07      0.71    100.0% nvme3n1
06:12:05     16919.50      6.7M     75.8M      0.0k      5.0k     12.00      0.71    100.0% nvme3n1
06:12:09     17088.25     10.2M     82.2M      0.0k      5.5k     12.14      0.71    100.0% nvme3n1
06:12:13     17024.00      4.4M     69.2M      0.0k      4.4k     12.11      0.71     82.3% nvme3n1
06:12:17       165.25      7.5M      0.0k      0.0k     46.3k      0.11      0.67     10.7% nvme3n1
06:12:21       257.50      8.9M      0.0k      0.0k     35.2k      0.16      0.62     12.7% nvme3n1
06:12:25       113.75      4.7M      0.0k      0.0k     42.2k      0.08      0.68      8.0% nvme3n1
06:12:29       159.00      6.0M      0.0k      0.0k     38.4k      0.10      0.65     10.3% nvme3n1
06:12:33       194.75      8.1M      0.0k      0.0k     42.8k      0.13      0.67     13.5% nvme3n1
06:12:37       278.25      9.5M      6.0k      0.0k     35.0k      0.18      0.65     13.8% nvme3n1
06:12:41        98.50      3.1M      0.0k      0.0k     32.3k      0.06      0.63      6.6% nvme3n1
06:12:45      1712.50      3.1M      7.2M      0.0k      6.2k      1.21      0.71     17.9% nvme3n1
06:12:49        52.75      1.6M      0.0k      0.0k     30.5k      0.03      0.61      4.1% nvme3n1
06:12:53        82.25      3.0M      0.0k      0.0k     36.9k      0.05      0.65      6.6% nvme3n1
06:12:57       206.25      8.7M      0.0k      0.0k     43.2k      0.13      0.64     10.9% nvme3n1
06:13:01       128.25      5.1M      0.0k      0.0k     41.1k      0.09      0.68      9.1% nvme3n1
06:13:05       508.25     13.6M      8.0k      0.0k     27.4k      0.30      0.59     19.4% nvme3n1
06:13:09      3866.25    100.3M      0.0k      0.0k     26.6k      2.23      0.58     99.1% nvme3n1
06:13:13      3955.25    101.8M      0.0k      0.0k     26.4k      2.26      0.57    100.0% nvme3n1
06:13:17     15750.00    101.6M     46.9M      0.0k      9.7k     10.58      0.67    100.0% nvme3n1
06:13:21      9752.00    101.0M     23.6M      0.0k     13.1k      6.38      0.65    100.0% nvme3n1
06:13:25      9652.00    102.8M     22.0M      0.0k     13.2k      6.25      0.65    100.0% nvme3n1
06:13:29      5278.25    103.4M      5.9M      0.0k     21.2k      3.25      0.62    100.0% nvme3n1
06:13:33      3765.75     99.7M      0.0k      0.0k     27.1k      2.21      0.59    100.0% nvme3n1
06:13:37      3850.25    101.6M      2.0k      0.0k     27.0k      2.25      0.58    100.1% nvme3n1
06:13:41      3931.50    102.5M      0.0k      0.0k     26.7k      2.25      0.57     99.9% nvme3n1
06:13:45      3995.00    102.3M      0.0k      0.0k     26.2k      2.27      0.57    100.0% nvme3n1
06:13:49      3817.25    102.6M      0.0k      0.0k     27.5k      2.18      0.57    100.0% nvme3n1
06:13:53      4018.75    103.5M     20.0k      0.0k     26.4k      2.27      0.57    100.1% nvme3n1
06:13:57      7332.50    106.1M     13.5M      0.0k     16.7k      4.61      0.63    100.0% nvme3n1
06:14:01     20858.25    103.5M     67.7M      0.0k      8.4k     14.13      0.68    100.0% nvme3n1
06:14:05     20691.00    103.8M     68.8M      0.0k      8.5k     14.10      0.68    100.0% nvme3n1
06:14:09     15516.50    103.0M     46.4M      0.0k      9.9k     10.42      0.67     99.9% nvme3n1
06:14:13      3990.50    105.4M      0.0k      0.0k     27.1k      2.27      0.57    100.1% nvme3n1
06:14:17      4075.00    101.6M      0.0k      0.0k     25.5k      2.33      0.57    100.0% nvme3n1
06:14:21      4215.75    103.2M      0.0k      0.0k     25.1k      2.38      0.56    100.0% nvme3n1
06:14:25      4030.25     99.7M      0.0k      0.0k     25.3k      2.36      0.59    100.0% nvme3n1
06:14:29      4156.75    104.6M      0.0k      0.0k     25.8k      2.34      0.56     99.9% nvme3n1
06:14:33      4033.50    101.1M      0.0k      0.0k     25.7k      2.33      0.58    100.1% nvme3n1
06:14:37      4005.25    100.7M      2.0k      0.0k     25.7k      2.32      0.58    100.0% nvme3n1
06:14:41     14700.75    102.9M     43.1M      0.0k     10.2k      9.82      0.67    100.0% nvme3n1
06:14:45     20975.25    102.1M     67.9M      0.0k      8.3k     14.22      0.68    100.0% nvme3n1
06:14:49     20332.00    101.3M     65.6M      0.0k      8.4k     13.66      0.67    100.0% nvme3n1
06:14:53      4915.75    103.6M      0.0k      0.0k     21.6k      2.69      0.55     99.9% nvme3n1
06:14:57      4839.00     99.2M      0.0k      0.0k     21.0k      2.67      0.55    100.0% nvme3n1
heifner commented 1 day ago

Unrelated

lcgogo commented 11 hours ago
  • database-map-mode = mapped_private

After add database-map-mode = mapped_private, the node failed to start with followed err.

info  2024-12-03T10:48:18.384 nodeos    main.cpp:165                  operator()           ] nodeos version v1.0.3 v1.0.3-c954fe87382b62716815d9e3fcc11482bd1d0c17
info  2024-12-03T10:48:18.384 nodeos    main.cpp:69                   log_non_default_opti ] Non-default options: data-dir = /data/eos, config-dir = /data/eos, verbose-http-errors, http-validate-host = false, access-control-allow-origin = *, agent-name = "Cobo EOS", enable-stale-production = true, max-irreversible-block-age = -1, producer-name = coboeos, chain-state-db-size-mb = 131072, max-transaction-time = 60000, abi-serializer-max-time-ms = 5000, http-server-address = 0.0.0.0:8888, http-max-response-time-ms = 1000, sync-fetch-span = 100, allowed-connection = any, max-clients = 150, http-threads = 20, signature-provider = EOS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV=KEY:***, trace-no-abis = true, database-map-mode = mapped_private, plugin = eosio::producer_plugin, plugin = eosio::producer_api_plugin, plugin = eosio::chain_plugin, plugin = eosio::chain_api_plugin, plugin = eosio::http_plugin, plugin = eosio::net_plugin, plugin = eosio::net_api_plugin, plugin = eosio::trace_api_plugin, p2p-peer-address = p2p.dex223.io:9876, p2p-peer-address = p2p.eos.cryptolions.io:9876, p2p-peer-address = mainnet.eosamsterdam.net:9876, p2p-peer-address = node869-mainnet.eosauthority.com:9393, p2p-peer-address = eos.seed.eosnation.io:9876, p2p-peer-address = p2p.mainnet.eosrio.io:9876, p2p-peer-address = eos.p2p.eosusa.io:9882
error 2024-12-03T10:48:18.384 nodeos    main.cpp:224                  main                 ] 13 N5boost12interprocess22interprocess_exceptionE: Cannot allocate memory
rethrow Cannot allocate memory:
    {"what":"Cannot allocate memory"}
    nodeos  chain_plugin.cpp:1109 plugin_initialize
heifner commented 8 hours ago

That means you have less than 131072 MB of memory available. If you don't have enough memory for the database to be in memory then you will see higher disk utilization.