search

nanocurrency / nano-node

Nano is digital currency. Its ticker is: XNO and its currency symbol is: Ӿ
https://nano.org
BSD 3-Clause "New" or "Revised" License
3.47k stars 786 forks source link

Node V14.2 crash, before crash theare are more than 2M blocks in quere, may be dead lock #966

Closed huye0728 closed 5 years ago

huye0728 commented 6 years ago

root@US1:~/rai-14.2.0-Linux/bin#
root@US1:~/rai-14.2.0-Linux/bin# ./rai_node --daemon
Killed
root@US1:~/rai-14.2.0-Linux/bin#
root@US1:~/rai-14.2.0-Linux/bin# root@US1:~/rai-14.2.0-Linux/bin# dmesg -T| grep -E -i -B100 'killed process' [Wed Jun 13 05:10:09 2018] audit: type=1400 audit(1528866610.196:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=555 comm="apparmor_parser" [Wed Jun 13 05:10:09 2018] audit: type=1400 audit(1528866610.204:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=556 comm="apparmor_parser" [Wed Jun 13 05:10:09 2018] audit: type=1400 audit(1528866610.204:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=556 comm="apparmor_parser" [Wed Jun 13 05:10:09 2018] audit: type=1400 audit(1528866610.204:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=556 comm="apparmor_parser" [Wed Jun 13 05:10:09 2018] audit: type=1400 audit(1528866610.204:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=556 comm="apparmor_parser" [Wed Jun 13 05:10:09 2018] audit: type=1400 audit(1528866610.204:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=557 comm="apparmor_parser" [Wed Jun 13 05:10:10 2018] new mount options do not match the existing superblock, will be ignored [Wed Jun 13 05:10:11 2018] virtio_net virtio0 ens3: renamed from eth0 [Wed Jun 13 05:10:12 2018] IPv6: ADDRCONF(NETDEV_UP): ens3: link is not ready [Wed Jun 13 05:10:12 2018] IPv6: ADDRCONF(NETDEV_CHANGE): ens3: link becomes ready [Wed Jun 27 15:53:56 2018] hrtimer: interrupt took 19192628 ns [Sun Jul 1 06:25:29 2018] Core dump to |/usr/share/apport/apport 27113 6 0 1 27113 pipe failed [Tue Jul 3 11:18:14 2018] rai_node invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 [Tue Jul 3 11:18:14 2018] rai_node cpuset=/ mems_allowed=0 [Tue Jul 3 11:18:14 2018] CPU: 0 PID: 28105 Comm: rai_node Not tainted 4.15.0-22-generic #24-Ubuntu [Tue Jul 3 11:18:14 2018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014 [Tue Jul 3 11:18:14 2018] Call Trace: [Tue Jul 3 11:18:14 2018] dump_stack+0x63/0x8b [Tue Jul 3 11:18:14 2018] dump_header+0x71/0x285 [Tue Jul 3 11:18:14 2018] ? security_capable_noaudit+0x4b/0x70 [Tue Jul 3 11:18:14 2018] oom_kill_process+0x220/0x440 [Tue Jul 3 11:18:14 2018] out_of_memory+0x2d1/0x4f0 [Tue Jul 3 11:18:14 2018] alloc_pages_slowpath+0xa93/0xe60 [Tue Jul 3 11:18:14 2018] ? alloc_pages_current+0x6a/0xe0 [Tue Jul 3 11:18:14 2018] alloc_pages_nodemask+0x263/0x280 [Tue Jul 3 11:18:14 2018] alloc_pages_current+0x6a/0xe0 [Tue Jul 3 11:18:14 2018] page_cache_alloc+0x81/0xa0 [Tue Jul 3 11:18:14 2018] filemap_fault+0x389/0x6f0 [Tue Jul 3 11:18:14 2018] ? page_add_file_rmap+0x134/0x180 [Tue Jul 3 11:18:14 2018] ? filemap_map_pages+0x181/0x390 [Tue Jul 3 11:18:14 2018] ext4_filemap_fault+0x31/0x44 [Tue Jul 3 11:18:14 2018] do_fault+0x24/0xf0 [Tue Jul 3 11:18:14 2018] handle_pte_fault+0x8b5/0xdb0 [Tue Jul 3 11:18:14 2018] handle_mm_fault+0x47b/0x5c0 [Tue Jul 3 11:18:14 2018] handle_mm_fault+0xb1/0x1f0 [Tue Jul 3 11:18:14 2018] do_page_fault+0x250/0x4d0 [Tue Jul 3 11:18:14 2018] ? __schedule+0x29f/0x8b0 [Tue Jul 3 11:18:14 2018] do_page_fault+0x2e/0xe0 [Tue Jul 3 11:18:14 2018] ? async_page_fault+0x2f/0x50 [Tue Jul 3 11:18:14 2018] do_async_page_fault+0x51/0x80 [Tue Jul 3 11:18:14 2018] async_page_fault+0x45/0x50 [Tue Jul 3 11:18:14 2018] RIP: 0033:0x70acbc [Tue Jul 3 11:18:14 2018] RSP: 002b:00007f0ec2d86ba0 EFLAGS: 00010246 [Tue Jul 3 11:18:14 2018] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00007f0ec3d8b020 [Tue Jul 3 11:18:14 2018] RDX: 00007eef870a1000 RSI: 00000000000cb0a1 RDI: 00007f0ec3d8b010 [Tue Jul 3 11:18:14 2018] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [Tue Jul 3 11:18:14 2018] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [Tue Jul 3 11:18:14 2018] R13: 00007f0ec2d86bc0 R14: 00007f0ec2d86e80 R15: 0000000000000037 [Tue Jul 3 11:18:14 2018] Mem-Info: [Tue Jul 3 11:18:14 2018] active_anon:218283 inactive_anon:62 isolated_anon:0 active_file:10 inactive_file:12 isolated_file:0 unevictable:1352 dirty:0 writeback:0 unstable:0 slab_reclaimable:4101 slab_unreclaimable:6163 mapped:1018 shmem:158 pagetables:3935 bounce:0 free:12159 free_pcp:30 free_cma:0 [Tue Jul 3 11:18:14 2018] Node 0 active_anon:873132kB inactive_anon:248kB active_file:40kB inactive_file:48kB unevictable:5408kB isolated(anon):0kB isolated(file):0kB mapped:4072kB dirty:0kB writeback:0kB shmem:632kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [Tue Jul 3 11:18:14 2018] Node 0 DMA free:4392kB min:752kB low:940kB high:1128kB active_anon:10164kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:16kB pagetables:60kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [Tue Jul 3 11:18:14 2018] lowmem_reserve[]: 0 911 911 911 911 [Tue Jul 3 11:18:14 2018] Node 0 DMA32 free:44244kB min:44300kB low:55372kB high:66444kB active_anon:862968kB inactive_anon:248kB active_file:72kB inactive_file:44kB unevictable:5408kB writepending:0kB present:1032048kB managed:993140kB mlocked:5408kB kernel_stack:1840kB pagetables:15680kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB [Tue Jul 3 11:18:14 2018] lowmem_reserve[]: 0 0 0 0 0 [Tue Jul 3 11:18:14 2018] Node 0 DMA: 34kB (ME) 128kB (UME) 2416kB (UME) 2032kB (UME) 1764kB (UME) 9128kB (UME) 2256kB (UE) 1512kB (M) 01024kB 02048kB 04096kB = 4396kB [Tue Jul 3 11:18:14 2018] Node 0 DMA32: 4194kB (UME) 2518kB (UME) 15516kB (ME) 18032kB (MEH) 11964kB (UMEH) 47128kB (MEH) 15256kB (ME) 7512kB (UME) 91024kB (UME) 12048kB (M) 04096kB = 44244kB [Tue Jul 3 11:18:14 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Tue Jul 3 11:18:14 2018] 1197 total pagecache pages [Tue Jul 3 11:18:14 2018] 0 pages in swap cache [Tue Jul 3 11:18:14 2018] Swap cache stats: add 0, delete 0, find 0/0 [Tue Jul 3 11:18:14 2018] Free swap = 0kB [Tue Jul 3 11:18:14 2018] Total swap = 0kB [Tue Jul 3 11:18:14 2018] 262010 pages RAM [Tue Jul 3 11:18:14 2018] 0 pages HighMem/MovableOnly [Tue Jul 3 11:18:14 2018] 9748 pages reserved [Tue Jul 3 11:18:14 2018] 0 pages cma reserved [Tue Jul 3 11:18:14 2018] 0 pages hwpoisoned [Tue Jul 3 11:18:14 2018] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [Tue Jul 3 11:18:14 2018] [ 379] 0 379 45431 1527 401408 0 0 systemd-journal [Tue Jul 3 11:18:14 2018] [ 396] 0 396 26476 153 98304 0 0 lvmetad [Tue Jul 3 11:18:14 2018] [ 399] 0 399 6344 59 90112 0 0 iscsid [Tue Jul 3 11:18:14 2018] [ 401] 0 401 6470 1317 98304 0 -17 iscsid [Tue Jul 3 11:18:14 2018] [ 407] 0 407 11357 748 114688 0 -1000 systemd-udevd [Tue Jul 3 11:18:14 2018] [ 505] 62583 505 35477 407 184320 0 0 systemd-timesyn [Tue Jul 3 11:18:14 2018] [ 508] 101 508 17683 686 180224 0 0 systemd-resolve [Tue Jul 3 11:18:14 2018] [ 566] 102 566 65759 334 163840 0 0 rsyslogd [Tue Jul 3 11:18:14 2018] [ 568] 0 568 7083 547 98304 0 0 atd [Tue Jul 3 11:18:14 2018] [ 571] 103 571 12514 510 143360 0 -900 dbus-daemon [Tue Jul 3 11:18:14 2018] [ 589] 0 589 72475 339 204800 0 0 accounts-daemon [Tue Jul 3 11:18:14 2018] [ 591] 0 591 7898 533 114688 0 0 cron [Tue Jul 3 11:18:14 2018] [ 592] 0 592 43196 2561 225280 0 0 networkd-dispat [Tue Jul 3 11:18:14 2018] [ 595] 0 595 137004 2265 196608 0 -900 snapd [Tue Jul 3 11:18:14 2018] [ 596] 0 596 17645 495 176128 0 0 systemd-logind [Tue Jul 3 11:18:14 2018] [ 597] 0 597 161113 668 159744 0 0 lxcfs [Tue Jul 3 11:18:14 2018] [ 607] 0 607 4120 387 73728 0 0 agetty [Tue Jul 3 11:18:14 2018] [ 620] 0 620 72217 696 208896 0 0 polkitd [Tue Jul 3 11:18:14 2018] [ 874] 100 874 20003 580 167936 0 0 systemd-network [Tue Jul 3 11:18:14 2018] [ 1410] 0 1410 19149 638 188416 0 0 systemd [Tue Jul 3 11:18:14 2018] [ 1417] 0 1417 48900 610 274432 0 0 (sd-pam) [Tue Jul 3 11:18:14 2018] [ 1545] 0 1545 8034 208 98304 0 0 screen [Tue Jul 3 11:18:14 2018] [ 1546] 0 1546 5471 500 90112 0 0 bash [Tue Jul 3 11:18:14 2018] [24773] 0 24773 18074 497 180224 0 -1000 sshd [Tue Jul 3 11:18:14 2018] [28100] 0 28100 33889309 208242 12648448 0 0 rai_node [Tue Jul 3 11:18:14 2018] Out of memory: Kill process 28100 (rai_node) score 812 or sacrifice child [Tue Jul 3 11:18:14 2018] Killed process 28100 (rai_node) total-vm:135557236kB, anon-rss:832968kB, file-rss:0kB, shmem-rss:0kB root@US1:~/rai-14.2.0-Linux/bin# -------------------------------------logs when crash---------------------------------------------------------- [2018-07-03 11:18:02.348963]: Block 60D682BDFEEF620D5B14A76EB6309FC679B37BB0D7573A947F19AFD564D1FD5F was republished to peers [2018-07-03 11:18:05.013462]: Aborting frontier req because it was too slow [2018-07-03 11:18:05.093999]: frontier_req failed, reattempting [2018-07-03 11:18:05.353178]: frontier_req failed, reattempting [2018-07-03 11:18:05.578610]: frontier_req failed, reattempting [2018-07-03 11:18:05.708767]: frontier_req failed, reattempting [2018-07-03 11:18:12.109756]: frontier_req failed, reattempting [2018-07-03 11:18:13.947417]: 2307756 blocks in processing queue [2018-07-03 11:18:14.293160]: Block 0EFD8A3F4AF959F9D5F63D2469AC6B21AC195EA29234E96F0BFB1791CE4B2907 was republished to peers [2018-07-03 11:18:14.377831]: Block 1CC44DCF1B21AA7FE080C3767CF4255E8D8B70FD94FF26684B7F8D62F12EC933 was republished to peers [2018-07-03 11:18:14.376220]: Broadcasting confirm req for block E3DFB65D0B0A3F45870A661CC8B7EB37EF69900640C986F82A09859F530F89A3 to 4 representatives [2018-07-03 11:18:14.441656]: Block CD52E96E97D823652831F4549330069B42D42DD56AAA01CAA7544B893A0E6BB2 was republished to peers [2018-07-03 11:18:14.499707]: Block FA459286398DE5B9CC85C3030D0CC2380985E8B553D811097D765346996BA180 was republished to peers [2018-07-03 11:18:14.494486]: Broadcasting confirm req for block BB92C7DA352988F85733F4280D345682E46F2E6B2140B1776020D1BA404CC671 to 4 representatives [2018-07-03 11:18:14.561073]: Block 215095E432325ABCC5C6050CAE0638AB9AC04316D6850A57682A1A6BF63628F3 was republished to peers [2018-07-03 11:18:14.636184]: Block E3DFB65D0B0A3F45870A661CC8B7EB37EF69900640C986F82A09859F530F89A3 was republished to peers [2018-07-03 11:18:14.678075]: Broadcasting confirm req for block F53E1BA1C289A5FC69731C5A26E3B0C297919776706CF26A3A92A93BC198633B to 5 representatives [2018-07-03 11:18:14.696408]: Block 75B7C9D6A7D915BB4ED68D836DC7C32A02782122825D3CAEC3C121C6DC398420 was republished to peers [2018-07-03 11:18:14.737780]: Block A66052A1AAAB7C7802198ECEB93EC2D63C3356272A8A97DCBF28F85B0BE64D3D was republished to peers [2018-07-03 11:18:14.800493]: Block AAB08FD75DBA104DE6B132FD4165678DE9E47F78AB68D87AA4BB6BE5FCB7AF1E was republished to peers [2018-07-03 11:18:14.868243]: Block BB92C7DA352988F85733F4280D345682E46F2E6B2140B1776020D1BA404CC671 was republished to peers [2018-07-03 11:18:14.864651]: Broadcasting confirm req for block 210E4890AFDAAC28E76506C120F7002CF8AFDB426BF563849B05FBE619A412D8 to 6 representatives [2018-07-03 11:18:14.931672]: Block 47898F8F743E8A99E73692E46958B855FB5B39946A727FD6C4199238AA87CE51 was republished to peers [2018-07-03 11:18:14.997010]: Block 6E3A5B1B9DF518CC88D8286BDCA4C42C1C108C64F1FB293F2440A9B20C249AA5 was republished to peers [2018-07-03 11:18:15.062496]: Block EB65AF33828497024F3154E4FC81E3DA80F7E083FD1C39B262D15AF1F854F90A was republished to peers [2018-07-03 11:18:15.060442]: Broadcasting confirm req for block 88F2106D48A7E0390DEC8FA49A211E5D95DA53AFC4E7DEB247CEF932FFBFD3C9 to 7 representatives [2018-07-03 11:18:15.134234]: Block 241992CB63E9143BC2799F55D86486E2CAA67883B6D6377D2C0072723B45C115 was republished to peers [2018-07-03 11:18:15.156748]: Broadcasting confirm req for block 80FD6954A745FFA8F2BEBBB1D1A800A41AD55B2A5FF71732F65ABEC6D73CFD4C to 5 representatives [2018-07-03 11:18:15.188421]: Block 09A33CE8F9A1E4440CB320F747FE45BB8BD8314F3DB28F7652FD13711070B42F was republished to peers [2018-07-03 11:18:15.187899]: Broadcasting confirm req for block EE54DC5A89FD43C35DADC90DE65613E8F2601546C4DD13B10D25C9DB85D45FFC to 6 representatives [2018-07-03 11:18:15.249471]: Block 5BC150990BBBEC873A1934CADD26778F089A699B1D785C9CB86A757C7320A774 was republished to peers [2018-07-03 11:18:15.307891]: Block F53E1BA1C289A5FC69731C5A26E3B0C297919776706CF26A3A92A93BC198633B was republished to peers [2018-07-03 11:18:15.380065]: Block BDB70899616520563B4D6D7FACAABD11F614F44A9E7A86650EDB500A6F4B113C was republished to peers [2018-07-03 11:18:15.380002]: Broadcasting confirm req for block C02B2918EB4B3C7431834B9F9D12984049E62C12555F2D502A0BC995410D0A6A to 5 representatives [2018-07-03 11:18:15.450194]: Block CAB27252A6B86BE01D7806F7A3C8F3752DB3B40F9B51BF967C0BB2A0FAC16D13 was republished to peers [2018-07-03 11:18:15.517043]: Block 8B0794DDF2A61AB408B93D345D4F63CD82C49C8549ED47CEC08C408334283667 was republished to peers [2018-07-03 11:18:15.557384]: Broadcasting confirm req for block 33DE9EAA5E4BAA76556E42AE8F1876BE56791EE327ED6A5F806CB2B176B735EA to 6 representatives [2018-07-03 11:18:15.572996]: Block 4FF459CC1B6FDF78300FA60FE26418BE808CAA3EA718E09648538A0F93BEB579 was republished to peers [2018-07-03 11:18:15.603987]: Block F5156B178009718DFC95D9DCF37F25F973A0618129421E2BD243F2ED2BD99507 was republished to peers [2018-07-03 11:18:15.603912]: Broadcasting confirm req for block BFFB5DD7DBFEA58B47E33D74904553A5A11BFA4911C634C40600A812EB45CCFF to 5 representatives [2018-07-03 11:18:15.666449]: Broadcasting confirm req for block 26FFBD10CA25C0A5619C99E3755D69A49D7EB5AA58A196DD07A5DCF01A423421 to 7 representatives [2018-07-03 11:18:15.669418]: Block 2FB947CC71AE254071659A426CA909B19DF67C6C35312CCD1226417B85B5785D was republished to peers [2018-07-03 11:18:15.726439]: Block A7196DA5BAA5EFEDA1F7070D0BA59C95C9AC29D9FDD7BBC297961EF6C03339E5 was republished to peers [2018-07-03 11:18:15.750950]: Broadcasting confirm req for block 7A12C7A6EA64CD59719286670561A2FE0BBE7500741AEBCBB61095ABD4C9CD19 to 5 representatives [2018-07-03 11:18:15.803657]: Block 210E4890AFDAAC28E76506C120F7002CF8AFDB426BF563849B05FBE619A412D8 was republished to peers [2018-07-03 11:18:15.903444]: Block DBC080CD00A4D9987B5FAF1F1B9BFE679E59393E52B35B5BC5CA9EA20EE09BA5 was republished to peers [2018-07-03 11:18:15.986279]: Broadcasting confirm req for block A32CF6116CA0D6758996A2D094D02798109F60717C61A0B1C3CD997E624B579C to 6 representatives [2018-07-03 11:18:15.986224]: Block 284035C6EFBA952051282448F26AAB8B7AAE9F08B1C652715D3E6148891778F9 was republished to peers [2018-07-03 11:18:16.078018]: Block 5AC304895D4AA901C47D4D9A61126BFEE0CF50E5876BD62B523D2DAD2EFC05E3 was republished to peers [2018-07-03 11:18:16.109870]: Broadcasting confirm req for block B8BCDAC8D52DE81C8CC362C3E62B7283CA807E3B4A1A376C83588BD48C0091EA to 7 representatives [2018-07-03 11:18:16.153613]: Block A243527FDBBFEBEB502F4121403B9CFB4D1EA1180380C82C68281CD1EDDF54FC was republished to peers [2018-07-03 11:18:16.174704]: Broadcasting confirm req for block ADC950289DEE74AE24E493C984D4C078379F777E0FE2F84AB12FFFF149BF98F6 to 6 representatives [2018-07-03 11:18:16.217772]: Block C8442766B4CCBBE36189A9635FDD8E256F97F60934E34F8C5DB8FDF51D560C7E was republished to peers [2018-07-03 11:18:16.273870]: Block E3895E72AA6396ECE8450FF80E1666C09795C76871A3AF75213FEB6C28596DBB was republished to peers [2018-07-03 11:18:16.260375]: Broadcasting confirm req for block 9F3C6875BD4FF52AFF2084E3DFA07CD814BC4423FEFB7EEB2E8D582908B08EC0 to 5 representatives [2018-07-03 11:18:16.279345]: Block 7EE99A00338927D4D2E74D806EA0DC726205F521C27BEFC4EB202022D8FDFA31 was republished to peers [2018-07-03 11:18:16.279609]: Block 8A1DABEA7892EA7C7A2F4EC63E7A2840AE69CA60ED25C4F0C025069B5F7E2AD2 was republished to peers [2018-07-03 11:18:16.279851]: Block 88F2106D48A7E0390DEC8FA49A211E5D95DA53AFC4E7DEB247CEF932FFBFD3C9 was republished to peers [2018-07-03 11:18:16.280127]: Block BBDCE4E5B4B05E1F09022FB8C58E513D2F63694B22866505E110E5D6AE407FDD was republished to peers [2018-07-03 11:18:16.280417]: Block CFAE26359BA2D866C63052B068304F97425F69AFA9EB8C1D138E0CEB17D2DAE4 was republished to peers [2018-07-03 11:18:16.280704]: Block 80FD6954A745FFA8F2BEBBB1D1A800A41AD55B2A5FF71732F65ABEC6D73CFD4C was republished to peers [2018-07-03 11:18:16.281098]: Block EE54DC5A89FD43C35DADC90DE65613E8F2601546C4DD13B10D25C9DB85D45FFC was republished to peers [2018-07-03 11:18:16.315426]: Broadcasting confirm req for block FB2F76D16B84BFA6F81CC4DAEEEED9908F5047254D260F1CC49E4982B7EEF818 to 3 representatives [2018-07-03 11:18:16.363474]: Broadcasting confirm req for block D34C9C93A04D9EDCBD56A213D2206D110163F79D3709B880D070F78017152653 to 6 representatives [2018-07-03 11:18:16.355128]: Block 33493B35353660DE0EA496CF179E674C3B2154244C7E46515EFA7D0E0B2A04FD was republished to peers [2018-07-03 11:18:16.416739]: Broadcasting confirm req for block 97DEC93A8BFF3E5FA0AC10BBEA145F119353C69BD94F025DE4DDFC3ED60F24A4 to 6 representatives [2018-07-03 11:18:16.397765]: Block 4957EC0BE7236FA52743940DE3D0EF30FF04A3191C4C746190BB76185011D065 was republished to peers [2018-07-03 11:18:16.465436]: Broadcasting confirm req for block 0ED2F4AA1A273E28252B0BB788C466A99C785490C19B54491E276D2F0AF810CC to 7 representatives [2018-07-03 11:18:16.444628]: Block FF9380BCB25D8BE44ECAFDE8EDB8F4DCB12A70E35AFDFB330018105FD12C5D6F was republished to peers [2018-07-03 11:18:16.494662]: Block CC02B8F55377BA2AE610AB4AC692F0C6E9475F69BC667947F5AFB0F6AFD11B27 was republished to peers [2018-07-03 11:18:16.554464]: Block 469B960029EF3D47BFE82AD2CE707316922FE872F5F0A98DA8EFC520DA336401 was republished to peers [2018-07-03 11:18:16.628758]: Block 8EAC5773DA2DFB5F1C81847004C8A1E32D4607972DEECE2793708FD00701B0A9 was republished to peers [2018-07-03 11:18:16.696266]: Block 3478FF5610165D952120BDCEC8C779D832B6EE94B328ED66B40FCE69F5B65D56 was republished to peers [2018-07-03 11:18:16.783355]: Block 8F98450EAD27070C127E3380B2B09F70FD4CF5C84FE0C7258DA2BCEDA9EDE7E3 was republished to peers [2018-07-03 11:18:16.863285]: Block C02B2918EB4B3C7431834B9F9D12984049E62C12555F2D502A0BC995410D0A6A was republished to peers [2018-07-03 11:18:16.898375]: Broadcasting confirm req for block A131A04A01AFCEF90AB54C505B05E198C8DD24992F01A30ABA2A8DD88AD31FA9 to 6 representatives [2018-07-03 11:18:16.910541]: Block 5E8E0E685BD90671DAC2E2368414181E6C1CB84B35596B22463295A7167F423C was republished to peers [2018-07-03 11:18:16.912241]: Block 26EDBE7F188D8D285A9CD8AEE3183050E3DFCFE6F87922E2473500B6613AEA7B was republished to peers [2018-07-03 11:18:16.912531]: Block 46EA659CFDED9FC67FC8149997E60637DC1D29DC9712691C63D3E464A9442F8A was republished to peers [2018-07-03 11:18:16.912757]: Block C5102B7FD6F7965D42BD438AFCC5A14C0056773364ED1597907F65C99F16B73B was republished to peers [2018-07-03 11:18:16.912939]: Block 81E1D39B37A1C8D884A9E8FAE432F4FFD75C6ECB6D1A4BD830C57B31777ABF23 was republished to peers [2018-07-03 11:18:16.913146]: Block 30F85DA1B3B499A7E48BA31CCD2E6C1D1E8403DFC98061B4FCF8262036804B60 was republished to peers [2018-07-03 11:18:16.913338]: Block 33DE9EAA5E4BAA76556E42AE8F1876BE56791EE327ED6A5F806CB2B176B735EA was republished to peers [2018-07-03 11:18:16.913551]: Block BFFB5DD7DBFEA58B47E33D74904553A5A11BFA4911C634C40600A812EB45CCFF was republished to peers [2018-07-03 11:18:16.913741]: Block 26FFBD10CA25C0A5619C99E3755D69A49D7EB5AA58A196DD07A5DCF01A423421 was republished to peers [2018-07-03 11:18:16.913970]: Block 7A12C7A6EA64CD59719286670561A2FE0BBE7500741AEBCBB61095ABD4C9CD19 was republished to peers [2018-07-03 11:18:16.914826]: Block E28AA7D92D23B97208777CA37C76909AB6CB83C814B81401F12EE8A89FB955CC was republished to peers [2018-07-03 11:18:16.916796]: Block CA218792495EA4B43DFB97283012D50B1EB6FEA362FE242889161410CCB49AFD was republished to peers [2018-07-03 11:18:16.917032]: Block C78252D6F3E869A4C1957CBD03700790E9E582E6807C1AA4A6C07AD2A02EDDF5 was republished to peers [2018-07-03 11:18:16.917215]: Block 66CBCFAC26C935FA282175F17870E070C1208EA846B4B0506B9498C4D275933B was republished to peers [2018-07-03 11:18:16.917562]: Block 02952E557A97D53997E7DD70D1773D5E4229CEA360E93F5B8B92583B46A90C4A was republished to peers [2018-07-03 11:18:16.917773]: Block D0E3B9E40022613F1147B7FFA813800D74B5A1EB49116A932205EB45F03A5BF1 was republished to peers [2018-07-03 11:18:16.918039]: Block 615518881D2CA56F490E0254BC11B9A4A68BC373FB5D5170586894EF9F0CDD21 was republished to peers [2018-07-03 11:18:16.926329]: Block E07E1E8F71CF4655CE0D5346DD3FE30710E06360F73EEDDAC8E806900EB4175D was republished to peers [2018-07-03 11:18:16.929605]: Block EEE4480C97C9E32A6A3E284CB2D3834359AF9D86DC7B7D6670B87CE914B79D9E was republished to peers [2018-07-03 11:18:16.929863]: Block 2926C858B9E9EE1C9EFF1C61C826174703448FE0BBC0F60AA76C27BED743B2AB was republished to peers [2018-07-03 11:18:16.930059]: Block A32CF6116CA0D6758996A2D094D02798109F60717C61A0B1C3CD997E624B579C was republished to peers [2018-07-03 11:18:16.930309]: Block B8BCDAC8D52DE81C8CC362C3E62B7283CA807E3B4A1A376C83588BD48C0091EA was republished to peers [2018-07-03 11:18:16.930534]: Block ADC950289DEE74AE24E493C984D4C078379F777E0FE2F84AB12FFFF149BF98F6 was republished to peers [2018-07-03 11:18:16.931579]: Block 9F3C6875BD4FF52AFF2084E3DFA07CD814BC4423FEFB7EEB2E8D582908B08EC0 was republished to peers [2018-07-03 11:18:16.932149]: Block FB2F76D16B84BFA6F81CC4DAEEEED9908F5047254D260F1CC49E4982B7EEF818 was republished to peers [2018-07-03 11:18:16.932392]: Block D34C9C93A04D9EDCBD56A213D2206D110163F79D3709B880D070F78017152653 was republished to peers [2018-07-03 11:18:16.932722]: Block 97DEC93A8BFF3E5FA0AC10BBEA145F119353C69BD94F025DE4DDFC3ED60F24A4 was republished to peers [2018-07-03 11:18:16.932970]: Block 0ED2F4AA1A273E28252B0BB788C466A99C785490C19B54491E276D2F0AF810CC was republished to peers [2018-07-03 11:18:16.933207]: Block D77A17494209D275301AC4CF492ED47C30F6A048924829DE9109DC4FDE9B9F90 was republished to peers [2018-07-03 11:18:16.933408]: Block 4A1BA0BBDB4D4851B15B2C61A22F5C485477E16956F86591542DECBF783C6E08 was republished to peers [2018-07-03 11:18:16.933764]: Block AD4AC6B4AEEF02B5EC0FB76E52B1E7DD7B6019AF80C6AFB6193B76A00B8F97E2 was republished to peers [2018-07-03 11:18:16.934007]: Block F19B858B9EF401CBCBB8AFA4FF92B53E820BB67BE6F3A62079ACD181948152F4 was republished to peers [2018-07-03 11:18:16.934192]: Block 930510BC5DC2D7175FBB5E9D8E8BEB35260D6FF7C40A7EC9600D0EE30CFFCC75 was republished to peers [2018-07-03 11:18:16.936827]: Block 6C08D7790BE83C3FDEE7D2C0659E18EEC6826158026469B7171CE9BB6909B1E9 was republished to peers [2018-07-03 11:18:16.937171]: Block 95927E174894E0B9D2F38ED96063F2239A9FDE42C759849A493929C8A4F7022C was republished to peers [2018-07-03 11:18:16.937393]: Block 9ED24BE3226EDEB6E3BB303EB062A6098EA50BBFF63694E069E14176C72174EA was republished to peers [2018-07-03 11:18:16.937716]: Block 949DF88E873C30B88A72BEF747572A2C433C7822605650C1F233E2918D4FFC37 was republished to peers [2018-07-03 11:18:16.937887]: Block 4EB90BFAB18DEABAE6D62D4073BE0F9F799B8A4F03B719D748D5FB7348CDFA5B was republished to peers [2018-07-03 11:18:16.943403]: Broadcasting confirm req for block 029F902BE7F64404D8B9EFBF15D4BDE3805555446E1352AC44C28C5E5F757EF2 to 7 representatives [2018-07-03 11:18:16.944320]: Block CBA45BD898BFC1CBB697EC3BBE0904BC6CEEF272485674524F6656CC0E919020 was republished to peers [2018-07-03 11:18:16.944507]: Block FD6ED01AD22B486E33AEC67B2F169B00DED9151215946EFC6EC7D541C775B95B was republished to peers [2018-07-03 11:18:16.945699]: Block C92D424632A2913EE374A82F9D4D893BDF6577B71F14E38B745E5C7D19ADD087 was republished to peers [2018-07-03 11:18:16.946008]: Block A17E05BEA1EAC4E7F067734F905A0127ECB9B3FD9CD4889A4684D8925A364326 was republished to peers [2018-07-03 11:18:16.946282]: Block A131A04A01AFCEF90AB54C505B05E198C8DD24992F01A30ABA2A8DD88AD31FA9 was republished to peers [2018-07-03 11:18:16.949590]: Block 029F902BE7F64404D8B9EFBF15D4BDE3805555446E1352AC44C28C5E5F757EF2 was republished to peers [2018-07-03 11:18:16.964734]: Block D6F8E5CF463BBACE94A3429E2E319334B31A849AC19AB1961D928BA946E5FAD2 was republished to peers [2018-07-03 11:18:16.964405]: Broadcasting confirm req for block D6F8E5CF463BBACE94A3429E2E319334B31A849AC19AB1961D928BA946E5FAD2 to 6 representatives [2018-07-03 11:18:16.972319]: Broadcasting confirm req for block C4B6242031749069569D92547AFC17DC7514FB96F0F8FDADE659488619B7D36A to 7 representatives [2018-07-03 11:18:16.973649]: Block C4B6242031749069569D92547AFC17DC7514FB96F0F8FDADE659488619B7D36A was republished to peers [2018-07-03 11:18:17.010851]: Broadcasting confirm req for block 3587424C6BCD03BEA673743F0860B73D708669C3D28373638026EF3ED8B38B2A to 5 representatives [2018-07-03 11:18:17.014038]: Block 8D1271FBCB8E7A60869F44CE835B2E57F18284BA432DABC879AC75712A9B3AEB was republished to peers [2018-07-03 11:18:17.042873]: Block 3587424C6BCD03BEA673743F0860B73D708669C3D28373638026EF3ED8B38B2A was republished to peers [2018-07-03 11:18:17.101565]: Block E4B5C858FD848B5EE1315A0064C527C292E4F50C52167D12A37393381AFE3814 was republished to peers [2018-07-03 11:18:17.171342]: Block 81F0F377532668365E47F265FBB4532029DA76788D3021DCB5054F8B02FEE19A was republished to peers [2018-07-03 11:18:17.193917]: Broadcasting confirm req for block E68C6E3B17A9E96DF987877F553D479A9BB324D7205B01BE75C110732D18ADC8 to 6 representatives [2018-07-03 11:18:17.219121]: Block 771F2BF8C2D4A9725AE357A21B00D10BF40380BA3A72B83999EBDB38A4148C36 was republished to peers [2018-07-03 11:18:17.259859]: 97 blocks have been unconfirmed averaging 127 announcements [2018-07-03 11:18:17.273168]: Block A94EEFC8CE4BC8D76AE2FD23413BA1841A7D2AF1E582083D7FD47EB138A7D8AE was republished to peers [2018-07-03 11:18:17.279379]: Block EA82535D4C73A4120816F0B9F63C62CBC5B3518D2B3CC04D8C55052EEC721467 was republished to peers [2018-07-03 11:18:17.279627]: Block E68C6E3B17A9E96DF987877F553D479A9BB324D7205B01BE75C110732D18ADC8 was republished to peers [2018-07-03 11:18:17.279867]: Block 60D682BDFEEF620D5B14A76EB6309FC679B37BB0D7573A947F19AFD564D1FD5F was republished to peers [2018-07-03 11:18:17.284300]: Block AF30D590E8F52A7AADD2B527B06ADAC5C4EDE9FAC1157BD3D01D3487B0FFA892 was republished to peers [2018-07-03 11:18:17.284651]: Block F6BD43596EAD3AA60F4562B6794A6A2F5C96685B321619A415C80CC391B629FE was republished to peers [2018-07-03 11:18:18.009716]: frontier_req failed, reattempting [2018-07-03 11:18:18.119515]: frontier_req failed, reattempting [2018-07-03 11:18:18.390611]: frontier_req failed, reattempting [2018-07-03 11:18:18.572607]: frontier_req failed, reattempting [2018-07-03 11:18:18.698580]: frontier_req failed, reattempting [2018-07-03 11:18:18.833746]: frontier_req failed, reattempting [2018-07-03 11:18:18.921643]: frontier_req failed, reattempting [2018-07-03 11:18:19.166492]: frontier_req failed, reattempting [2018-07-03 11:18:19.282656]: frontier_req failed, reattempting

root@US1:~/RaiBlocks/log#

cryptocode commented 6 years ago

@huye0728 The Linux OOM Killer picked rai_node. Seems like you're running on a 1GB system, which is probably just too small.

huye0728 commented 6 years ago

yes, the direct cause is OOM, what i wonder is why the queqe has more than 2M blocks waiting process, i think may be there a deadlock or bootstrap receive too many blocks in short time.

rkeene commented 6 years ago

This should be resolved be on-going efforts to reduce memory utilization. I'm going to leave this ticket open for now until a specific change is planned that will significantly reduce memory utilization for nodes.

cryptocode commented 5 years ago

Closing this as we haven't had deadlock reports lately and https://github.com/nanocurrency/nano-node/issues/1794 more specifically targets memory issues.