search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.87k stars 906 forks source link

FATAL SIGNAL 6 after getblockhash #2042

Open HugoDoyon opened 6 years ago

HugoDoyon commented 6 years ago

Hello,

Lightning Node stop responding after getblockhash with the following error :

2018-10-21T10:51:05.178Z lightningd(954): bitcoin-cli: finished bitcoin-cli -rpcconnect=localhost -rpcport=8332 -rpcuser=bitcoin -rpcpassword=ZZZZZZ getblockhash 546689 (5 ms)
2018-10-21T10:51:05.178Z lightningd(954): bitcoin-cli: starting bitcoin-cli -rpcconnect=localhost -rpcport=8332 -rpcuser=bitcoin -rpcpassword=ZZZZZZ getblock 000000000000000000117f6c58e7b5f9d6980e2270feb4767c055c6e3d0724b9 false
2018-10-21T10:51:05.416Z lightningd(954): bitcoin-cli: finished bitcoin-cli -rpcconnect=localhost -rpcport=8332 -rpcuser=bitcoin -rpcpassword=ZZZZZZ getblock 000000000000000000117f6c58e7b5f9d6980e2270feb4767c055c6e3d0724b9 false (234
 ms)
2018-10-21T10:51:05.492Z lightningd(954): Adding block 546689: 000000000000000000117f6c58e7b5f9d6980e2270feb4767c055c6e3d0724b9
2018-10-21T10:51:05.917Z lightningd(954): bitcoin-cli: starting bitcoin-cli -rpcconnect=localhost -rpcport=8332 -rpcuser=bitcoin -rpcpassword=ZZZZZZ getblockhash 546690
2018-10-21T10:51:06.096Z lightningd(954): bitcoin-cli: finished bitcoin-cli -rpcconnect=localhost -rpcport=8332 -rpcuser=bitcoin -rpcpassword=ZZZZZZ getblockhash 546690 (119 ms)
2018-10-21T10:51:06.097Z lightningd(954): bitcoin-cli -rpcconnect=localhost -rpcport=8332 -rpcuser=bitcoin -rpcpassword=ZZZZZZ getblockhash 546690 died with signal 11
2018-10-21T10:51:06.097Z lightningd(954): FATAL SIGNAL 6 (version v0.6.1-125-gd6fcfe0)

Current setup : ubuntu 16, Thundroid, c-lightning (0.6.1), bitcoin core node 0.16.3.

When the LN node drop and stop working bitcoin core was still up and it was possible to ping the host as well. However, SSH connection fail and I had to reboot in order to bring everything back up.

What happened to this node? Thank you!

rustyrussell commented 6 years ago

bitcoin-cli crashed, and c-lightning stopped because of it. Did you run out of memory? Or is your machine generally unreliable? It seems weird...

HugoDoyon commented 6 years ago

Hi Rusty,

As per my bitcoin debug log it appear that the bitcoin node was still up and running after c-lightning crash :

2018-10-21 10:25:17 Pre-allocating up to position 0x4000000 in blk01406.dat
2018-10-21 10:25:17 Pre-allocating up to position 0x700000 in rev01406.dat
2018-10-21 10:25:17 UpdateTip: new best=000000000000000000239297505b09b0da5dc7a0eaf76ea2269089cfe3118412 height=546688 version=0x3fffe000 log2_work=89.887954 tx=350290410 date='2018-10-21 10:25:20' progress=1.000000 cache=31.3MiB(274324t
xo) warning='8 of last 100 blocks have unexpected version'
2018-10-21 10:34:29 connect() to [2001:0:9d38:953c:10e9:ab7b:8eb1:9032]:8333 failed after select(): No route to host (113)
2018-10-21 10:45:48 connect() to [2001:0:9d38:6abd:202f:fe2:a1d2:c739]:8333 failed after select(): No route to host (113)
2018-10-21 10:50:41 UpdateTip: new best=000000000000000000117f6c58e7b5f9d6980e2270feb4767c055c6e3d0724b9 height=546689 version=0x20000000 log2_work=89.887993 tx=350293226 date='2018-10-21 10:50:14' progress=1.000000 cache=32.1MiB(281408t
xo) warning='8 of last 100 blocks have unexpected version'
2018-10-21 10:53:37 UpdateTip: new best=0000000000000000002534812cceb5f7841cb6d0374428e0daf913c1daeefe71 height=546690 version=0x20000000 log2_work=89.888032 tx=350294627 date='2018-10-21 10:53:25' progress=1.000000 cache=32.2MiB(282839t
xo) warning='7 of last 100 blocks have unexpected version'
2018-10-21 10:59:09 UpdateTip: new best=00000000000000000001442f4b25233c4a667c5c0a2d8a3874449a1bed802f32 height=546691 version=0x20000000 log2_work=89.888071 tx=350295514 date='2018-10-21 10:58:31' progress=1.000000 cache=32.3MiB(283279t
xo) warning='6 of last 100 blocks have unexpected version'
2018-10-21 10:59:45 connect() to [240e:ff:b310:9f0d:c13:19df:f966:918f]:8333 failed after select(): No route to host (113)
2018-10-21 11:05:36 UpdateTip: new best=00000000000000000011159a335b7e8573f47fab85d5dafc3f459625e20809b0 height=546692 version=0x20000000 log2_work=89.88811 tx=350296610 date='2018-10-21 11:05:11' progress=1.000000 cache=32.4MiB(284068tx
o) warning='6 of last 100 blocks have unexpected version'
2018-10-21 11:10:12 connect() to [2001:0:3d83:d0d2:3420:1621:3f57:d420]:8333 failed after select(): No route to host (113)
2018-10-21 11:10:43 UpdateTip: new best=000000000000000000022ddd01667f47e5119dec217d81dd707c6d85582a6696 height=546693 version=0x3fffe000 log2_work=89.888149 tx=350297425 date='2018-10-21 11:10:20' progress=1.000000 cache=32.5MiB(285262t
xo) warning='7 of last 100 blocks have unexpected version'
2018-10-21 11:13:14 UpdateTip: new best=0000000000000000001a0903b0b73d554f2110349dfc75fcce2f375dfd51c24c height=546694 version=0x20000000 log2_work=89.888188 tx=350297817 date='2018-10-21 11:13:01' progress=1.000000 cache=32.6MiB(285927t
xo) warning='6 of last 100 blocks have unexpected version'
2018-10-21 11:14:04 UpdateTip: new best=00000000000000000004f5ad43e2f4ca56aa3587b5d497bcf97738436d4d8dc2 height=546695 version=0x20000000 log2_work=89.888226 tx=350297882 date='2018-10-21 11:13:20' progress=1.000000 cache=32.6MiB(286146t
xo) warning='6 of last 100 blocks have unexpected version'
2018-10-21 11:15:31 connect() to [2001:0:9d38:6ab8:3caf:d79:a307:4359]:8333 failed after select(): No route to host (113)
2018-10-21 11:15:58 connect() to 85.220.165.232:8333 failed after select(): No route to host (113)
2018-10-21 11:23:07 Misbehaving: 164.132.12.120:51472 peer=390 (0 -> 1)
2018-10-21 11:25:09 connect() to [2001:16b8:170f:ed00:76da:38ff:fe3e:d415]:8333 failed after select(): No route to host (113)
2018-10-21 11:26:39 connect() to [2001:0:5cf2:8c44:3892:18bc:3f57:ff9b]:8333 failed after select(): No route to host (113)
2018-10-21 11:28:05 UpdateTip: new best=0000000000000000000d09ff769a791d62a5ca7de237f8a9d73dfcd695bf1b55 height=546696 version=0x20000000 log2_work=89.888265 tx=350299890 date='2018-10-21 11:27:16' progress=1.000000 cache=33.0MiB(289689t
xo) warning='6 of last 100 blocks have unexpected version'
2018-10-21 11:36:48 UpdateTip: new best=00000000000000000018f21499c23fd076b46e7c3546e275c2ae292332781b4a height=546697 version=0x20000000 log2_work=89.888304 tx=350300897 date='2018-10-21 11:36:45' progress=1.000000 cache=33.2MiB(291814t
xo) warning='6 of last 100 blocks have unexpected version'
2018-10-21 11:43:56 UpdateTip: new best=0000000000000000000ae9f4707505a0cad35bcf12dcd536f2e319ee817d0ad2 height=546698 version=0x20000000 log2_work=89.888343 tx=350302281 date='2018-10-21 11:43:27' progress=1.000000 cache=33.4MiB(293905t
xo) warning='6 of last 100 blocks have unexpected version'

As I said before, I was able to ping the node, the port 8333 was still accessible, but the SSH and LN crash.

I have search system log, fail2ban, UFW log and nothing seem suspect to me. I have review as well atop perf history for memory:

image

Any idea what could have cause this issues? Thank you!

darosior commented 5 years ago

Hello @HugoDoyon, are you still experiencing this ? Otherwise this can be closed..