search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.84k stars 901 forks source link

tried to upgrade to rc3 from rc2. Node crashed after "make install". Keep crashing after restart. Backup doesn't help. #5191

Closed heyambob closed 2 years ago

heyambob commented 2 years ago

Issue and Steps to Reproduce

I was on "v0.11.0rc2-2-g9a8bc77-modded", thought I should update myself to rc3. I didn't shutdown the node while I was compiling & make install, because that's what I used to do all the time. (restarting node after make install)

This is from the terminal:

2022-04-18T09:59:28.813Z INFO    03f2005837649dd49811bbb7e78993581da41c09d3bdcbc6933492276acccd8b6d-channeld-chan#22121: Peer connection lost
2022-04-18T09:59:48.482Z **BROKEN** 03f2005837649dd49811bbb7e78993581da41c09d3bdcbc6933492276acccd8b6d-channeld-chan#22121: version 'v0.11.0rc3-modded' not 
'v0.11.0rc2-2-g9a8bc77-modded': restarting

**** The above line happened right after I typed "make install" *****
**** Then I restarted it, and it's crashing like this *****

2022-04-18T10:00:16.771Z INFO    plugin-gracefulshutdown.py: RPC method 'shutdown' does not have a docstring.
2022-04-18T10:00:16.798Z INFO    plugin-inflighthtlcs.py: RPC method 'inflight_htlcs' does not have a docstring.
2022-04-18T10:00:16.834Z INFO    plugin-backup.py: root        : Comparing backup version 14524212 versus first write version 14524213
2022-04-18T10:00:16.834Z INFO    plugin-backup.py: Comparing backup version 14524212 versus first write version 14524213
2022-04-18T10:00:16.834Z INFO    plugin-backup.py: root        : Versions match up
2022-04-18T10:00:16.834Z INFO    plugin-backup.py: Versions match up
2022-04-18T10:00:20.651Z INFO    connectd: Static Tor service onion address: "buqjxr6ha4oob7knppq2ky4urrtohruowpdpldjdapgn2k5gt2nx74id.onion:9735,127.0.0.1:9735" bound from 
extern port 9735 
2022-04-18T10:00:20.693Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2022-04-18T10:00:27.214Z INFO    plugin-gracefulshutdown.py: Plugin gracefulshutdown initialized
2022-04-18T10:00:27.227Z INFO    plugin-inflighthtlcs.py: Plugin inflighthtlcs initialized
2022-04-18T10:00:27.227Z INFO    lightningd: Restarting onchaind for channel 7194
2022-04-18T10:00:27.236Z UNUSUAL 0211ebcabfd47de3de259d54c06fbcb7b4cbe58a95277c42f818752ab2e4925032-chan#7194: Peer permanent failure in ONCHAIN: Funding transaction spent
2022-04-18T10:00:27.236Z INFO    0211ebcabfd47de3de259d54c06fbcb7b4cbe58a95277c42f818752ab2e4925032-chan#7194: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2022-04-18T10:00:27.244Z INFO    lightningd: Restarting onchaind for channel 19201
2022-04-18T10:00:27.249Z UNUSUAL 038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5-chan#19201: Peer permanent failure in ONCHAIN: Funding transaction spent
2022-04-18T10:00:27.250Z INFO    038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5-chan#19201: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2022-04-18T10:00:27.257Z INFO    lightningd: Restarting onchaind for channel 22037
2022-04-18T10:00:27.266Z UNUSUAL 03ac61c971d146787a036f75e80a9fbede238a75d0c396f1fe996def00f0ac5dbe-chan#22037: Peer permanent failure in ONCHAIN: Funding transaction spent
2022-04-18T10:00:27.267Z INFO    03ac61c971d146787a036f75e80a9fbede238a75d0c396f1fe996def00f0ac5dbe-chan#22037: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2022-04-18T10:00:27.275Z INFO    lightningd: Restarting onchaind for channel 16927
2022-04-18T10:00:27.276Z UNUSUAL 0219a20e9afa8cbac8fd18fdbccb4a720ed313a6685f0a85ced255a3c250d48776-chan#16927: Peer permanent failure in ONCHAIN: Funding transaction spent
2022-04-18T10:00:27.277Z INFO    0219a20e9afa8cbac8fd18fdbccb4a720ed313a6685f0a85ced255a3c250d48776-chan#16927: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2022-04-18T10:00:27.311Z INFO    lightningd: --------------------------------------------------
2022-04-18T10:00:27.311Z INFO    lightningd: Server started with public key 020d1617e27ac022395352f2b3774969593d3d6ddff6fb117d820a9dda8da45217, alias needleInTheHay (color 
#d7cfad) and lightningd v0.11.0rc3-modded
2022-04-18T10:00:27.386Z INFO    plugin-rebalance.py: Plugin rebalance initialized with 0msat base / 3000 ppm fee  cltv_final:18  maxhops:5  msatfactor:4.0  erringnodes:5  
getroute:getroute_iterative  
2022-04-18T10:00:27.386Z INFO    0211ebcabfd47de3de259d54c06fbcb7b4cbe58a95277c42f818752ab2e4925032-chan#7194: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2022-04-18T10:00:27.396Z INFO    038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5-chan#19201: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2022-04-18T10:00:27.525Z INFO    03ac61c971d146787a036f75e80a9fbede238a75d0c396f1fe996def00f0ac5dbe-chan#22037: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2022-04-18T10:00:27.544Z INFO    0219a20e9afa8cbac8fd18fdbccb4a720ed313a6685f0a85ced255a3c250d48776-chan#16927: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2022-04-18T10:00:27.575Z UNUSUAL plugin-plugin.js: --- Starting the cl-rest server ---
2022-04-18T10:00:27.759Z UNUSUAL plugin-plugin.js: --- cl-rest api server is ready and listening on port: 3001 ---
2022-04-18T10:00:27.759Z UNUSUAL plugin-plugin.js: --- cl-rest doc server is ready and listening on port: 4001 ---
2022-04-18T10:00:28.211Z **BROKEN** lightningd: (null):?????U?
2022-04-18T10:00:28.236Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v0.11.0rc3-modded)
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x558c89425560
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (crashdump) 0x558c894255af
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fed9ae4955f
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fed9ae9634c
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fed9ae494b7
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fed9ae33533
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/log.c:821 (fatal_vfmt) 0x558c893c997f
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/log.c:829 (fatal) 0x558c893c9a30
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/htlc_end.c:87 (corrupt) 0x558c893b9b7d
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/htlc_end.c:205 (htlc_out_check) 0x558c893ba352
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1471 (check_already_failed) 0x558c893ea9c9
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1575 (onchain_failed_our_htlc) 0x558c893eb098
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:438 (handle_onchain_htlc_timeout) 0x558c893cd6ec
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:548 (onchain_msg) 0x558c893cdbdc
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/subd.c:556 (sd_msg_read) 0x558c893fa532
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x558c8948e3cd
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x558c8948ef9e
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x558c8948efdc
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x558c894912a8
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x558c893c0966
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1181 (main) 0x558c893c6bf9
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fed9ae3430f
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fed9ae343c0
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x558c8939bd14
2022-04-18T10:00:28.236Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff

This is from the crash log

+11.630127578 onchaind-chan#7194DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 16
+11.630147990 onchaind-chan#22037DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 2
+11.630196464 hsmdDEBUG: Got WIRE_HSMD_ECDH_REQ
+11.630207903 connectdDEBUG: Connect IN
+11.630325549 onchaind-chan#7194DEBUG: billboard: 3 outputs unresolved: in 224 blocks will spend DELAYED_OUTPUT_TO_US (04b9692023e78a3f660537d677a0b3d0620058b41a8df746d114cf193c263614:1) using OUR_DELAYED_RETURN_TO_WALLET
+11.630340083 onchaind-chan#22037DEBUG: billboard: 7 outputs unresolved: in 1199 blocks will spend DELAYED_OUTPUT_TO_US (d8cf90ec70c5d275e2f4ef5d8bf8149f5ab4f19d8e2026df90d318dd2fd67525:2) using OUR_DELAYED_RETURN_TO_WALLET
+11.630377957 hsmdDEBUG: Client: Received message 1 from client
+11.630387075 connectdDEBUG: peer_out WIRE_INIT
+11.630423098 onchaind-chan#19201DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.630565269 hsmdDEBUG: Got WIRE_HSMD_ECDH_REQ
+11.630574787 connectdDEBUG: peer_in WIRE_INIT
+11.630609495 onchaind-chan#19201DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 45
+11.630640643 onchaind-chan#7194DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.630683036 onchaind-chan#22037DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.630722522 hsmdDEBUG: Client: Received message 1 from client
+11.630785567 chan#110DEBUG: Peer has reconnected, state CHANNELD_NORMAL: telling connectd to make active
+11.630811981 onchaind-chan#19201DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 45
+11.630876649 onchaind-chan#7194DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 17
+11.630889940 onchaind-chan#22037DEBUG: OUR_UNILATERAL/OUR_HTLC->OUR_HTLC_TIMEOUT_TX depth 2
+11.630912306 connectdDEBUG: Handed peer, entering loop
+11.630933840 onchaind-chan#19201DEBUG: billboard: 7 outputs unresolved: in 6 blocks will spend THEIR_HTLC (1abaddce547787ae72bfe53f7f54f1ef3cb7e7993f0035906555e440eb351ffb:2) using THEIR_HTLC_TIMEOUT_TO_THEM
+11.630964224 onchaind-chan#7194DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 17
+11.630977208 onchaind-chan#22037DEBUG: billboard: 7 outputs unresolved: in 1199 blocks will spend DELAYED_OUTPUT_TO_US (d8cf90ec70c5d275e2f4ef5d8bf8149f5ab4f19d8e2026df90d318dd2fd67525:2) using OUR_DELAYED_RETURN_TO_WALLET
+11.631050771 onchaind-chan#7194DEBUG: billboard: 3 outputs unresolved: in 223 blocks will spend DELAYED_OUTPUT_TO_US (04b9692023e78a3f660537d677a0b3d0620058b41a8df746d114cf193c263614:1) using OUR_DELAYED_RETURN_TO_WALLET
+11.642537513 channeld-chan#110DEBUG: pid 2060929, msgfd 71
+11.642562167 chan#110DEBUG: Already have funding locked in (and ready to announce)
+11.642565535 chan#110DEBUG: Ignoring fee limits!
+11.643024548 lightningdDEBUG: update_feerates: feerate = 255, min=253, max=4294967295, penalty=253
+11.643036812 chan#110DEBUG: attempting update blockheight a0391a54718a59a9735c4a13a919c61038a5f8330d333d5a85f75e713acce002
+11.643106351 onchaind-chan#19201DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.643207154 onchaind-chan#22037DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.643237921 hsmdDEBUG: new_client: 110
+11.643257942 onchaind-chan#19201DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 46
+11.643288976 onchaind-chan#7194DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.643303280 onchaind-chan#22037DEBUG: OUR_UNILATERAL/OUR_HTLC->OUR_HTLC_TIMEOUT_TX depth 2
+11.643331327 onchaind-chan#19201DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 46
+11.643358423 onchaind-chan#7194DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 18
+11.643370159 onchaind-chan#22037DEBUG: billboard: 7 outputs unresolved: in 1199 blocks will spend DELAYED_OUTPUT_TO_US (d8cf90ec70c5d275e2f4ef5d8bf8149f5ab4f19d8e2026df90d318dd2fd67525:2) using OUR_DELAYED_RETURN_TO_WALLET
+11.643398786 onchaind-chan#19201DEBUG: billboard: 7 outputs unresolved: in 5 blocks will spend THEIR_HTLC (1abaddce547787ae72bfe53f7f54f1ef3cb7e7993f0035906555e440eb351ffb:2) using THEIR_HTLC_TIMEOUT_TO_THEM
+11.643427157 onchaind-chan#7194DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 18
+11.643529101 onchaind-chan#7194DEBUG: billboard: 3 outputs unresolved: in 222 blocks will spend DELAYED_OUTPUT_TO_US (04b9692023e78a3f660537d677a0b3d0620058b41a8df746d114cf193c263614:1) using OUR_DELAYED_RETURN_TO_WALLET
+11.643542636 onchaind-chan#22037DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.643571479 onchaind-chan#19201DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.643617491 onchaind-chan#22037DEBUG: Sending 0 missing htlc messages
+11.643644827 onchaind-chan#19201DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 47
+11.643669930 onchaind-chan#7194DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.643681156 onchaind-chan#22037DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 3
+11.643714218 onchaind-chan#19201DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 47
+11.643739818 onchaind-chan#7194DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 19
+11.643752360 onchaind-chan#22037DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 3
+11.643780829 onchaind-chan#19201DEBUG: billboard: 7 outputs unresolved: in 4 blocks will spend THEIR_HTLC (1abaddce547787ae72bfe53f7f54f1ef3cb7e7993f0035906555e440eb351ffb:2) using THEIR_HTLC_TIMEOUT_TO_THEM
+11.643807344 onchaind-chan#7194DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 19
+11.643818444 onchaind-chan#22037DEBUG: billboard: 7 outputs unresolved: in 1198 blocks will spend DELAYED_OUTPUT_TO_US (d8cf90ec70c5d275e2f4ef5d8bf8149f5ab4f19d8e2026df90d318dd2fd67525:2) using OUR_DELAYED_RETURN_TO_WALLET
+11.643882332 onchaind-chan#7194DEBUG: billboard: 3 outputs unresolved: in 221 blocks will spend DELAYED_OUTPUT_TO_US (04b9692023e78a3f660537d677a0b3d0620058b41a8df746d114cf193c263614:1) using OUR_DELAYED_RETURN_TO_WALLET
+11.643934937 onchaind-chan#19201DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.643980954 onchaind-chan#22037DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.644008027 onchaind-chan#19201DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 48
+11.644033569 onchaind-chan#7194DEBUG: Got new message WIRE_ONCHAIND_DEPTH
+11.644044838 onchaind-chan#22037DEBUG: OUR_UNILATERAL/OUR_HTLC->OUR_HTLC_TIMEOUT_TX depth 3
+11.644071831 onchaind-chan#19201DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 48
+11.644097337 onchaind-chan#7194DEBUG: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 20
+11.644109771 onchaind-chan#22037DEBUG: OUR_UNILATERAL/OUR_HTLC reached reasonable depth 3
+11.644139011 onchaind-chan#19201DEBUG: billboard: 7 outputs unresolved: in 3 blocks will spend THEIR_HTLC (1abaddce547787ae72bfe53f7f54f1ef3cb7e7993f0035906555e440eb351ffb:2) using THEIR_HTLC_TIMEOUT_TO_THEM
+11.644166762 onchaind-chan#7194DEBUG: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 20
+11.644180718 chan#22037DEBUG: onchain_failed_our_htlc
+11.644183509 chan#22037DEBUG: HTLC id 619 failonion = (nil), failmsg = 0x558c963858c8, preimage = (nil)
+11.644212023 lightningdBROKEN: (null):?????U?
+11.668847704 lightningdBROKEN: FATAL SIGNAL 6 (version v0.11.0rc3-modded)
+11.668892811 lightningdBROKEN: backtrace: common/daemon.c:38 (send_backtrace) 0x558c89425560
+11.668901219 lightningdBROKEN: backtrace: common/daemon.c:46 (crashdump) 0x558c894255af
+11.668908357 lightningdBROKEN: backtrace: (null):0 ((null)) 0x7fed9ae4955f
+11.668918008 lightningdBROKEN: backtrace: (null):0 ((null)) 0x7fed9ae9634c
+11.668943039 lightningdBROKEN: backtrace: (null):0 ((null)) 0x7fed9ae494b7
+11.668949734 lightningdBROKEN: backtrace: (null):0 ((null)) 0x7fed9ae33533
+11.668971802 lightningdBROKEN: backtrace: lightningd/log.c:821 (fatal_vfmt) 0x558c893c997f
+11.668978165 lightningdBROKEN: backtrace: lightningd/log.c:829 (fatal) 0x558c893c9a30
+11.668984935 lightningdBROKEN: backtrace: lightningd/htlc_end.c:87 (corrupt) 0x558c893b9b7d
+11.668991262 lightningdBROKEN: backtrace: lightningd/htlc_end.c:205 (htlc_out_check) 0x558c893ba352
+11.669016705 lightningdBROKEN: backtrace: lightningd/peer_htlcs.c:1471 (check_already_failed) 0x558c893ea9c9
+11.669023345 lightningdBROKEN: backtrace: lightningd/peer_htlcs.c:1575 (onchain_failed_our_htlc) 0x558c893eb098
+11.669043122 lightningdBROKEN: backtrace: lightningd/onchain_control.c:438 (handle_onchain_htlc_timeout) 0x558c893cd6ec
+11.669049818 lightningdBROKEN: backtrace: lightningd/onchain_control.c:548 (onchain_msg) 0x558c893cdbdc
+11.669056372 lightningdBROKEN: backtrace: lightningd/subd.c:556 (sd_msg_read) 0x558c893fa532
+11.669063030 lightningdBROKEN: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x558c8948e3cd
+11.669069093 lightningdBROKEN: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x558c8948ef9e
+11.669075470 lightningdBROKEN: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x558c8948efdc
+11.669081900 lightningdBROKEN: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x558c894912a8
+11.669087916 lightningdBROKEN: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x558c893c0966
+11.669094531 lightningdBROKEN: backtrace: lightningd/lightningd.c:1181 (main) 0x558c893c6bf9
+11.669101015 lightningdBROKEN: backtrace: (null):0 ((null)) 0x7fed9ae3430f
+11.669125723 lightningdBROKEN: backtrace: (null):0 ((null)) 0x7fed9ae343c0
+11.669161233 lightningdBROKEN: backtrace: (null):0 ((null)) 0x558c8939bd14
+11.669168698 lightningdBROKEN: backtrace: (null):0 ((null)) 0xffffffffffffffff

After that I went back to the old rc2, didn't work showing similar errors, then I restored from the backup and still no luck.

So my guess is it's not about rc2/3, it's just some funky data was written on the db when it crashed, and it got replicated to the backup.

All is lost ? What is my next step ?

getinfo output

btweenthebars commented 2 years ago

I'm looking at this line "lightningd/htlc_end.c:205 (htlc_out_check) 0x558c893ba352"

                       if (hout->in->failonion)
                                return corrupt(abortstr,
                                               "Output failmsg, input failonion");

Not sure why "hout->in->failonion" is so dangerous that we have to stop the daemon. Is possible that there's some missing record/field in my db causing "failonion" ? If so, is there a way to do sanity check and find out which records in which tables are causing trouble ?

Edit - Ha . that's me using different laptop

rustyrussell commented 2 years ago

This check does seem overzealous. It was added just before rc1. I will loosen it once I get to my laptop...

rustyrussell commented 2 years ago

Please check https://github.com/ElementsProject/lightning/pull/5193, or just do the same thing locally remove lines 206-209 of lightningd/htlc_end.c:

diff --git a/lightningd/htlc_end.c b/lightningd/htlc_end.c
index bd80eecde..d9a80ac1b 100644
--- a/lightningd/htlc_end.c
+++ b/lightningd/htlc_end.c
@@ -201,9 +201,6 @@ struct htlc_out *htlc_out_check(const struct htlc_out *hout,
                return corrupt(abortstr,
                           "Output failmsg, input preimage");
        } else if (hout->failmsg) {
-           if (hout->in->failonion)
-               return corrupt(abortstr,
-                          "Output failmsg, input failonion");
            if (hout->in->preimage)
                return corrupt(abortstr,
                           "Output failmsg, input preimage");
btweenthebars commented 2 years ago

Thank you. That works. What does the failonion failonion do ? Should we write the log if it's useful somehow ?

Also, on the check_already_failed

We've had a bug report about not failing incoming HTLCs. This does a sanity check, if we think we've already failed this HTLC

Where does this refer to ?

rustyrussell commented 2 years ago

We actually found the real bug, in this case. But I put a whole pile of new checks in the code at various points around where the failure was; this one was too strict.

In detail: onchaind can tell us "this HTLC can be failed now" but if we're restarting, that might be old news. We now do extra checks to make sure it failed the same way, but this isn't actually guaranteed, nor a problem. Hence, we tripped up on that check.