search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.84k stars 901 forks source link

Stuck crashing on boot after `sendonion`: "lightningd/pay.c:549: payment_failed: Assertion `!hout->failcode' failed." #3370

Closed jarret closed 4 years ago

jarret commented 4 years ago

Initial crash was with v0.8.0rc2 with a small patch that adds an extra field to a plugin notification. (I don't beleive this patch is related to the issue, unless something really gnarly is going on).

Reverting to unpatched v0.8.0, the same crash happens on boot.

I am writing scripts to use createonion and sendonion. Upon sending an onion, it crashed like this:

2019-12-23T20:17:48.656Z DEBUG 02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7-channeld-chan#3: Sending master 1022
2019-12-23T20:17:48.656Z DEBUG 02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7-chan#3: got revoke 8617: 1 changed
2019-12-23T20:17:48.656Z DEBUG 02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7-chan#3: HTLC out 3849 SENT_REMOVE_ACK_COMMIT->RCVD_REMOVE_ACK_REVOCATION
2019-12-23T20:17:48.657Z DEBUG 02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7-chan#3: Removing out HTLC 3849 state RCVD_REMOVE_ACK_REVOCATION WIRE_INVALID_ONION_HMAC
lightningd: lightningd/pay.c:549: payment_failed: Assertion `!hout->failcode' failed.
lightningd: FATAL SIGNAL 6 (version v0.8.0rc2-1-gef08efd)
0x562394c762ff send_backtrace
        common/daemon.c:41
0x562394c76389 crashdump
        common/daemon.c:54
0x7fb0a6c5083f ???
        ???:0
0x7fb0a6c507bb ???
        ???:0
0x7fb0a6c3b534 ???
        ???:0
0x7fb0a6c3b40e ???
        ???:0
0x7fb0a6c49101 ???
        ???:0
0x562394c63829 payment_failed
        lightningd/pay.c:549
0x562394c6788e fail_out_htlc
        lightningd/peer_htlcs.c:163
0x562394c682d6 remove_htlc_out
        lightningd/peer_htlcs.c:1190
0x562394c68453 update_out_htlc
        lightningd/peer_htlcs.c:1278
0x562394c68724 changed_htlc
        lightningd/peer_htlcs.c:1287
0x562394c6a8e2 peer_got_revoke
        lightningd/peer_htlcs.c:1687
0x562394c4c279 channel_msg
        lightningd/channel_control.c:276
0x562394c6ff14 sd_msg_read
        lightningd/subd.c:479
0x562394cb125d next_plan
        ccan/ccan/io/io.c:59
0x562394cb1730 do_plan
        ccan/ccan/io/io.c:407
0x562394cb1762 io_ready
        ccan/ccan/io/io.c:417
0x562394cb301a io_loop
        ccan/ccan/io/poll.c:445
0x562394c5386c io_loop_with_timers
        lightningd/io_loop_with_timers.c:24
0x562394c57da2 main
        lightningd/lightningd.c:855
0x7fb0a6c3d09a ???
        ???:0
0x562394c46139 ???
        ???:0
0xffffffffffffffff ???
        ???:0
2019-12-23T20:17:48.672Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v0.8.0rc2-1-gef08efd)
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (send_backtrace) 0x562394c76347
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: common/daemon.c:54 (crashdump) 0x562394c76389
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fb0a6c5083f
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fb0a6c507bb
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fb0a6c3b534
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fb0a6c3b40e
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fb0a6c49101
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/pay.c:549 (payment_failed) 0x562394c63829
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:163 (fail_out_htlc) 0x562394c6788e
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1190 (remove_htlc_out) 0x562394c682d6
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1278 (update_out_htlc) 0x562394c68453
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1287 (changed_htlc) 0x562394c68724
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1687 (peer_got_revoke) 0x562394c6a8e2
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/channel_control.c:276 (channel_msg) 0x562394c4c279
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/subd.c:479 (sd_msg_read) 0x562394c6ff14
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x562394cb125d
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x562394cb1730
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x562394cb1762
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x562394cb301a
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:24 (io_loop_with_timers) 0x562394c5386c
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:855 (main) 0x562394c57da2
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fb0a6c3d09a
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x562394c46139
2019-12-23T20:17:48.673Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
Log dumped in crash.log.20191223201748
lightning_channeld: Could not set sync read from master: Connection reset by peer (version v0.8.0rc2-1-gef08efd)
./runld.sh: line 4:  7233 Aborted                 /home/jarret/lightningd-run/lightning/lightningd/lightningd --lightning-dir=/home/jarret/lightningd-run/lightning-dir --pid-file=/home/jarret/lightningd-run/lrun.pid --bitcoin-datadir=/home/jarret/.bitcoin --bitcoin-cli=/home/jarret/.bitcoin/bitcoin-0.18.1/bin/bitcoin-cli --bitcoin-rpcuser=pepe --bitcoin-rpcpassword=thefrog --fee-per-satoshi 1 --log-level=debug --plugin-dir=/home/jarret/lightningd-run/plugins --autocleaninvoice-cycle 3600 --alias "Banana Phone!" --rgb "C4FF5A" --network=bitcoin --zmq-pub-htlc-accepted=tcp://127.0.0.1:6666 --zmq-pub-invoice-payment=tcp://127.0.0.1:5555 --zmq-pub-forward-event=tcp://127.0.0.1:6666 --zmq-pub-connect=tcp://127.0.0.1:5556 --zmq-pub-disconnect=tcp://127.0.0.1:5556 $@
(ins)jarret@renn:~/lightningd-run$ 0x5557af425857 send_backtrace
        common/daemon.c:41
0x5557af42d689 status_failed
        common/status.c:206
0x5557af416547 master_wait_sync_reply
        channeld/channeld.c:800
0x5557af418e0b handle_peer_revoke_and_ack
        channeld/channeld.c:1599
0x5557af41975a peer_in
        channeld/channeld.c:1864
0x5557af41adb7 main
        channeld/channeld.c:3242
0x7ff3357f109a ???
        ???:0
0x5557af414d49 ???
        ???:0
0xffffffffffffffff ???
        ???:0

The crash log of the first crash crash.log.20191223201609.gz

The crash log from rebooting: crash.log.20191223201748.gz

My script was creating an onion to attempt to send along a circular route:

1) I created an invoice:

bolt11: lnbc10n1p0qzgygpp5snrz96w9ns7fxk3dl2y765fzz0jcvxsryl90n4q9jx93ez85lcaqdpjvd5hycm4d3shygrsv9uk6etwwssxvmmjyp3xzmnwv4e8qatwdvxqyjw5qcqp2sp59uwwexrgrajlf0lluhqmqajrmyyca7kg5r75de8ymgay3339uv9srzjqt3cnkrp4nvadatspjvudsea63rq6mc79a4638glfumtap0uvrudwz2xlgqqvpgqqqqqqqykqqqqqqgqpc9qy9qsqhw6ydx2fr0urfk8ppacskn58z5mg9n6y8c9y6qadx66vzdr4zfqnzp2wxde3kyyqkxjhsyllx35w24dq2w2p8a32mwpw3xw8dsyqeygqe60rku
payment_secret: 2f1cec98681f65f4bfffe5c1b07643d9098efac8a0fd46e4e4da3a48c625e30b

2) I queried an outgoing route:

outgoing:
{'route': [{'amount_msat': 2005msat,
            'channel': '607994x1541x0',
            'delay': 23,
            'direction': 1,
            'id': '02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7',
            'msatoshi': 2005,
            'style': 'tlv'},
           {'amount_msat': 2000msat,
            'channel': '584191x607x0',
            'delay': 9,
            'direction': 0,
            'id': '0379c41f28a38c49998fec42437db78a17af508fb19338e7360d7ffee2607ea036',
            'msatoshi': 2000,
            'style': 'legacy'}]}

3) I queried a returning route:

{'route': [{'amount_msat': 1150msat,
            'channel': '584191x607x0',
            'delay': 23,
            'direction': 1,
            'id': '02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7',
            'msatoshi': 1150,
            'style': 'tlv'},
           {'amount_msat': 1000msat,
            'channel': '607994x1541x0',
            'delay': 9,
            'direction': 0,
            'id': '035c77dc0a10fe60e1304ae5b57d8fef87751add5d016b896d854fb706be6fc96c',
            'msatoshi': 1000,
            'style': 'tlv'}]}

4) I assembled the two routes into a circular route (using logic from the sendinvoiceless plugin):

[{'amount_msat': 3158msat,
  'channel': '607994x1541x0',
  'delay': 52,
  'direction': 1,
  'id': '02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7',
  'msatoshi': 3158,
  'style': 'tlv'},
 {'amount_msat': 3152msat,
  'channel': '584191x607x0',
  'delay': 38,
  'direction': 0,
  'id': '0379c41f28a38c49998fec42437db78a17af508fb19338e7360d7ffee2607ea036',
  'msatoshi': 3152,
  'style': 'legacy'},
 {'amount_msat': 1151msat,
  'channel': '584191x607x0',
  'delay': 24,
  'direction': 1,
  'id': '02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7',
  'msatoshi': 1151,
  'style': 'tlv'},
 {'amount_msat': 1000msat,
  'channel': '607994x1541x0',
  'delay': 10,
  'direction': 0,
  'id': '035c77dc0a10fe60e1304ae5b57d8fef87751add5d016b896d854fb706be6fc96c',
  'msatoshi': 1000,
  'style': 'tlv'}]

5) I constructed the hops parameter, encoding the legacy payloads myself:

NOTE - I am not sure I did this entirely correctly, figuring that out is the purpose of this experiment. Particlulary, I am not entirely certain how to set the outgoing_cltv_value in the legacy payload from the above route using the delay value. The documentation and example for createonion coule possible be clearer.

hops:
[{'payload': '0008e9ff00025f00000000000000000c5000000026000000000000000000000000',
  'pubkey': '0379c41f28a38c49998fec42437db78a17af508fb19338e7360d7ffee2607ea036',
  'style': 'legacy'},
 {'payload': '0008e9ff00025f0000000000000000047f00000018000000000000000000000000',
  'pubkey': '02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7',
  'style': 'legacy'},
 {'payload': '000946fa000605000000000000000003e80000000a000000000000000000000000',
  'pubkey': '035c77dc0a10fe60e1304ae5b57d8fef87751add5d016b896d854fb706be6fc96c',
  'style': 'legacy'},
 {'payload': '000946fa000605000000000000000003e80000000a000000000000000000000000',
  'pubkey': '035c77dc0a10fe60e1304ae5b57d8fef87751add5d016b896d854fb706be6fc96c',
  'style': 'legacy'}]

6) I passed the hops to createonion using the payment_hash dug out of the BOLT11 (via a decodepay call) as the assocdata parameter to get the onion and list of shared secrets:

onion: 00030385d9de0d0c6ac609758cd5f6ad9167cc70279c026801ddeed01588f64aff6009a8fac98216974bf64c50cf6486308e643a9934aa5a9b285cbd5f82af318a690733b5fc88f1b7759fc39728ea72822b6b118b22fd71108c24fcdb6a2f6bf9f6ff3fb6d0cca6b728aab9cbb732f096c65af6045d2f259e7a35650012fcf95e1a130703fa948cdd98f9227bd8a77b355eae59c30826fb81bf0e6a80666f4b348c3ad74ca2669eb7f31f6edbaf32feb929e13295fc27c1b7db150442805bfb800e23a4ec2e8707a36e0044d369b0c9959cb33c9c582db53cfb3a347fea99df1d96f4f01cdd9680a9f51adc8b935583ee8a8aecf078d6b54ab43e93ba4e79fb73b94984aa4343b4ffca2a1c66dcf45d2f122a271777fbaaadb57127cd623341dec36edd8b7ef5a26c81bd3c19315c118e8b515c7a2d78f47514a864e26cd3223f5c210d56f95a8079d26a80b2c4eaa821bcceb5506d526c1a008aaacc1121bafcb43e8169d1861bc9206470792bc1e0f8e570cdedaef7ad222befc1c59fe5fa51ff88547810fc2bc6b54a822d8ee3838191e585a14a4b02b75891603353ab548034aecf703c1619a1b5b024ba51dec43828038895ada82344b65c60c4251dfd39ebfe7951b775c8cf712cacde06b369e952de2eef92e18a2e8c2900abfeec32c70a907942152354bcef3ba49c98e4253f2a4ed2a446e942035f238194b86175ef7b6e231a60d5c0fd53a11c37cfb0b7d0947c46e25ffe83b67a16f1b08f64ebe3de1de86440aa3e7dfea43dea023cf538f1bbc32c0fb3d8a764cea30ec98b8b67e9e4106529511f9e6e70a3fcf5c84b6ebdfd50075a449f4205def633d3e573dcb31ee4da38e4d8f3a09c2f329b7ef621b800afa561b47c82f51feb6ae3ac5d2007ec109a5caa06dddcdc8e99e1c8307ca127015ccfb02048221459bfbd14f8780f9d6b618bbd0ea4dae82e793dbdf4f1b17e3a5e198d4dfc78f900beb5679d0b849e87da86b69a62fb9afd7b6b109d6f24e1c2784af6562ec3a9c9dce6664e9310c66413d8ce0e81c00cf41e60d35308eff5e97bd21afa7069a69c6a9bc989cc8a488513f662982d337d3e4233b2f001530a4025b98d4504b015fdd907d8f84e745c61a17096a253fc41f623a180e8917dffc072fb25fb27af3bce1fabbfcb3dc23aae5bd64bf746a12e8aa2772f37d56394ef04ec4649f3a50f78d42531df037ca5565b95b1a251096edac31411ff6f88e052682908b55419685bb63cd5e1b54261da7a4850e72f734ee6a99147d6c31fefcbf4ed0e98340008bb137c4749d2d01f979017a4806609d9de32572f4b375d4ec320da554c7c5859877e11cc343f33be9efe36dd7b5eacb499bb57ced276bc8af2eef624aa208b864ce5310dcad7279b3d8451d259a5385eb568c85bff1ee0ace7f6c2d4fd6039465d1a9fd37aea6644d13af0af46b5e39f78c7248ccdd5124e75451bf07bb6631b15383d71c370244a5818f0554867d03b6e0fbae20f9c765c771987fa7f157483f4abb5161a2c00e8d7508d9c38ebc039a48d39557cd11f771c6f90ab8b9a978a0e50b06419dd8feff0a3ebd626808165e6d2643aa034b5200b6c5376ff44c95c9ee59b03c1e36c2e543ed1521cd94dcb6547045df564f5057fc534f402b841fdda49d3ab25e19854644454c4acd94ae19e3fabff185e113931e0523e99b7d97bec08c07aae27a154c9972d52f4daad110cbbf47531b026b04586846ad6680b2ab882424c4345254e7af63cbe238c1304802173cd424cad766511933f8c87aafe676fd304ecbf3f3d366ecf1ef110d87b84525d90e8513b8e3037bbe003cd4c9408379e4b17b707273b07fcc0d6ed752a7d7670b9efc81aaaf36c5fc6569a13ba21cc1c01abbf095d2942721478bab5f8587379d3fb6a1799e579f7
shared_secrets: ['99e7337b13a5d5382b5a04a4ee1cea5032409eb2554ced80ad1af18b5afa2430', '7e86743c7074c5cfe68500b1bf00b34b847d61f66f3fbfa24def7068f18f4692', '4005d26c91b253f8ee81daf64b8fb1fb540e5b42cb487f6d86d91359773bf2f1', 'cbe34aa37dd6e469e64d57cc743170d65e56c09d4709e55921ba9f8f9e33043d']

7) I called sendonion with the onion, the first hop from the circular route, the payment_hash value, a randomly generated uuid4 label, and the shared_secrets array. It returned to my script:

{'message': 'Monitor status with listpayments or waitsendpay', 'id': 6486, 'payment_hash': '84c622e9c59c3c935a2dfa89ed512213e5861a0327caf9d405918b1c88f4fe3a', 'amount_msat': None, 'msatoshi_sent': 3158, 'amount_sent_msat': 3158msat, 'created_at': 1577132168, 'status': 'pending', 'label': '73c40079-260f-40c1-a125-a52e4451e036'}

However, at that point the node crashed.

The line prior to the failed assertion makes me believe there is something wrong with the HMAC on the onion:

2019-12-23T20:17:48.657Z DEBUG 02e389d861acd9d6f5700c99c6c33dd4460d6f1e2f6ba89d1f4f36be85fc60f8d7-chan#3: Removing out HTLC 3849 state RCVD_REMOVE_ACK_REVOCATION WIRE_INVALID_ONION_HMAC

At that point, I tried restarting and was met with the same crash

At that point, I reverted to an unmodified v0.8.0 branch and was met with the same crash.

(aside: Is there a clean way to get rid of this bad HTLC so my node boots?)

It's entirely possible I am misunderstand/misusing something in this procedure, this is literally the first time I ever called sendonion, but I guess it shouldn't ever crash like this.

jarret commented 4 years ago

looking over my script: the last two hops have the same pubkey along with the same legacy payload, which is wrong due to a bug in my script when creating the final hop. This almost certainly makes the onion invalid in that specific way, which could be a clue to the crash.

{'payload': '000946fa000605000000000000000003e80000000a000000000000000000000000',
  'pubkey': '035c77dc0a10fe60e1304ae5b57d8fef87751add5d016b896d854fb706be6fc96c',
  'style': 'legacy'},
 {'payload': '000946fa000605000000000000000003e80000000a000000000000000000000000',
  'pubkey': '035c77dc0a10fe60e1304ae5b57d8fef87751add5d016b896d854fb706be6fc96c',
  'style': 'legacy'}]

EDIT: the pubkey is in fact wrong on all hops for the onion due to the same bug. :S

jarret commented 4 years ago

I was able to get the node booted again by unilaterally closing the channel of the first hop from the other side - a node which I control.

I did a sqlite3 dump and found the onion as the last entry in the channel_htlcs table. I am not confident whether it would be safe to just drop that entry to recover that way, so just closing the channel seemed safer.


INSERT INTO channel_htlcs VALUES(12212,3,3849,1,NULL,3158,609547,X'84c622e9c59c3c935a2dfa89ed512213e5861a0327caf9d405918b1c88f4fe3a',NULL,X'00030385d9de0d0c6ac609758cd5f6ad9167cc70279c026801ddeed01588f64aff6009a8fac98216974bf64c50cf6486308e643a9934aa5a9b285cbd5f82af318a690733b5fc88f1b7759fc39728ea72822b6b118b22fd71108c24fcdb6a2f6bf9f6ff3fb6d0cca6b728aab9cbb732f096c65af6045d2f259e7a35650012fcf95e1a130703fa948cdd98f9227bd8a77b355eae59c30826fb81bf0e6a80666f4b348c3ad74ca2669eb7f31f6edbaf32feb929e13295fc27c1b7db150442805bfb800e23a4ec2e8707a36e0044d369b0c9959cb33c9c582db53cfb3a347fea99df1d96f4f01cdd9680a9f51adc8b935583ee8a8aecf078d6b54ab43e93ba4e79fb73b94984aa4343b4ffca2a1c66dcf45d2f122a271777fbaaadb57127cd623341dec36edd8b7ef5a26c81bd3c19315c118e8b515c7a2d78f47514a864e26cd3223f5c210d56f95a8079d26a80b2c4eaa821bcceb5506d526c1a008aaacc1121bafcb43e8169d1861bc9206470792bc1e0f8e570cdedaef7ad222befc1c59fe5fa51ff88547810fc2bc6b54a822d8ee3838191e585a14a4b02b75891603353ab548034aecf703c1619a1b5b024ba51dec43828038895ada82344b65c60c4251dfd39ebfe7951b775c8cf712cacde06b369e952de2eef92e18a2e8c2900abfeec32c70a907942152354bcef3ba49c98e4253f2a4ed2a446e942035f238194b86175ef7b6e231a60d5c0fd53a11c37cfb0b7d0947c46e25ffe83b67a16f1b08f64ebe3de1de86440aa3e7dfea43dea023cf538f1bbc32c0fb3d8a764cea30ec98b8b67e9e4106529511f9e6e70a3fcf5c84b6ebdfd50075a449f4205def633d3e573dcb31ee4da38e4d8f3a09c2f329b7ef621b800afa561b47c82f51feb6ae3ac5d2007ec109a5caa06dddcdc8e99e1c8307ca127015ccfb02048221459bfbd14f8780f9d6b618bbd0ea4dae82e793dbdf4f1b17e3a5e198d4dfc78f900beb5679d0b849e87da86b69a62fb9afd7b6b109d6f24e1c2784af6562ec3a9c9dce6664e9310c66413d8ce0e81c00cf41e60d35308eff5e97bd21afa7069a69c6a9bc989cc8a488513f662982d337d3e4233b2f001530a4025b98d4504b015fdd907d8f84e745c61a17096a253fc41f623a180e8917dffc072fb25fb27af3bce1fabbfcb3dc23aae5bd64bf746a12e8aa2772f37d56394ef04ec4649f3a50f78d42531df037ca5565b95b1a251096edac31411ff6f88e052682908b55419685bb63cd5e1b54261da7a4850e72f734ee6a99147d6c31fefcbf4ed0e98340008bb137c4749d2d01f979017a4806609d9de32572f4b375d4ec320da554c7c5859877e11cc343f33be9efe36dd7b5eacb499bb57ced276bc8af2eef624aa208b864ce5310dcad7279b3d8451d259a5385eb568c85bff1ee0ace7f6c2d4fd6039465d1a9fd37aea6644d13af0af46b5e39f78c7248ccdd5124e75451bf07bb6631b15383d71c370244a5818f0554867d03b6e0fbae20f9c765c771987fa7f157483f4abb5161a2c00e8d7508d9c38ebc039a48d39557cd11f771c6f90ab8b9a978a0e50b06419dd8feff0a3ebd626808165e6d2643aa034b5200b6c5376ff44c95c9ee59b03c1e36c2e543ed1521cd94dcb6547045df564f5057fc534f402b841fdda49d3ab25e19854644454c4acd94ae19e3fabff185e113931e0523e99b7d97bec08c07aae27a154c9972d52f4daad110cbbf47531b026b04586846ad6680b2ab882424c4345254e7af63cbe238c1304802173cd424cad766511933f8c87aafe676fd304ecbf3f3d366ecf1ef110d87b84525d90e8513b8e3037bbe003cd4c9408379e4b17b707273b07fcc0d6ed752a7d7670b9efc81aaaf36c5fc6569a13ba21cc1c01abbf095d2942721478bab5f8587379d3fb6a1799e579f7',NULL,49157,8,NULL,NULL,0);```
ZmnSCPxj commented 4 years ago

This looks like a real bug to me, but the crashing you are experiencing seems to be due to some kind of db corruption. The root cause is what caused the db corruption in the first place, and is what should be fixed I think. Do you happen to have any logs for the original crash?

jarret commented 4 years ago

yes, the crash.log is attached in the middle of the first post, but here is it again. crash.log.20191223201609.gz From Dec 15th up until the crash

rustyrussell commented 4 years ago

pretty sure this is fixed by https://github.com/ElementsProject/lightning/pull/3434 where I ran into a similar issue.

That's now merged into master; can you please test?

Thanks!

niftynei commented 4 years ago

Going to close this for now; will reopen if it's still an issue.