Closed manreo closed 4 years ago
Roasbeef
commented
4 years ago Were there unresolved HTLCs on the commitment transaction (indicating an HTLC timeout)? If not, then it may have been channel state de-sync, in which case we have a number of open PRs that should fix the issue such as: #4430.
Crypt-iQ
commented
4 years ago Even though we have fixes for the latter case, we'd still like logs (if that's okay) to determine why nodes are disconnecting and causing this de-sync. But only if an htlc didn't time out.
manreo
commented
4 years ago Were there unresolved HTLCs on the commitment transaction (indicating an HTLC timeout)? If not, then it may have been channel state de-sync, in which case we have a number of open PRs that should fix the issue such as: #4430.
@Roasbeef Not sure how can I check this?
@Crypt-iQ I'll be haapy to send full log file, but I assume there is some private info in there, I'll like to send it privately - maybe I can send it by Slack? PM ? what is your slack username?
Crypt-iQ
commented
4 years ago @MrManPew I am "eugene" on the lnd slack
manreo
commented
4 years ago Great @Crypt-iQ , I've sent you the log on slack. Is it the same issue?
Crypt-iQ
commented
4 years ago Sorry, the logs only show the other peer disconnecting. I would need the opposite peer's logs to figure out why they're disconnecting.
manreo
commented
4 years ago The opposite side is https://ln.bitfinex.com/. Maybe @bitfinexcom, @prdn, @mafintosh can help?
manreo
commented
4 years ago Another one @Crypt-iQ , now my node force-closed a channel with @acinq. @t-bast could you help with finding out why we got disconnected ? I've added the full log to lnd Slack.
MykelSIlver
commented
2 years ago Same (force close transaction with bfx-lnd1 not sure why) happened on 0.13.1-beta commit=v0.13.1-beta. I do not know if it necessary to open a new issue, or is this still an on-going issue? The latter is still not clear for me.
Crypt-iQ
commented
2 years ago Need logs, did you force close or did they force close? Also may need a version for counterparty to help diagnose. New issue preferable
MykelSIlver
commented
2 years ago Need logs, did you force close or did they force close? Also may need a version for counterparty to help diagnose. New issue preferable
They did force close. The only log I have mentioned only
2021-09-28 04:31:09.312 [ERR] LTND: Unable to lookup witness: no witnesses
2021-09-28 04:31:09.694 [INF] CNCT: Broadcasting force close transaction 98bb6e8a45cd6344744889fb9b8276ba907a2969f950799266af66b3b59190f8, ChannelPoint(29eafccf92fa405a64f2fc965294bff4eb8d4d806638e584f2544f
5b2f6280bd:0): (*wire.MsgTx)(0x40026d95c0)({
....
nothing more detailed probably because I have not specified the debuglevel in lnd.conf. So I guess in this case I have too less information. To be honest, the only reason I posted my case here because I am curious if this is still an ongoing issue after the merge and in my lnd 0.13.1-beta commit=v0.13.1-beta version. From your answer I understand, that I should report it, but with more detailed information. I will add a line in lnd.conf debuglevel=debug so the next time I can place a more useful bug report. Thanks for your answer.
Crypt-iQ
commented
2 years ago In most cases, we need logs from the force-closing node's side to determine why they decided to force close
MykelSIlver
commented
2 years ago In most cases, we need logs from the force-closing node's side to determine why they decided to force close
AHa. Thanks. In that case how should I contact them? Any hint on that? It is big boy bitfinex i guess? Thanks in advance.
Crypt-iQ
commented
2 years ago If you know the pubkey & IP, maybe that would give you a starting point to contact them, but I don't know who your channel was with. If you are unable to contact them, I could ask around given the pubkey & IP combo
MykelSIlver
commented
2 years ago If you know the pubkey & IP, maybe that would give you a starting point to contact them, but I don't know who your channel was with. If you are unable to contact them, I could ask around given the pubkey & IP combo
Wow really appreciate your help.
Their pubkey comes from https://ln.bitfinex.com/
Their pubkey is: 033d8656219478701227199cbd6f670335c8d408a92ae88b962c49d4dc0e83e025
(is alias bfx-lnd0 not alias bfx-lnd1 mentioned earlier in this ticket, but this does not really matter)
Their IP_Node is:
033d8656219478701227199cbd6f670335c8d408a92ae88b962c49d4dc0e83e025@34.65.85.39:9735
My pubkey is (if it matters):
0204a91bb5802ad0a799acfd86ef566da03d80cc9e13acb01e680634bf64188a0d
My IP_Node is (if it matters):
0204a91bb5802ad0a799acfd86ef566da03d80cc9e13acb01e680634bf64188a0d@3kvh64k3s5gxcbimped34nncw4p2jlbfbwhnrzoljkwjwjlslurreuqd.onion:9735 (Sorry no real ip address, I run my node only via Tor.
Hopefully you or someone else can now find out how to contact them?
Thx in adv.
MykelSIlver
commented
2 years ago FYI. I just send the following private message to their Twitter channel:
Hi, for technical problems regarding lightning, how should I contact you? Your Telegram channel looks not a very safe medium to me in this case. More in detail: I have an issue regarding a forced close channel with my node to your node which has been issued by YOUR side. According to the developer of the issue this should be further investigated also on your side. The whole issue is already reported at Github: https://github.com/lightningnetwork/lnd/issues/4435#issuecomment-929121841 Thanks in advance and I hope on a response.
guggero
commented
2 years ago If you have this in your log: Broadcasting force close transaction, it means your node force closed.
MykelSIlver
commented
2 years ago If you have this in your log:
Broadcasting force close transaction, it means your node force closed.
Ooopps So my node did it?
2021-09-28 04:31:09.312 [ERR] LTND: Unable to lookup witness: no witnesses
2021-09-28 04:31:09.694 [INF] CNCT: Broadcasting force close transaction 98bb6e8a45cd6344744889fb9b8276ba907a2969f950799266af66b3b59190f8, ChannelPoint(29eafccf92fa405a64f2fc965294bff4eb8d4d806638e584f2544f5b2f6280bd:0): (*wire.MsgTx)(0x40026d95c0)({
Version: (int32) 2,
TxIn: ([]*wire.TxIn) (len=1 cap=1) {
(*wire.TxIn)(0x40071449c0)({
PreviousOutPoint: (wire.OutPoint) 29eafccf92fa405a64f2fc965294bff4eb8d4d806638e584f2544f5b2f6280bd:0,
SignatureScript: ([]uint8) <nil>,
Witness: (wire.TxWitness) (len=4 cap=4) {
([]uint8) <nil>,
([]uint8) (len=72 cap=144) {
00000000 30 45 02 21 00 c8 0e 30 c1 d8 e2 85 13 e2 59 8d |0E.!...0......Y.|
00000010 05 56 d8 55 5f e6 83 28 6b 4e 32 24 8c 8b 42 62 |.V.U_..(kN2$..Bb|
00000020 ec 22 92 3a f1 02 20 52 62 2a 0f da b3 d7 a4 c7 |.".:.. Rb*......|
00000030 d8 5e 61 23 cf 5f 98 63 1c 17 02 b2 22 0a 4f 0c |.^a#._.c....".O.|
00000040 9b 6b 93 bd 9a 25 60 01 |.k...%`.|
},
([]uint8) (len=71 cap=144) {
00000000 30 44 02 20 1f 9d 30 e0 9d a9 29 06 bf df b5 42 |0D. ..0...)....B|
00000010 a0 51 a8 f5 aa ce 13 30 ac 5b c4 78 bd 59 f5 a7 |.Q.....0.[.x.Y..|
00000020 9b c6 18 e7 02 20 0d 37 ca b8 0c f4 4b 95 97 71 |..... .7....K..q|
00000030 05 9b 6e 3d 2f 1a 54 c9 e8 41 9a 0b 08 a3 d2 c9 |..n=/.T..A......|
00000040 f0 9f 7b d5 dc 3b 01 |..{..;.|
},
([]uint8) (len=71 cap=500) {
00000000 52 21 03 3c 97 b0 37 cc eb c5 f8 a4 ba fd d9 5f |R!.<..7........_|
00000010 d8 61 d9 ae df b2 95 ae 67 bd 72 18 4f c9 29 0e |.a......g.r.O.).|
00000020 1b 15 1b 21 03 8b ce 9f 36 0d af ef 09 e4 25 75 |...!....6.....%u|
00000030 a8 d5 e9 af a0 cb 77 16 74 a5 dc 82 82 b9 de 0d |......w.t.......|
00000040 ed ec b7 e3 b5 52 ae |.....R.|
}
},
Sequence: (uint32) 2154036151
})
},
TxOut: ([]*wire.TxOut) (len=2 cap=2) {
(*wire.TxOut)(0x40041e7e60)({
Value: (int64) 7173819,
PkScript: ([]uint8) (len=34 cap=34) {
00000000 00 20 a8 b7 04 a2 f0 91 ae 0d e0 bc 17 ed 67 b9 |. ............g.|
00000010 8b 33 59 34 c0 a0 84 25 e7 c6 b9 bd ef 92 2a e3 |.3Y4...%......*.|
00000020 6a 97 |j.|
}
}),
(*wire.TxOut)(0x40041e7e80)({
Value: (int64) 9602827,
PkScript: ([]uint8) (len=22 cap=22) {
00000000 00 14 95 d7 79 91 95 bb ac 21 2f 8f 70 fb 54 a9 |....y....!/.p.T.|
00000010 74 9c 1f 06 76 67 |t...vg|
}
})
},
LockTime: (uint32) 553558940
})
2021-09-28 04:31:09.716 [INF] LNWL: Inserting unconfirmed transaction 98bb6e8a45cd6344744889fb9b8276ba907a2969f950799266af66b3b59190f8
Yes. But without the [DBG] log level you'll probably not know why.
MykelSIlver
commented
2 years ago Yes. But without the [DBG] log level you'll probably not know why.
Exactly. So I leave it as it is for now. Wonder if I got a reaction from the bitfinex private tweet.... Again: So much thanks for your time.!
lessless
commented
2 years ago We had a similar issue with lnd version 0.13.3-beta commit=v0.13.3-beta and bfx-lnd0
2022-01-27 11:53:50.565 [INF] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): force closing chan
2022-01-27 11:53:50.579 [INF] CNCT: Broadcasting force close transaction 66fbba77c37973efb0463d90c4adb2bc681f342facae1b43e2579b5edb50e61a, ChannelPoint(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): (*wire.MsgTx)(0xc01b22b400)({
PreviousOutPoint: (wire.OutPoint) 40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1,
2022-01-27 11:53:52.405 [WRN] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): deadline is passed with deadlineMinHeight=720593, heightHint=720593
2022-01-27 11:53:54.359 [INF] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): trigger chainTrigger moving from state StateCommitmentBroadcasted to StateCommitmentBroadcasted
2022-01-27 13:02:16.036 [INF] NTFN: Dispatching confirmed spend notification for outpoint=40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1, script=0 b85e58d2d07ed586df8f3cb6cf9319cffa7cccf7facef63c1685ae17fb39c8aa at current height=720598: 66fbba77c37973efb0463d90c4adb2bc681f342facae1b43e2579b5edb50e61a[0] spending 40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1 at height=720598
2022-01-27 13:02:16.039 [INF] CNCT: Local unilateral close of ChannelPoint(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1) detected
2022-01-27 13:02:16.040 [INF] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): local on-chain channel close
2022-01-27 13:02:17.789 [INF] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): trigger localCloseTrigger moving from state StateCommitmentBroadcasted to StateContractClosed
2022-01-27 13:02:17.797 [INF] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): still awaiting contract resolution
2022-01-27 15:52:02.573 [INF] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): a contract has been fully resolved!
2022-01-27 15:52:02.573 [INF] CNCT: ChannelArbitrator(40a4915568dd91d99d665ebdd697c75034b1934e5fd25b7ea6c9be917e154530:1): still awaiting contract resolutionI bumped log-level to debug and will report back if there will be anything more to add
Background
My node foce-closed channel with bfx-lnd0 so I opened a new one with bfx-lnd1 and my node also closed this channel. I did not do anything to foce-close the channel.
Here are some of the logs:
Your environment
0.10.1 Linux rock64 4.4.132-1075-rockchip-ayufan-ga83beded8524 #1 SMP Thu Jul 26 08:22:22 UTC 2018 aarch64 GNU/Linux bitcoind
Thanks