lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.56k stars 2.06k forks source link

[bug]: Channel down. Connectivity good. Logs showing ping-pong ok. "...read next header: EOF" #8331

Open CoanLuciano opened 6 months ago

CoanLuciano commented 6 months ago

Background

Me : LND 0.17.3, bitcoind 25 (full indexed, mempool = 2Gb), Ubuntu, hybrid mode My peer : LND 0.17.3, bitcoind 25, Ubuntu, hybrid

I will put as many info as I think can help. Sorry for the long list.

Collected logs from my node:

2024-01-02 19:40:49.095 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending Ping(ping_bytes=0000002ca6054fadad240dc7ff229a6c5885e5b0fd2240615540010000000000000000004ee761c92773f379599bd49800cfc7bdfb4470967de11d81edc8d77642ce7d5445669465b3e803175ceeb8ac) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
2024-01-02 19:40:49.336 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received Ping(ping_bytes=0000002ca6054fadad240dc7ff229a6c5885e5b0fd2240615540010000000000000000004ee761c92773f379599bd49800cfc7bdfb4470967de11d81edc8d77642ce7d5445669465b3e803175ceeb8ac) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
2024-01-02 19:40:49.336 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending Pong(pong_bytes=00000000000000000000000000000000) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
2024-01-02 19:40:49.352 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received Pong(pong_bytes=00000000000000000000000000000000) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
**2024-01-02 19:40:54.614 [INF] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): unable to read message from peer: read next header: EOF**
2024-01-02 19:40:54.614 [INF] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): disconnecting 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666, reason: read handler closed
2024-01-02 19:40:54.714 [DBG] SRVR: Peer 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666 has been disconnected
2024-01-02 19:40:54.714 [DBG] FNDG: Cancelling all reservations for peer 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
2024-01-02 19:40:54.714 [DBG] FNDG: No active reservations for node: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
2024-01-02 19:40:54.714 [INF] DISC: Removing GossipSyncer for peer=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
2024-01-02 19:40:54.714 [DBG] SRVR: Notifying that peer 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666 is offline
2024-01-02 19:40:54.714 [DBG] SRVR: removing peer 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
2024-01-02 19:40:54.714 [DBG] PRNF: PeerNotifier notifying peer: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6 offline
2024-01-02 19:40:54.714 [DBG] DISC: Stopping GossipSyncer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6)
2024-01-02 19:40:54.715 [DBG] DISC: GossipSyncer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6) stopped
2024-01-02 19:42:54.559 [DBG] RPCS: [connectpeer] requested connection to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:54.559 [DBG] SRVR: Connecting to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:54.559 [DBG] RPCS: Connected to peer: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:54.559 [DBG] CMGR: Attempting to connect to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735 (reqid 182)
2024-01-02 19:42:54.560 [DBG] RPCS: [connectpeer] requested connection to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
2024-01-02 19:42:54.560 [WRN] SRVR: Already have 1 persistent connection requests for 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666, connecting anyway.
2024-01-02 19:42:54.560 [DBG] SRVR: Connecting to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
2024-01-02 19:42:54.560 [DBG] RPCS: Connected to peer: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666
2024-01-02 19:42:54.560 [DBG] CMGR: Attempting to connect to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666 (reqid 183)
2024-01-02 19:42:58.800 [DBG] CMGR: Connected to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735 (reqid 182)
2024-01-02 19:42:58.800 [INF] SRVR: Established connection to: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:58.800 [INF] SRVR: Finalizing connection to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735, inbound=false
2024-01-02 19:42:58.800 [DBG] CMGR: Canceling: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666 (reqid 183)
2024-01-02 19:42:58.800 [DBG] PRNF: PeerNotifier notifying peer: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6 online
2024-01-02 19:42:58.800 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending Init to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.543 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received Init from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.543 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Loaded 1 active channels from database
2024-01-02 19:42:59.558 [INF] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Loading ChannelPoint(f72ee2c9b466c4a0aba77bb247b7e96295c6894952dc45b09d6a3fe4a085942e:0), isPending=false
2024-01-02 19:42:59.559 [INF] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Negotiated chan series queries
2024-01-02 19:42:59.559 [DBG] SRVR: Notifying that peer 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735 is online
2024-01-02 19:42:59.559 [INF] DISC: Creating new GossipSyncer for peer=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
2024-01-02 19:42:59.559 [DBG] DISC: Created new GossipSyncer[state=chansSynced type=PassiveSync] for peer=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.559 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending ChannelReestablish(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, next_local_height=624566, remote_tail_height=626204) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.559 [DBG] DISC: Starting GossipSyncer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6)
2024-01-02 19:42:59.559 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending NodeAnnouncement(node=023369b071005d376bfa94b76d2ec52d01d6709f57d6b67bf26250f2ef57a66aea, update_time=2024-01-02 19:42:59 +0000 UTC) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.559 [DBG] DISC: GossipSyncer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): state=chansSynced, type=PassiveSync
2024-01-02 19:42:59.591 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received NodeAnnouncement(node=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6, update_time=2024-01-02 19:42:59 +0000 UTC) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.591 [DBG] CRTR: Waiting for dependent on job=lnwire.NodeAnnouncement, pub=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
2024-01-02 19:42:59.591 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received ChannelReestablish(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, next_local_height=626205, remote_tail_height=624565) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.591 [DBG] DISC: Processing NodeAnnouncement: peer=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735, timestamp=2024-01-02 19:42:59 +0000 UTC, node=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
2024-01-02 19:42:59.594 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending UpdateFailHTLC(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, id=153171, reason=32fe027ae7c710204cbc716729ce3e3bebebf2675816d2089366c42a206c963131a04c8c16055f50eb6d21c6a136d476f0b14c1b0a9aca43b9349e1e937cefeaeb9319f688b90d057f09d03efd530238bd33684d9322f480aa805261e6b51ce4aaeafae2818718db509896f85a7cbfc923710a95c4c20020315b201e614d214ca606b21055514e5dfe0719338d919a1fc9e927b5a8082975b6d576db58921e912421b05402d307a58c23a7bdaaf816092075d0833512f5f33e157317820864e34541ae69595bbb1942bd5b989d6ea95cdd7435e10ff2b4978a57d56aada0b41227af3e007db4ea3f15873a56c9ae672d308fee094d01d24b6e14d92590ea6ac449282fd3de018c5d38f688f73d0d29c93bcad9c4be0c93224412ecf5d13774804ddb712f) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.594 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending CommitSig(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, num_htlcs=13) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:43:00.099 [DBG] DISC: Processed NodeAnnouncement: peer=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735, timestamp=2024-01-02 19:42:59 +0000 UTC, node=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
2024-01-02 19:43:00.344 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received UpdateFailHTLC(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, id=191493, reason=028e711f2414f92eb7e9467d24e15dca9acb9f245be31558396852fe2fb45e0f3992c2d95cac172e771c77b4a8ce63dbe5ab9e141ff39be271be445b513f8dbfe8f83ffbda047c0b8f77d31f2e93ed10f6dd16314048a5b035cf3fd7cb7fb721d73774744d731799128330e9fbf0a8e5a73fb1fbb8094c0eb27bcf1688a9260cedefaa25fdf9c245ea098b10b822bdc5393db86569d3e50e1a1d24f1a6d9ee990f0cc7e248d41b23e184d212900669592e2cfcdf92f114b519b30a03eb073de801ee39e565caff13214be8dcf65c423daf54462b222105fcc34154795b9358d60c4ef0da68e755f17d95a2a28821e9bfa86d9eeda87211f9942ae0ab2dcd2aa571917959692ed82e53a9b7119ce2c7aa223e4e577bb9944f901bc10a96b296eb2225c765) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:43:00.345 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received Error(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, err=internal error) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:43:00.345 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending Error(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, err=remote error) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:43:13.657 [DBG] CMGR: Ignoring connection for canceled conn req: 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:59666 (reqid 183)
2024-01-02 19:43:16.114 [DBG] RPCS: [listpeers] yielded 
2024-01-02 19:43:18.689 [DBG] CRTR: Reporting pair success to Mission Control: pair=026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2 -> 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6, amt=10021100 mSAT
2024-01-02 19:43:18.689 [DBG] CRTR: Setting 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2->023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6 range to [19804593 mSAT-0 mSAT]
2024-01-02 19:43:18.689 [DBG] CRTR: Reporting pair failure to Mission Control: pair=023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6 -> 02e4971e61a3f55718ae31e2eed19aaf2e32caf3eb5ef5ff03e01aa3ada8907e78, amt=10021000 mSAT
2024-01-02 19:43:18.689 [DBG] CRTR: Setting 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6->02e4971e61a3f55718ae31e2eed19aaf2e32caf3eb5ef5ff03e01aa3ada8907e78 range to [0 mSAT-10021000 mSAT]
2024-01-02 19:43:40.606 [DBG] RPCS: [listpeers] yielded 

{LOTS OF PEERS}

2024-01-02 19:43:57.272 [DBG] SRVR: Sending 0 messages to peer 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6
**2024-01-02 19:43:57.749 [INF] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): unable to read message from peer: read next header: EOF**
2024-01-02 19:43:57.749 [INF] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): disconnecting 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735, reason: read handler closed
2024-01-02 19:43:57.849 [DBG] SRVR: Peer 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735 has been disconnected
Roasbeef commented 6 months ago

That happens when the remote peer hangs up the connection. So unless there's some other issue, I think it's something with that remote peer. They keep hanging up the TCP connection.

jvxis commented 6 months ago

That happens when the remote peer hangs up the connection. So unless there's some other issue, I think it's something with that remote peer. They keep hanging up the TCP connection.

Thanks for the reply. My node is connected and working well. Why is it hanging up only this peer? Can it be a side effect of a bigger problem, like the channel reserve bug? Thanks JVX

CoanLuciano commented 6 months ago

I am talking with my peer, and it is happening only with our channel. What could cause a hang up? Is there any other debug information that would help? Any test that can be done? I have 98% outbound. max-channel-fee-allocation=1 already configured by me and my peer. Channel reserve bug?

CoanLuciano commented 6 months ago

I think we have found the problem. The Ballance in each peer is different. Check the images. Screenshot_20240102-185228 IMG_20240102_185624_732

How can we solve this situation?

ziggie1984 commented 6 months ago

These two log-lines stand out:

2024-01-02 19:43:00.344 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received UpdateFailHTLC(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, id=191493, reason=028e711f2414f92eb7e9467d24e15dca9acb9f245be31558396852fe2fb45e0f3992c2d95cac172e771c77b4a8ce63dbe5ab9e141ff39be271be445b513f8dbfe8f83ffbda047c0b8f77d31f2e93ed10f6dd16314048a5b035cf3fd7cb7fb721d73774744d731799128330e9fbf0a8e5a73fb1fbb8094c0eb27bcf1688a9260cedefaa25fdf9c245ea098b10b822bdc5393db86569d3e50e1a1d24f1a6d9ee990f0cc7e248d41b23e184d212900669592e2cfcdf92f114b519b30a03eb073de801ee39e565caff13214be8dcf65c423daf54462b222105fcc34154795b9358d60c4ef0da68e755f17d95a2a28821e9bfa86d9eeda87211f9942ae0ab2dcd2aa571917959692ed82e53a9b7119ce2c7aa223e4e577bb9944f901bc10a96b296eb2225c765) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:43:00.345 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received Error(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, err=internal error) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735

Your Peer is sending you an internal error for this channel, the detailed logs why he sends this error should be found in his logs.

jvxis commented 6 months ago

These two log-lines stand out:

2024-01-02 19:43:00.344 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received UpdateFailHTLC(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, id=191493, reason=028e711f2414f92eb7e9467d24e15dca9acb9f245be31558396852fe2fb45e0f3992c2d95cac172e771c77b4a8ce63dbe5ab9e141ff39be271be445b513f8dbfe8f83ffbda047c0b8f77d31f2e93ed10f6dd16314048a5b035cf3fd7cb7fb721d73774744d731799128330e9fbf0a8e5a73fb1fbb8094c0eb27bcf1688a9260cedefaa25fdf9c245ea098b10b822bdc5393db86569d3e50e1a1d24f1a6d9ee990f0cc7e248d41b23e184d212900669592e2cfcdf92f114b519b30a03eb073de801ee39e565caff13214be8dcf65c423daf54462b222105fcc34154795b9358d60c4ef0da68e755f17d95a2a28821e9bfa86d9eeda87211f9942ae0ab2dcd2aa571917959692ed82e53a9b7119ce2c7aa223e4e577bb9944f901bc10a96b296eb2225c765) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:43:00.345 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Received Error(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, err=internal error) from 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735

Your Peer is sending you an internal error for this channel, the detailed logs why he sends this error should be found in his logs.

The channel was Forced Closed yesterday. @ziggie1984 Why did we have this difference in channel balance?

ziggie1984 commented 6 months ago

That is normal, your peer and you do have a different version of the commitment transaction. During the build-up of an HTLC the states of those 2 commitment transactions can be different. And according to your logs they are different because you sent your peer this:

2024-01-02 19:42:59.594 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending UpdateFailHTLC(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, id=153171, reason=32fe027ae7c710204cbc716729ce3e3bebebf2675816d2089366c42a206c963131a04c8c16055f50eb6d21c6a136d476f0b14c1b0a9aca43b9349e1e937cefeaeb9319f688b90d057f09d03efd530238bd33684d9322f480aa805261e6b51ce4aaeafae2818718db509896f85a7cbfc923710a95c4c20020315b201e614d214ca606b21055514e5dfe0719338d919a1fc9e927b5a8082975b6d576db58921e912421b05402d307a58c23a7bdaaf816092075d0833512f5f33e157317820864e34541ae69595bbb1942bd5b989d6ea95cdd7435e10ff2b4978a57d56aada0b41227af3e007db4ea3f15873a56c9ae672d308fee094d01d24b6e14d92590ea6ac449282fd3de018c5d38f688f73d0d29c93bcad9c4be0c93224412ecf5d13774804ddb712f) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735
2024-01-02 19:42:59.594 [DBG] PEER: Peer(023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6): Sending CommitSig(chan_id=2e9485a0e43f6a9db045dc524989c69562e9b747b27ba7aba0c466b4c9e22ef7, num_htlcs=13) to 023e24602891c28a7872ea1ad5c1bb41abe4206ae1599bb981e3278a121e7895d6@201.68.114.49:9735

essentially progressing the state without an HTLC, but your peer did not sent you the Commit_Sig for this new state but sent you an error, you need to ask your peer what his logs say in detail, does he run on 17.2 ?