Unable to keep specific channel online

[PsySc0rpi0n]

Bitcoin Core version

$ bitcoin-cli --version
Bitcoin Core RPC client version v23.0.0

Core Lightning Version

$ lightning-cli --version
v23.02rc3-modded

Core Lightning config

$ cat .lightning/config 
#c-lightning configuration file

network=bitcoin
bitcoin-cli=/home/psysc0rpi0n/Downloads/bitcoin-23.0/bin/bitcoin-cli
bitcoin-datadir=/home/psysc0rpi0n/.bitcoin
# debug
log-level=debug
log-file=/home/psysc0rpi0n/.lightning/debug.log

alias=Crypt04n4rkMoV
rgb=ffff00
watchtime-blocks=720

important-plugin=/home/psysc0rpi0n/Downloads/plugins/backup/backup.py
plugin=/home/psysc0rpi0n/Downloads/plugins/rebalance/rebalance.py
plugin=/home/psysc0rpi0n/Downloads/plugins/circular/circular
proxy=127.0.0.1:9050
bind-addr=127.0.0.1:9735
announce-addr=2obm3yvfj5m3zabnea5y2xolimeuc4gbelika3pa7div5pk2eolqrtad.onion
always-use-proxy=true
min-capacity-sat=400000

# BOLT12
experimental-offers

liquidity avertisement
experimental-dual-fund
funder-policy=match
funder-policy-mod=100
lease-fee-base-msat=1000msat
lease-fee-basis=10
channel-fee-max-base-msat=1250msat
channel-fee-max-proportional-thousandths=1
funder-lease-requests-only=false

SCID of the channel in question: 711727x1414x1

Description

Me and @vincenzopalazzo opened this channel quite some time ago and it has been working until maybe 2 months ago when the channel became offline like for 90% or more of the time. I'll paste some of the messages I can see in my log file and @vincenzopalazzo may add more messages if I fail any important ones in my post.

When my node tries to connect to @vincenzopalazzo I can see messages like this:

03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-lightningd: Will try reconnect in 300 seconds
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-chan#83: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: warning channel 2076b72451223c7a8aec12bcac4eb61f41b988cc7a60a       6913389c5988106ae81: update_fee 5796 outside range 253-3753 (currently 5513)
...
...
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-connectd: Connect OUT
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-connectd: peer_out WIRE_INIT
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-connectd: peer_in WIRE_INIT
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-gossipd: seeker: disabling gossip
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-chan#83: Peer transient failure in CHANNELD_NORMAL: Disconnected
...
...
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-chan#83: Peer has reconnected, state CHANNELD_NORMAL: connecting subd
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-channeld-chan#83: pid 133812, msgfd 82
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-chan#83: Already have funding locked in (and ready to announce)
...
...
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-chan#83: attempting update blockheight 2076b72451223c7a8aec12bcac4eb61f41b988cc7a60a6913389c5988106ae81
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-channeld-chan#83: option_static_remotekey = 1, option_anchor_outputs = 0
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-channeld-chan#83: init LOCAL: remote_per_commit = 03f5a1f2ef89517c6043e221025581005e3f9f54ef9238d1697e67b97c66d0ca45, old_remote_pe       r_commit = 02b5512f6bb7b719a7edfde9f41edb25d62b17a49e29cf727019023f8af7b83a9a next_idx_local = 16963 next_idx_remote = 16964 revocations_received = 16962 feerates { SENT_ADD_COMMIT:5796 SENT_ADD_ACK_REVOCATION:5513 } range 253-3754       5 blockheights { SENT_ADD_ACK_REVOCATION:0 }, our current 777547
...
...
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-channeld-chan#83: peer_in WIRE_WARNING
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-channeld-chan#83: billboard perm: Received warning channel 2076b72451223c7a8aec12bcac4eb61f41b988cc7a60a6913389c5988106ae81: update       _fee 5796 outside range 253-3753 (currently 5513)
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-lightningd: peer_disconnect_done
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-lightningd: Will try reconnect in 300 seconds
03e2408a49f07d2f4083a47344138ef89e7617e63919202c92aa8d49b574a560ae-chan#83: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: warning channel 2076b72451223c7a8aec12bcac4eb61f41b988cc7a60a       6913389c5988106ae81: update_fee 5796 outside range 253-3753 (currently 5513)

I have tried to deisable experimental-offers, experimental-offers AND experimental-dual-fund and experimental-dual-fund but none of the 3 options made the chanel online!

@vincenzopalazzo can I add anything else to the report? What coul be the problem and what could be the solution?

[vincenzopalazzo]

The core error is the following one update_fee 5796 outside range 253-3753 (currently 5513), apparently you are proposing a fee too high, and in this case, the specs say that we should close the connection.

Now you fix your case we should understand why your are proposing this kind of fee, and my node rejected it.

Could you run the following bitcoin command and report the output?

bitcoin-cli estimatesmartfee 6

Mine is

{
  "feerate": 0.00001186,
  "blocks": 6
}

lightning-cli feerates perkb

mine is

{
   "perkb": {
      "opening": 1012,
      "mutual_close": 1012,
      "unilateral_close": 1012,
      "delayed_to_us": 1012,
      "htlc_resolution": 1012,
      "penalty": 1012,
      "min_acceptable": 1012,
      "max_acceptable": 5692
   },
   "onchain_fee_estimates": {
      "opening_channel_satoshis": 177,
      "mutual_close_satoshis": 170,
      "unilateral_close_satoshis": 151,
      "htlc_timeout_satoshis": 167,
      "htlc_success_satoshis": 177
   }
}

[PsySc0rpi0n]

Hi @vincenzopalazzo. Here are the 2 outputs from my nodes

$ bitcoin-cli estimatesmartfee 6

{
  "feerate": 0.00005680,
  "blocks": 6
}

$ lightning-cli feerates perkb

{
   "perkb": {
      "opening": 1452,
      "mutual_close": 1452,
      "unilateral_close": 2124,
      "delayed_to_us": 1452,
      "htlc_resolution": 2124,
      "penalty": 1452,
      "min_acceptable": 1012,
      "max_acceptable": 119060
   },
   "onchain_fee_estimates": {
      "opening_channel_satoshis": 254,
      "mutual_close_satoshis": 244,
      "unilateral_close_satoshis": 317,
      "htlc_timeout_satoshis": 352,
      "htlc_success_satoshis": 373
   }
}

[hMsats]

I run Core Lightning version v23.02.1 and have the same problem with node 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f ("BCash_Is_Trash"). Have 2 channels open with this node but they are now disconnected (since a week or so). All my other 15 nodes are OK.

EDIT: I have added --ignore-fee-limits=true to lightningd (as suggested here) and now the node connects (but this seems to be a bit dangerous). EDIT: have now removed it!

Error:

2023-03-13T16:30:41.752Z INFO 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f-chan#49342: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: update_fee 253 outside range 526-39608 (currently 263)

bitcoin-cli estimatesmartfee 6 { "feerate": 0.00010697, "blocks": 6 }

lightning-cli feerates perkb { "perkb": { "opening": 4968, "mutual_close": 4208, "unilateral_close": 10700, "delayed_to_us": 4968, "htlc_resolution": 10700, "penalty": 4968, "min_acceptable": 2104, "max_acceptable": 158432 }, "onchain_fee_estimates": { "opening_channel_satoshis": 871, "mutual_close_satoshis": 707, "unilateral_close_satoshis": 1599, "htlc_timeout_satoshis": 1773, "htlc_success_satoshis": 1880 } }

lightning-cli connect 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f { "id": "0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f", "features": "2000080269a2", "direction": "out", "address": { "type": "ipv6", "address": "2601:18c:8080:300f:219:d1ff:fe75:dc2f", "port": 9735 } }

seems OK but the node remains disconnected.

[pabpas]

I also have the same problem ONLY with node 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f ("BCash_Is_Trash").

I have dug into my info log and this has been going on for a month. Interesting that update_fee 253 is the same for user hMsats and has been like that since this started happening.

2023-02-16T23:24:22.778Z INFO    0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f-chan#89: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: update_fee 253 outside range 392-67538 (currently 264)
2023-02-16T23:24:24.387Z INFO    0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f-chan#89: Peer transient failure in CHANNELD_NORMAL: Disconnected
2023-02-16T23:24:35.675Z INFO    0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f-chan#89: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: update_fee 253 outside range 392-67543 (currently 264)
...
2023-03-14T07:02:52.716Z INFO    0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f-chan#89: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: update_fee 253 outside range 526-47703 (currently 263)

[hMsats]

I also have the same problem ONLY with node 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f ("BCash_Is_Trash").

@pabpas This is useful information because this made me remember that "BCash_Is_Trash" has implemented some kind of intelligent fee scaling, as described here which seems to cause the problem.

[whitslack]

intelligent fee scaling

That has nothing to do with this. That's about Lightning payment forwarding fees. The warnings being discussed here are about mining fees for commitment transactions. The node 0298 ("BCash_Is_Trash") wants a greater commitment transaction fee rate than your node is proposing and is refusing a fee rate that it sees as too low. This is likely because of mempool congestion on the Bitcoin network in recent weeks. The Lightning protocol is operating as designed, preventing a channel from connecting while its peers disagree about acceptable on-chain transaction fee rates. Indeed, 253 sats likely would not be sufficient to get an on-chain transaction confirmed in a safe amount of time under present mempool conditions.

[pabpas]

Agree that the protocol is working as expected, but why is node 0298 offering the lowest possible fee? Using --ignore-fee-limits=true is a no-go especially in the current mempool environment.

[whitslack]

I may have misinterpreted the warning message. (It's not clear from the log which node sent the update_fee and which sent the warning.) I don't know why 0298 is proposing such a low fee.

It seems like the protocol spec was botched in this regard. The sender of update_fee has a range of fee rates it would accept, but it can propose only a singular rate. Thus, a pair of channel peers may fail to agree on a fee rate despite that their respective ranges of acceptable fee rates do overlap. It seems like update_fee really should have been specified to send a range, from which the receiver could select the lowest rate it is willing to accept and transmit its commitment transaction with the selected fee rate. Am I missing a nuance here?

[whitslack]

bitcoin-cli estimatesmartfee 6

Why 6? The commit-fee config variable gives “the percentage of estimatesmartfee 2/CONSERVATIVE to use for the commitment transactions.” That's 2, not 6.

0298 has commit-fee=5 and

$ bitcoin-cli estimatesmartfee 2 CONSERVATIVE
{
  "feerate": 0.00032600,
  "blocks": 2
}

Note that estimatesmartfee returns fee rates in BTC/kvB, but the fee rates in CLN log messages are stated in sat/kWU, so you have to convert:

(0.00032600 BTC/kvB) × (1e8 sat/BTC) × (0.25 kvB/kWU) = 8150 sat/kWU

5% of 8150 sat/kWU is ~408 sat/kWU, which is greater than 253 sat/kWU (although still less than the 526 sat/kWU minimum that @pabpas is demanding). I don't know where that 253 is coming from.

[whitslack]

The commit-fee config variable gives “the percentage of estimatesmartfee 2/CONSERVATIVE to use for the commitment transactions.” That's 2, not 6.

I think the man page is wrong. The perkw.unilateral_close returned from lightning-cli feerates perkw matches the feerate returned from bitcoin-cli estimatesmartfee 6, times 2.5e7, times commit-fee/100.

[pabpas]

I think the man page is wrong.

On my machine the calculation matches yours. It looks like the man page is not accurate.

I don't know where that 253 is coming from.

This is the main issue here. If the text about --force-feerates in the manual applies also for mainnet, 253sat/kw is the fallback when CLN can't get any fee estimates.

[hMsats]

@pabpas I removed the "--ignore-fee-limits=true" and after a restart no longer have any problems with node 0298f607 ("BCash_Is_Trash"). Maybe something has changed (at his side). Are your problems over too?

[pabpas]

@hMsats after doing the same you did, the problem is over. Thanks!

In the end the feerate with the peer was not the minimum but:

"feerate": {
                  "perkw": 1991,
                  "perkb": 7964

In the following fee environment:

$ lightning-cli feerates perkb
{
   "perkb": {
      "opening": 15000,
      "mutual_close": 4212, 
      "unilateral_close": 15924,
      "delayed_to_us": 15000,
      "htlc_resolution": 15924,
      "penalty": 15000,
      "min_acceptable": 2104,
      "max_acceptable": 269980
   },
   "onchain_fee_estimates": {
      "opening_channel_satoshis": 2632,
      "mutual_close_satoshis": 708,
      "unilateral_close_satoshis": 2380,
      "htlc_timeout_satoshis": 2639,
      "htlc_success_satoshis": 2798
   }
}

The mystery remains on why it got stuck for one month offering only the minimum feerate.

[pabpas]

The same issue is still happening from time to time with node 0298f607 ("BCash_Is_Trash"), probably because of the aggressive setting of commit-fee=5, but last time it was not stuck with the minimum fee of 253 but with 574 and my node had a min_acceptable=581. Again the same value was stuck for days, --ignore-fee-limits unblocked it and the final agreed fee was much higher (2086), maybe this indicates an issue the how CLN manages update_fee?

[whitslack]

maybe this indicates an issue the how CLN manages update_fee?

Commitment transaction fees seem to have been poorly designed in the Lightning spec. Each party to a channel knows a range of fee rates that it would accept, yet only the channel initiator can communicate a desired fee rate, and it can only communicate a single rate, not its range. A sane protocol would have both sides communicate their acceptable range, and the effectively negotiated rate would be the minimum rate that is within both ranges (or if the ranges don't overlap, the channel automatically gets force-closed).

[pabpas]

I expected the fee negotiation to work like in a mutual close, but even if it is a shortcoming of the spec, I don't think that 0298f607 wants to offer always the same fee. The mempool is changing all the time and the commit tx fee should also change. Here it is not updating for some reason until it is accepted by my node, then a new and reasonable fee will be agreed on without problem.

[pabpas]

I had almost forgotten about this and now I have 3 peers with this issue. This time they seem to be LND (based on their TimeLockDelta) but the behavior is the same: they keep offering only the lowest possible fee (update_fee 253) until I restart with ignore fee limits, then a normal fee will be agreed upon. The only common thing I see among them is that they are all remote openers.

UPDATE: with v23.05 it seems that the workaround of temporarily using ignore-fee-limits=true is not working any more, e.g.

INFO    03cea9082ce9916fefe91e6212e5b61960bc151e7ec404b327e75a705c77d803b1-chan#199: Peer transient failure in CHANNELD_NORMAL: channeld WARNING: update_fee 253 outside range 1333-4294967295 (currently 14713)

[vincenzopalazzo]

This was a problem of feature mismatch :/ unfortunately

So the only solution is to wait that the fee will match again in the correct one or use what @pabpas suggested to ignore the fee limit.

In the case @PsySc0rpi0n and me, I had a bug in my plugin pruning mode, so now the problem should be fixed,

[vincenzopalazzo]

I am Closing this because the original issue was caused by my buggy plugin, so if there is something that I am missing here, please feel free to open another one with the information for debugging.