Very expensive rebalance that way more than forward fee earning

[btweenthebars]

Based on discussion over here :

From what I have observed so far, the very cost rebalance ones came at the moment when I opened new channel. It could spend thousands of sat to rebalance which from the fee that clboss set and the traffic I see, my node has no chances to recover the spending.

ZmnSCPxj has some ideas to improve this here

Moreover, I would also say that the InitialRebalance is reckless too. When I first started turning clboss on my existing node, I gave it 20M to play with. It made 2 big channels to medium size node with only 20ish peers. Then the InitialRebalance kicked in, spent around 5000+ sat. For 2 weeks, I only made 30sat from one of them. I'm not sure what's the idea behind this but, if I made channel to a random node, I would have done nothing at first and see if other peers like to use the new node's outbound, if it's quiet, then I lower the fee, then I rebalance bit by bit and see what it does and of course if it's too costly to rebalance I will just sit tight wait for the future luck.

[ZmnSCPxj]

I would have done nothing at first and see if other peers like to use the new node's outbound, if it's quiet, then I lower the fee, then I rebalance bit by bit and see what it does and

Fee is done by another module. The main one is FeeModderByPriceTheory, you can see its report under price_theory_investigating key of lightning-cli clboss-status command output. 0 means the median price of competitors on that node, +1 +2 etc. means higher than the median price, -1 -2 means lower, it investigates various fee levels for a few days each, then selects the one which earned most money, then investigates several variations around the selected level, ad infinitum. This is based on the theory that there is some global maxima on the feerate, which we can discover using a hill-climbing algorithm.

InitialRebalancer usually kicks in on a channel with only outgoing capacity, the intent is to enable forwardings from that node by allocating some incoming capacity from them. In retrospect where we get that incoming capacity from should be selected by being cheap while still having incoming capacity to spare.

of course if it's too costly to rebalance I will just sit tight wait for the future luck.

What percent of the channel capacity is "too costly"? InitialRebalancer should allocate about 0.04% of channel capacity for its speculative rebalance in total before giving up (so 0.04% of 20M sats is 8000 sats, there is some fuzziness here due to roundoff and onchain fees and smaller channels existing), do you think should this be lowered?

[btweenthebars]

InitialRebalancer usually kicks in on a channel with only outgoing capacity, the intent is to enable forwardings from that node by allocating some incoming capacity from them.

I hear you, however it does not apply to nodes that can use incoming capacity. For instace, if I make a channel to liquidity sink such as Loop, I don't want to rebalance at all and set very high fee in the beginning, although liquidity sink is an extreme example. For good and busy nodes they don't need rebalance at all, just set low enough fee, then the liquidity will move to another side for free. When the channel reaches the target of outgoing/incoming ratio clboss can up the fee. So clboss InitialRebalance should wait and observe before deciding what to do, don't you think ?

Maybe there's a need to try to categorize peers and apply different strategies ?

What percent of the channel capacity is "too costly"? InitialRebalancer should allocate about 0.04% of channel capacity for its speculative rebalance in total before giving up (so 0.04% of 20M sats is 8000 sats

It shouldn't be static. Maybe basing on the earning of a node, for example if a node makes 1000sat a month, and clboss does 8000sat IntialRebalance for a channel, the operator would fall off the chair.

[ZmnSCPxj]

It shouldn't be static. Maybe basing on the earning of a node

EarningsRebalancer already bases off the earnings of the node, so what you are effectively saying is that InitialRebalancer should not exist.

The justification of InitialRebalancer is that prior to dual funding, a new channel we created is always going to be fully outgoing. Yet the node we make a channel to might have been selected because it is a good forwarding node, which is bidirectional --- the node might get HTLCs that it will forward to us, but if the channel is only going to them, it is not possible. InitialRebalancer is intended to get out of this situation, by redistributing incoming capacity a little more evenly among channels, letting all channels have a chance to get incoming forwards to us..

I suppose we could try relying on only JitRebalaner and EarningsRebalancer instead. After all, both are based on previous evidence that the rebalance would likely be a good idea: JitRebalancer because it is actually needed for a forward right now, and EarningsRebalancer based on historical statistics. You could experimet this by modifying Boss/Mod/all.cpp and commenting out the InitialRebalancer line.

[ZmnSCPxj]

I rebalance bit by bit

Because many nodes have non-zero base fee, this will end up spending more than just making one big rebalance. One can argue that rebalances are really a way of aggregating multiple small payments, and that nonzero base fee is needed to facilitate this: individual payers make lots of payments (each one paying base fees to the forwarding node), and every now and then a forwarding node rebalances (paying base fees only once, since it just rebalances once).

[btweenthebars]

I turned the debug on. I'm seeing a few successful EarningRebalance, but I have trouble figuring out what a successful JitRebalance looks like.

I see one line that stood out : JitRebalancer: HTLC 167720: Move 1025062817msat from AAA to BBB.

but I don't see any forwarding from BBB to any other nodes, and the capacity of BBB hasn't changed at all.

[ZmnSCPxj]

Rebalancing attempts can fail for the same reason that payments can fail. FundsMover does not report who triggered the rebalane, unfortunately, it probably should be informed the name of the module that triggered it. Also, JitRebalancer triggers "to BBB" if the outgoing forward is to BBB, that is why it is pushing funds at BBB, because we need to forward to BBB; you should be looking for forwards "to BBB", not from.

Rebalancing can take a long time, too, just as normal payments can occasionally experience black swan events where they are delayed for a long time. JitRebalancer hooks into HtlcAcceptor, which handles htlcs_accepted hooks from C-Lightning, and which can be used to defer incoming HTLCs. Since JitRebalancer rebalances can take a long time (due to network circumstances beyond our control) the policy of HtlcAcceptor is to limit the number of seconds that an HTLC can be deferred. That liit is currently 6 seconds. If it takes longer than that, then HtlcAcceptor will let the HTLC continue normal processing, and if the JitRebalaner rebalance has not completed by then, the underlying C-Lightning will fail the forward at our node at that point.

Removing that time limit would not be good for the network. Consider what if everyone were running CLBOSS. Then it is possible that a forwarding node triggers its own JitRebalancer when it gets a forward. This creates a payment attempt (to itself, which is what rebalances *are*). That goes through another forwarding node as well, which itself could find it needs to triggerJitRebalancer`, and so on, until large parts of the network are now doing rebalances triggered by one payment. And that payment has to wait for all of those rebalancces to finish. Sigh.

[carboncls]

On this point, it would be nice to be able to look up the initial rebalance. I don't find it with listpays or listinvoices. From clboss-status > out_expenditures I gather it spent 415979 (I assume msat, would also be nice to have this spelled out).

[btweenthebars]

I have a sink peer that I manually manage its fee (cl-unmanage lnfee) and run my own rebalancing script for it. I know that for this channel, occasionally, I can rebalance it for the cost of 550ppm and can route with 700ppm easy. I did this for a while and today I see a lot of clboss rebalance FundsMover line, into this channel with insane fee ranging from 1500ppm to 11384ppm (spent 566sat to move only 48864 sat, why did it do that ?). Not sure who triggered it. Is there a way to tell clboss to not rebalance some particular channels or a way to set max fee for them ?

The reason I have to set fee myself for these channels is the clboss fee strategy that relies on the ratio of inbound/outbound is not suitable for sink or source channels.

I guess disasters can come when not handling 100% control to clboss.

[ZmnSCPxj]

What version? 0.10 is known to be too aggressive with rebalancing.

[heyambob]

Btw, the known initial rebalance could be an attacking vector. I heard people hunting for the list clboss nodes to connect to and set high fee initially. Or this could be a good promotion for clboss nodes.

Shouldn't this be configurable? Initial rebalance or not/ how much ppm/depending on size of peer or fee that other peers set to him/try low fee first few days before initial rebalance/sometimes random , as such

[ZmnSCPxj]

I heard people hunting for the list clboss nodes to connect to and set high fee initially.

InitialRebalancer rebalances from channels that the CLBOSS node creates, so those people are gambling that the CLBOSS node creates a new channel later and their node is selected to be the destination.

shrug hey at least you get inbound liquidity, at a cheaper price than using Boltz Exchange.

I do have a branch which changes rebalancing somewhat, though. Specifically, rebalancers have the option of specifying only the source or only the destination, and the funds mover will select a destination / source that is cheap to move to / from. Not finished yet.

[heyambob]

Specifically, rebalancers have the option of specifying only the source or only the destination, and the funds mover will select a destination / source that is cheap to move to / from. Not finished yet.

Also, moving from source node whenever it can using cheap fee to other nodes would make a lot of sense.

Clboss should monitor traffic, if in is a lot more than out and the channel always has most liquidity. It is a source channel and should actively rebalance from it to non-source nodes.

[ZmnSCPxj]

Clboss should monitor traffic, if in is a lot more than out and the channel always has most liquidity. It is a source channel and should actively rebalance from it to non-source nodes.

See EarningsRebalancer, which I think is the inverse of what you said.

• Identifies channels with low outgoing-capacity capacity on our side.
• Moves funds from nodes that have been good sources.
  • "Source" is identified by keeping track of earned fees from incoming HTLCs over the channel with them.

[heyambob]

In practice, it's costly trying to rebalance into a depleted channel that's why it's depleted in the first place. The EarningRebalance might do it at loss or rarely run successfully.

My idea is different. Here's a real life scenario.

Most people having channels with Bolt or Zfr know that these channels push liquidity back pretty quick, even they set fee to towards them to 0, they would be sitting on the liquidity on their side most of the time. A lot of traffic back and forth though, but they only earn fee when the forwards come from other channels into them.

If clboss can activately get liquidity out of these source channels using low fee, doesn't matter what channels they goes into, the profit & increase in traffic is almost guaranteed.

That's one thing people using LNDg are doing.

[ZmnSCPxj]

Yes, I did say it was the inverse.

I do not quite understand what you mean here?

Most people having channels with Bolt or Zfr know that these channels push liquidity back pretty quick, even they set fee to towards them to 0, they would be sitting on the liquidity on their side most of the time. A lot of traffic back and forth though, but they only earn fee when the forwards come from other channels into them.

Who is setting 0 fee? Bolt or the peers of Bolt? Who is "their side", Bolt or the peers of Bolt? What do you mean "only earn fee when the forwards come from other channels into them", show is "they" and who is "them"?

Is this Boltz exchange or is this something else entirely?

[heyambob]

Sorry I mean, the Boltz.exchange node (1000 channels). 450 peers of Boltz set 0 or 1 ppm towards Boltz because, even though they don't earn fee by forwarding from their Boltz channels, they earn by forwarding into Boltz channels. Zfr is the Zero fee routing node.

[ZmnSCPxj]

450 peers of Boltz set 0 or 1 ppm towards Boltz because, even though they don't earn fee by forwarding from their Boltz channels, they earn by forwarding into Boltz channels

This confuses me. If the peers of Boltz set 0 fees towards Boltz (which is the feerate the peers of boltz control) then they cannot earn by forwarding into Boltz channels.

[heyambob]

well, I'm sure you know I mean but we just use different terms.

Supposed I set fee on channelA as 100ppm and channelBoltz as 0ppm. When there's a successful forward of 1Msat from channalA to channelBoltz. I earn 100sat. However, if Boltz don't not have 1M on their side, the forward is unsuccessful, so I have an incentive to set fee to 0 on channelBoltz in order to steer the liquidity to their side or even use channelBoltz as the outgoing channel to rebalance my other channels, as I know there would be a lot of traffic coming to channelBoltz.

[ZmnSCPxj]

See FeeModderByBalance, which does that already.