`listfunds` inaccurate after payment, `getroute` returns route when there is none

[dongcarl]

See below for a minimal reproducer.

If I spin up 2 nodes (l1, l2) with a 75k sats channel from l1 to l2, then make a payment of 50k sats from l1 to l2. I expect:

1. l1's listfunds to show that "our_amount_msat" to be 25k sats
   1. In actuality l1's "our_amount_msat" was the full 75k sats like the payment had never been made
2. l2's listfunds to show that "our_amount_msat" to be 50k sats
   1. This was (surprisingly) correct, even tho No. 1 wasn't
3. A call to l1's getroute for 50k sats to l2 to fail (since there are not enough sats)
   1. In actuality getroute returned the same channel, that wouldn't be able to route the 50k sats

I could be using things wrong, but this was quite surprising to me.

def test_cln_listfunds_weirdness(node_factory):
    pay_sats = 50_000

    # Open a channel between l1 and l2 for 150% of the amount we're going to
    # pay, meaning that we can only pay once
    l1, l2 = node_factory.line_graph(2, fundamount=int(pay_sats * 1.5))

    channels = l1.rpc.listchannels()
    print(f"CARL: l1's channels:\n{json.dumps(channels, indent=4)}")
    channels = l2.rpc.listchannels()
    print(f"CARL: l2's channels:\n{json.dumps(channels, indent=4)}")

    funds = l1.rpc.listfunds()
    print(f"CARL: l1's funds:\n{json.dumps(funds, indent=4)}")
    funds = l2.rpc.listfunds()
    print(f"CARL: l2's funds:\n{json.dumps(funds, indent=4)}")

    # Make the payment
    l1.pay(l2, pay_sats * 1_000)

    channels = l1.rpc.listchannels()
    print(f"CARL: l1's channels:\n{json.dumps(channels, indent=4)}")
    channels = l2.rpc.listchannels()
    print(f"CARL: l2's channels:\n{json.dumps(channels, indent=4)}")

    # l1 should have pay_sats less funds on their side of the channel
    funds = l1.rpc.listfunds()
    print(f"CARL: l1's funds:\n{json.dumps(funds, indent=4)}")
    # l2 should have pay_sats more funds on their side of the channel
    funds = l2.rpc.listfunds()
    print(f"CARL: l2's funds:\n{json.dumps(funds, indent=4)}")

    # Since we've made the payment, getroute should fail
    route = l1.rpc.getroute(l2.info['id'], pay_sats * 1_000, 0)['route']
    print(f"CARL: l1's route to l2 for pay amount {pay_sats * 1_000}msat: {json.dumps(route, indent=4)}")

[cdecker]

We had a quick call and we found a potential explanation for the inconsistency: while the HTLC is still pending we may still consider the HTLC's value to be on the sender side, while the recipient also considers it as theirs. The more correct model would be to have three buckets: ours, theirs, and undecided.

IIRC we are glossing over the last category in order to prevent situations where neither node would consider it as theirs (stuck HTLC?) thus causing a perceived loss of funds and any monitoring having frequent dips when summing up just ours + theirs.

It would be interesting to see if this intuition is true by either waiting for the HTLC removal to be committed before listfunds (which should not present this behavior) and checking that the HTLC is still present in the listpeers output when the behavior manifests.

There are a number of solutions here:

• Neither consider pending HTLCs ours not theirs (might break some tests, but likely they'd be racy anyway)
• Use the update_fulfill_htlc and update_fail_htlc as the switch-over point from sender to recipient (may flipflop if we happen to lose connection between HTLC removal and commitment since the HTLC will be re-played upon connecting)