lightninglabs / lightning-terminal

Lightning Terminal: Your Home for Lightning Liquidity
MIT License
487 stars 82 forks source link

Rebalances no longer appear in Spent total #609

Closed dekdynamics closed 10 months ago

dekdynamics commented 11 months ago

In the Lightning Terminal Web UI, sats spent on rebalancing are no longer included in the overall stats on Home landing page, or in the P&L file.

image

I know I have completed rebalances today, and earlier this week, and I can see that lnd knows about them with bos accounting payments

This is on the latest lnd 0.16.4 and litd 0.10.4 (--lnd-mode=remote).

Regards, The Wall

jamaljsr commented 11 months ago

I just tested this on regtest and was able to confirm that circular rebalance payments are reflected in the stats and CSV export.

image

The way Terminal detects a circular rebalance is by matching the payment hash and amount of invoices/payments. If it finds a match then it considers the payment a rebalance.

For example, the payment hash in the screenshot above is 8d4cdc296f3302c9eeb52ccdc31add903bbd54a8679c8ba45fa2d755da6b26f6.

Here's my output for `lncli listinvoices`: ``` { "invoices": [ { "memo": "", "r_preimage": "30b626b0976493179f718cc47276e67cf53d5d82a700f22ade35c2e7decf9cc3", "r_hash": "8d4cdc296f3302c9eeb52ccdc31add903bbd54a8679c8ba45fa2d755da6b26f6", "value": "10000", "value_msat": "10000000", "settled": true, "creation_date": "1690813668", "settle_date": "1690813669", "payment_request": "lnbcrt100u1pjv038ypp534xdc2t0xvpvnm449nxuxxkajqam649gv7wghfzl5tt4tkntymmqdqqcqzzsxqyz5vqsp5yspsjnrzl390wjk295qzdfzhl5ehnz8jxut6fv3nurrv07crmgvs9qyyssqr3hnx2a6rcaqqgj4xztugvw6w0ztecwnnvqj3ffezdfmxlamelkytxwsplcjy7hnll8qmlc3v8m8ea6zj3zw7lh34r2d5jlqptuqwxcphkfreh", "description_hash": null, "expiry": "86400", "fallback_addr": "", "cltv_expiry": "80", "route_hints": [ ], "private": false, "add_index": "1", "settle_index": "1", "amt_paid": "10000000", "amt_paid_sat": "10000", "amt_paid_msat": "10000000", "state": "SETTLED", "htlcs": [ { "chan_id": "286972534915072", "htlc_index": "0", "amt_msat": "10000000", "accept_height": 302, "accept_time": "1690813669", "resolve_time": "1690813669", "expiry_height": 385, "state": "SETTLED", "custom_records": { }, "mpp_total_amt_msat": "10000000", "amp": null } ], "features": { "9": { "name": "tlv-onion", "is_required": false, "is_known": true }, "14": { "name": "payment-addr", "is_required": true, "is_known": true }, "17": { "name": "multi-path-payments", "is_required": false, "is_known": true } }, "is_keysend": false, "payment_addr": "2403094c62fc4af74aca2d0026a457fd337988f23717a4b233e0c6c7fb03da19", "is_amp": false, "amp_invoice_state": { } } ], "last_index_offset": "1", "first_index_offset": "1" } ```
Here's the output for `lncli listpayments`: ``` { "payments": [ { "payment_hash": "8d4cdc296f3302c9eeb52ccdc31add903bbd54a8679c8ba45fa2d755da6b26f6", "value": "10000", "creation_date": "1690813668", "fee": "3", "payment_preimage": "30b626b0976493179f718cc47276e67cf53d5d82a700f22ade35c2e7decf9cc3", "value_sat": "10000", "value_msat": "10000000", "payment_request": "lnbcrt100u1pjv038ypp534xdc2t0xvpvnm449nxuxxkajqam649gv7wghfzl5tt4tkntymmqdqqcqzzsxqyz5vqsp5yspsjnrzl390wjk295qzdfzhl5ehnz8jxut6fv3nurrv07crmgvs9qyyssqr3hnx2a6rcaqqgj4xztugvw6w0ztecwnnvqj3ffezdfmxlamelkytxwsplcjy7hnll8qmlc3v8m8ea6zj3zw7lh34r2d5jlqptuqwxcphkfreh", "status": "SUCCEEDED", "fee_sat": "3", "fee_msat": "3030", "creation_time_ns": "1690813668905749420", "htlcs": [ { "attempt_id": "1", "status": "SUCCEEDED", "route": { "total_time_lock": 625, "total_fees": "3", "total_amt": "10003", "hops": [ { "chan_id": "313360813981696", "chan_capacity": "2000000", "amt_to_forward": "10002", "fee": "1", "expiry": 545, "amt_to_forward_msat": "10002020", "fee_msat": "1010", "pub_key": "03dd7fb67f7706dbe340f74e77a89fe08cf0a6cb12d594439d1adf56507c6ea18a", "tlv_payload": true, "mpp_record": null, "amp_record": null, "custom_records": { }, "metadata": null }, { "chan_id": "260584255848448", "chan_capacity": "16000000", "amt_to_forward": "10001", "fee": "1", "expiry": 465, "amt_to_forward_msat": "10001010", "fee_msat": "1010", "pub_key": "0319138cf382c8440ee967a1a63bc045d41137813c85f15c43bf1d9c1cb1f1d6ef", "tlv_payload": true, "mpp_record": null, "amp_record": null, "custom_records": { }, "metadata": null }, { "chan_id": "227598907015168", "chan_capacity": "16000000", "amt_to_forward": "10000", "fee": "1", "expiry": 385, "amt_to_forward_msat": "10000000", "fee_msat": "1010", "pub_key": "0203c741d1009fd324d91ecc480dcf98f1379a3faa7e0331b6081220c856da592a", "tlv_payload": true, "mpp_record": null, "amp_record": null, "custom_records": { }, "metadata": null }, { "chan_id": "286972534915072", "chan_capacity": "16000000", "amt_to_forward": "10000", "fee": "0", "expiry": 385, "amt_to_forward_msat": "10000000", "fee_msat": "0", "pub_key": "03e4dd278502245b2121535a5613854b9474069480d8c7e8248851ca4fa04608fa", "tlv_payload": true, "mpp_record": { "payment_addr": "2403094c62fc4af74aca2d0026a457fd337988f23717a4b233e0c6c7fb03da19", "total_amt_msat": "10000000" }, "amp_record": null, "custom_records": { }, "metadata": null } ], "total_fees_msat": "3030", "total_amt_msat": "10003030" }, "attempt_time_ns": "1690813668921111920", "resolve_time_ns": "1690813669317513004", "failure": null, "preimage": "30b626b0976493179f718cc47276e67cf53d5d82a700f22ade35c2e7decf9cc3" } ], "payment_index": "1", "failure_reason": "FAILURE_REASON_NONE" } ], "first_index_offset": "1", "last_index_offset": "1", "total_num_payments": "0" } ```

You can see that the amounts match and the invoice's r_hash is the same as the payment's payment_hash. So this payment gets classified as a rebalance instead of a normal off-chain payment.

Can you check your invoice and payment for the rebalance that you performed to see if these also match on your node?

dekdynamics commented 11 months ago

To be clear, this was working fine in previous versions. I dont know exactly when it stopped.

Here is my latest rebalance:

lncli listpayments | jq -r '.payments[] | select (.payment_hash == "7b49316e6bef3328f025bbdf1cc93f926bee8879226bec137105fa6f951e50b5")'
{
  "payment_hash": "7b49316e6bef3328f025bbdf1cc93f926bee8879226bec137105fa6f951e50b5",
  "value": "881011",
  "creation_date": "1690804036",
  "fee": "667",
  "payment_preimage": "86a277734967c88f81793a8f9820437bf9f9ed09ed0bdc1e305c2d9fd625bce0",
  "value_sat": "881011",
  "value_msat": "881011000",
  "payment_request": "",
  "status": "SUCCEEDED",
  "fee_sat": "667",
  "fee_msat": "667166",
  "creation_time_ns": "1690804036421279913",
  "htlcs": [
    {
      "attempt_id": "4427535",
      "status": "SUCCEEDED",
      "route": {
        "total_time_lock": 801469,
        "total_fees": "667",
        "total_amt": "881678",
        "hops": [
          {
            "chan_id": "878069986105491456",
            "chan_capacity": "2000000",
            "amt_to_forward": "881014",
            "fee": "663",
            "expiry": 801349,
            "amt_to_forward_msat": "881014762",
            "fee_msat": "663404",
            "pub_key": "023631624e30ef7bcb2887e600da8e59608a093718bc40d35b7a57145a0f3db9af",
            "tlv_payload": true,
            "mpp_record": null,
            "amp_record": null,
            "custom_records": {},
            "metadata": null
          },
          {
            "chan_id": "862381054617255937",
            "chan_capacity": "6000000",
            "amt_to_forward": "881012",
            "fee": "1",
            "expiry": 801205,
            "amt_to_forward_msat": "881012881",
            "fee_msat": "1881",
            "pub_key": "033d8656219478701227199cbd6f670335c8d408a92ae88b962c49d4dc0e83e025",
            "tlv_payload": true,
            "mpp_record": null,
            "amp_record": null,
            "custom_records": {},
            "metadata": null
          },
          {
            "chan_id": "834789909780692992",
            "chan_capacity": "10000000",
            "amt_to_forward": "881011",
            "fee": "1",
            "expiry": 801165,
            "amt_to_forward_msat": "881011000",
            "fee_msat": "1881",
            "pub_key": "03cde60a6323f7122d5178255766e38114b4722ede08f7c9e0c5df9b912cc201d6",
            "tlv_payload": true,
            "mpp_record": null,
            "amp_record": null,
            "custom_records": {},
            "metadata": null
          },
          {
            "chan_id": "834343508144095233",
            "chan_capacity": "20000000",
            "amt_to_forward": "881011",
            "fee": "0",
            "expiry": 801165,
            "amt_to_forward_msat": "881011000",
            "fee_msat": "0",
            "pub_key": "0203e5b16ebe87b089f22e18752f1f7a66a1bdf77879df8d1c9e8d912dbfb9beb4",
            "tlv_payload": true,
            "mpp_record": {
              "payment_addr": "405270897e3307ad2921c383252faaeb53f4deba252ac12194582d5e601ee713",
              "total_amt_msat": "881011000"
            },
            "amp_record": null,
            "custom_records": {},
            "metadata": null
          }
        ],
        "total_fees_msat": "667166",
        "total_amt_msat": "881678166"
      },
      "attempt_time_ns": "1690804036435714430",
      "resolve_time_ns": "1690804038904131027",
      "failure": null,
      "preimage": "86a277734967c88f81793a8f9820437bf9f9ed09ed0bdc1e305c2d9fd625bce0"
    }
  ],
  "payment_index": "4408536",
  "failure_reason": "FAILURE_REASON_NONE"
}

lncli listinvoices | jq -r '.[][] | select(.r_hash == "7b49316e6bef3328f025bbdf1cc93f926bee8879226bec137105fa6f951e50b5")'
{
  "memo": "Rebalance attempt",
  "r_preimage": "86a277734967c88f81793a8f9820437bf9f9ed09ed0bdc1e305c2d9fd625bce0",
  "r_hash": "7b49316e6bef3328f025bbdf1cc93f926bee8879226bec137105fa6f951e50b5",
  "value": "881011",
  "value_msat": "881011000",
  "settled": true,
  "creation_date": "1690803221",
  "settle_date": "1690804038",
  "payment_request": "lnbc8810110n1pjv08q4pp50dynzmntauej3up9h003ejfljf47azreyf47cym3qhaxl9g72z6sdqu2fjkyctvv9hxxefqv968getdwp6qcqzryxqyz5vqsp5gpf8pzt7xvr662fpcwpj2ta2adflfh46y54vzgv5tqk4ucq7uufs9qyyssq0v06ax78m3q8f0dfnu8p3d9nguudcawn50jlvhtfw2yfavhe8hmq5xzfdshr5deagxn2y5ekespxdd2geue3ptyfmn6klr4urn9nrwcqsep3yd",
  "description_hash": null,
  "expiry": "86400",
  "fallback_addr": "",
  "cltv_expiry": "100",
  "route_hints": [],
  "private": false,
  "add_index": "67087",
  "settle_index": "10305",
  "amt_paid": "881011000",
  "amt_paid_sat": "881011",
  "amt_paid_msat": "881011000",
  "state": "SETTLED",
  "htlcs": [
    {
      "chan_id": "834343508144095233",
      "htlc_index": "1362",
      "amt_msat": "881011000",
      "accept_height": 801065,
      "accept_time": "1690804038",
      "resolve_time": "1690804038",
      "expiry_height": 801165,
      "state": "SETTLED",
      "custom_records": {},
      "mpp_total_amt_msat": "881011000",
      "amp": null
    }
  ],
  "features": {
    "9": {
      "name": "tlv-onion",
      "is_required": false,
      "is_known": true
    },
    "14": {
      "name": "payment-addr",
      "is_required": true,
      "is_known": true
    },
    "17": {
      "name": "multi-path-payments",
      "is_required": false,
      "is_known": true
    }
  },
  "is_keysend": false,
  "payment_addr": "405270897e3307ad2921c383252faaeb53f4deba252ac12194582d5e601ee713",
  "is_amp": false,
  "amp_invoice_state": {}
}
image

My 1D spent:

image

Which is accounted for entirely by this channel opening:

"7/31/2023, 11:15:35 AM","-1890","Open Channel 880803372020203521 with 1sats.com 🤩 low fees","txid=4f9e42380d50e6f32ad4c9e77a78013fd959badb95adb36020b916f3f24323bd","880803372020203521"

jamaljsr commented 11 months ago

Ah ok, thanks for the details. It does look like this payment should be classified as a rebalance. Does it show up in the CSV export if you select the 1W or 1M time frame? Maybe there's an issue comparing the timestamps.

dekdynamics commented 11 months ago

It doesn't appear in any of the epochs including ALL.

jamaljsr commented 10 months ago

I've gotten a chance to dig deeper into this issue and think I have found the source of the problem. Terminal was only fetching the first 100 invoices from the backend, no matter how many you had created. This is why the it was unable to match the payment hash.

We have a fix implemented and under review. We should have it deployed in the next week or so.

jamaljsr commented 10 months ago

We have deployed a fix for this. Can you check now to see if all your reblances are reflected in the stats