[bug]: Unknown desc = edge not found

[dannydeezy]

Background

Seems like my LND will randomly not be able to find active channels it has. I do lncli listchannels and the channel shows up as active, but then when I do lncli getchaninfo with the id, it says edge not found.

This has happened a lot, especially when the other node is a Tor node with only 1 total channel.

lncli version 0.15.1-beta commit=v0.15.1-beta

Your environment

• lncli version 0.15.1-beta commit=v0.15.1-beta
• Linux Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
• bitcoind backend Bitcoin Core version v22.0.0

Steps to reproduce

I don't know how to reproduce, it's flaky.

Example:

lncli listchannels and i see the following entry (among my 800+ other channels)

{
            "active": true,
            "remote_pubkey": "027d92cde72c29ce5a5ae11d91c7ba84245e7917dfd20f1f96ad7f3ea54836a3ad",
            "channel_point": "f7debbfd729a234f4e37235a495e2103d9c1636cabb9e960223b450bd9359954:1",
            "chan_id": "832475437786923009",
            "capacity": "25750000",
            "local_balance": "25011680",
            "remote_balance": "736777",
            "commit_fee": "883",
            "commit_weight": "1116",
            "fee_per_kw": "785",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "25011680",
            "num_updates": "373",
            "pending_htlcs": [
            ],
            "csv_delay": 2016,
            "private": false,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "257500",
            "remote_chan_reserve_sat": "257500",
            "static_remote_key": false,
            "commitment_type": "ANCHORS",
            "lifetime": "43603",
            "uptime": "43048",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "257500",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "25492500000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "257500",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "25492500000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0"
        },

but then doing lncli getchaninfo 832475437786923009 immediately alfter, i get:

[lncli] rpc error: code = Unknown desc = edge not found

logs from lnd:

[ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found

If i restart the node, it fixes it. But I can't be restarting the node all the time.

[GeorgeTsagk]

Hmm.. I was also able to reproduce that a couple of times locally on a regtest setup.. Will see if I can get that error again and share some context. Although I was experimenting with 0-conf channels so could be same bug but different origin.

[crywolf]

I had the same problem with newly created channel today. Fixed by restarting lnd.

LND version: 0.15.2-beta Bitcoin Core version v23.0.0

[guggero]

@dannydeezy and @crywolf: this was fixed by https://github.com/lightningnetwork/lnd/pull/6892. Can you please upgrade to lnd v0.15.3-beta and confirm this doesn't happen anymore?

Closing this issue in the meantime, please re-open it if v0.15.3-beta does not fix.

[Wired4ncer]

Hello. Same issue just happened to me in 0.15.5

[Wired4ncer]

Closed a channel with peer and reopened new one, that one got confirmed, but strangely fees and other channel attributes were not set at all and immediately it went inactive. Any attempt to change the parameters fail with similar

_"reason": "UPDATE_FAILURE_UNKNOWN", "updateerror": "could not update policies"

Logs are showing channel trying to re-establish, but failing due to edge not found I'm unable to coop-close or to update the fee policy. Any advice would be great. Thank you

[sputn1ck]

Hey @Wired4ncer you could maybe try an update to 0.16 rc1.

For some context, smiliar to OP, the mentioned channel is shown as active in listchannels, but getchaninfo fails?

[Wired4ncer]

Yes that exactly was happening. Channel didn't receive any params when opened (fees, htlc size, ctv etc.) and was failing in logs due to not having it when trying to reestablish, but I was not able to update it. Ended up FC'ing it and reopening. Second time it all went fine.

I don't have it open anymore, so updating to 0.16 at the moment won't give us any results.

[gradientskier]

This just happened to me too. In particular, the opening transaction was paying a very low fee (1sat/vb), so the channel has been in state pending for a very long time, for about three weeks. Then I managed to CPFP fee bump the transactions (and I had to broadcast again the opening transaction via external block explorer). The opening transaction eventually got confirmed, but the channel is now in "inactive" state in listchannels and getchaninfo returns error "edge not found". Is FC'ing and reopening the only solution?

[guggero]

The remote peer will forget the channel after 2 weeks (2016 blocks). So yes, unfortunately force closing is the only option. We'll try to make this more clear with this: https://github.com/lightningnetwork/lnd/issues/7434

[bota87]

Just had the same issue on a batch open, all the channel have the same problem. Transaction was broadcasted and confirmed on the same day. There is something that I can do other closing? Restart didn't fix the issue. Running LND 0.16

[ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxx:2): edge not found
[ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxx:1): edge not found
[ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxx:3): edge not found

[bota87]

I can add other info about this: I chatted with one of the peers, on his side all looks fine, he tried to change the fees and worked. The channel isn't announced but isn't marked as private. On my side updatechanpolicy, updatechanstatus and payinvoice fails

$ lncli updatechanpolicy 0 1000 144 xxxxxxxxx:3
{
   "failed_updates":[
      {
         "outpoint":"xxxxxxxx:3",
         "reason":"UPDATE_FAILURE_UNKNOWN",
         "update_error":"could not update policies"
      }
   ]
}

$ lncli updatechanstatus --chan_point xxxxxxxx:3 --action enable
[lncli] rpc error: code = Unknown desc = edge not found

$ lncli payinvoice --outgoing_chan_id xxxx 
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   xxxxx
Payment status: FAILED, reason: FAILURE_REASON_INSUFFICIENT_BALANCE
[lncli] FAILED

on the logs I found this error

[ERR] FNDG: Unable to advance state(xxxxxxxxx:3): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found

Logs

[INF] BRAR: Breach arbiter starting [INF] NTFN: New block epoch subscription [INF] FNDG: Funding manager starting [INF] BRAR: Starting contract observer, watching for breaches. [INF] HSWC: HTLC Switch starting [INF] NTFN: New confirmation subscription: conf_id=2, txid=xxxxxxxxx, num_confs=3 height_hint=yyyyyy [INF] NTFN: New confirmation subscription: conf_id=3, txid=xxxxxxxxx, num_confs=2 height_hint=yyyyyy [INF] NTFN: New confirmation subscription: conf_id=4, txid=xxxxxxxxx, num_confs=3 height_hint=yyyyyy [INF] NTFN: New block epoch subscription [INF] FNDG: Waiting for funding tx (xxxxxxxxx) to reach 2 confirmations [INF] FNDG: Waiting for funding tx (xxxxxxxxx) to reach 3 confirmations [INF] FNDG: Waiting for funding tx (xxxxxxxxx) to reach 3 confirmations [INF] NTFN: New block epoch subscription [INF] CNCT: ChainArbitrator starting [INF] CNCT: Creating ChannelArbitrators for 19 active channels

[INF] NTFN: New spend subscription: spend_id=16, outpoint=xxxxxxxxx:1, script=0 zzzzzzz, height_hint=yyyyyy [INF] NTFN: Dispatching historical spend rescan for outpoint=xxxxxxxxx:1, script=0 zzzzzzz, start=yyyyyy, end=yyyyyy [INF] NTFN: New spend subscription: spend_id=18, outpoint=xxxxxxxxx:3, script=0 zzzzzzz, height_hint=yyyyyy [INF] NTFN: Dispatching historical spend rescan for outpoint=xxxxxxxxx:3, script=0 zzzzzzz, start=yyyyyy, end=yyyyyy [INF] NTFN: New spend subscription: spend_id=4, outpoint=xxxxxxxxx:2, script=0 zzzzzzz, height_hint=yyyyyy [INF] NTFN: Dispatching historical spend rescan for outpoint=xxxxxxxxx:2, script=0 zzzzzzz, start=yyyyyy, end=yyyyyy

[INF] CNCT: Close observer for ChannelPoint(xxxxxxxxx:1) active [INF] CNCT: Close observer for ChannelPoint(xxxxxxxxx:3) active [INF] CNCT: Close observer for ChannelPoint(xxxxxxxxx:2) active

[INF] CNCT: ChannelArbitrator(xxxxxxxxx:1): starting state=StateDefault, trigger=chainTrigger, triggerHeight=yyyyyy [INF] CNCT: ChannelArbitrator(xxxxxxxxx:3): starting state=StateDefault, trigger=chainTrigger, triggerHeight=yyyyyy [INF] CNCT: ChannelArbitrator(xxxxxxxxx:2): starting state=StateDefault, trigger=chainTrigger, triggerHeight=yyyyyy

[WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:3) w/ newer version [WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:1) w/ newer version [WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:2) w/ newer version

[INF] HSWC: Trimming open circuits for chan_id=x:y:z, start_htlc_id=695 [INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:3) [WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:3, using default values [INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:2) [WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:2, using default values [INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:1) [WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:1, using default values [INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:3) [WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:3, using default values

[INF] LNWL: Marking unconfirmed transaction xxxxxxxxx mined in block block111 [INF] CRTR: Pruning channel graph using block block111 (height=block111) [INF] CRTR: Block block111 (height=block111) closed 0 channels [INF] NTFN: New block: height=block111, sha=block111 [INF] UTXN: Attempting to graduate height=block111: num_kids=0, num_babies=0 [INF] HSWC: Garbage collected 5 shared secret hashes at height=block111

[INF] CRTR: Block block222 (height=block222) closed 2 channels [INF] NTFN: New block: height=block222, sha=block222 [INF] NTFN: Dispatching 2 confirmation notification for txid=xxxxxxxxx [INF] FNDG: ChannelPoint(xxxxxxxxx:3) is now active: ChannelID(chanid1) [INF] UTXN: Attempting to graduate height=block222: num_kids=0, num_babies=0 [INF] HSWC: Garbage collected 4 shared secret hashes at height=block222 [INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19 [INF] FNDG: Peer(ppppp) is online, sending FundingLocked for ChannelID(chanid1) [WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:3) w/ newer version [INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxxx:3): edge not found [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=18 ineligible=0) sessions(acquired=0 exhausted=106) [ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found [INF] PEER: Peer(ppppp): New channel active ChannelPoint(xxxxxxxxx:3) with peer [INF] HSWC: ChannelLink(xxxxxxxxx:3): starting [ERR] FNDG: Unable to advance state(xxxxxxxxx:3): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found [INF] HSWC: Trimming open circuits for chan_id=block111:y:3, start_htlc_id=0 [INF] HSWC: Adding live link chan_id=chanid1, short_chan_id=block111:y:3 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(xxxxxxxxx:3) [INF] HSWC: ChannelLink(xxxxxxxxx:3): HTLC manager started, bandwidth=4564566000 mSAT [INF] HSWC: ChannelLink(xxxxxxxxx:3): attempting to re-synchronize [INF] HSWC: ChannelLink(xxxxxxxxx:3): resending FundingLocked message to peer [INF] HSWC: ChannelLink(xxxxxxxxx:3): received re-establishment message from remote side [INF] FNDG: Received duplicate fundingLocked for ChannelID(chanid1), ignoring.

[INF] CRTR: Block block333 (height=block333) closed 0 channels [INF] NTFN: New block: height=block333, sha=block333 [INF] UTXN: Attempting to graduate height=block333: num_kids=0, num_babies=0 [INF] NTFN: Dispatching 3 confirmation notification for txid=xxxxxxxxx [INF] NTFN: Dispatching 3 confirmation notification for txid=xxxxxxxxx [INF] FNDG: ChannelPoint(xxxxxxxxx:1) is now active: ChannelID(chanid2) [INF] FNDG: ChannelPoint(xxxxxxxxx:2) is now active: ChannelID(chanid3) [INF] HSWC: Garbage collected 3 shared secret hashes at height=block333 [INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19 [INF] RPCS: channel: xxxxxxxxx:1 not found by channel event store [INF] RPCS: channel: xxxxxxxxx:2 not found by channel event store [INF] FNDG: Peer(pppppppppp) is online, sending FundingLocked for ChannelID(chanid2) [WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:3) w/ newer version [INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [INF] FNDG: Peer(pppppppppp) is online, sending FundingLocked for ChannelID(chanid3) [INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19 [ERR] FNDG: Unable to advance state(xxxxxxxxx:2): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found [INF] PEER: Peer(pppppppppp): New channel active ChannelPoint(xxxxxxxxx:2) with peer [INF] PEER: Peer(pppppppppp): New channel active ChannelPoint(xxxxxxxxx:1) with peer [INF] HSWC: ChannelLink(xxxxxxxxx:2): starting [INF] HSWC: Trimming open circuits for chan_id=block111:y:2, start_htlc_id=0 [INF] HSWC: Adding live link chan_id=chanid3, short_chan_id=block111:y:2 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(xxxxxxxxx:2) [INF] HSWC: ChannelLink(xxxxxxxxx:2): HTLC manager started, bandwidth=4564566000 mSAT [INF] HSWC: ChannelLink(xxxxxxxxx:2): attempting to re-synchronize [INF] HSWC: ChannelLink(xxxxxxxxx:1): starting [INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19 [INF] HSWC: Trimming open circuits for chan_id=block111:y:1, start_htlc_id=0 [INF] HSWC: Adding live link chan_id=chanid2, short_chan_id=block111:y:1 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(xxxxxxxxx:1) [INF] HSWC: ChannelLink(xxxxxxxxx:1): HTLC manager started, bandwidth=4564566000 mSAT [INF] HSWC: ChannelLink(xxxxxxxxx:1): attempting to re-synchronize [INF] HSWC: ChannelLink(xxxxxxxxx:1): resending FundingLocked message to peer [INF] HSWC: ChannelLink(xxxxxxxxx:1): received re-establishment message from remote side [INF] HSWC: ChannelLink(xxxxxxxxx:2): resending FundingLocked message to peer [INF] HSWC: ChannelLink(xxxxxxxxx:2): received re-establishment message from remote side [INF] FNDG: Received duplicate fundingLocked for ChannelID(chanid3), ignoring. [INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup [INF] FNDG: Received duplicate fundingLocked for ChannelID(chanid2), ignoring. [ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found [ERR] FNDG: Unable to advance state(xxxxxxxxx:1): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=41 ineligible=0) sessions(acquired=0 exhausted=910)

[ellemouton]

@bota87 - So I think we only add the channel to our local graph after receiving the ChannelReady message from our peer - if this all happened for you on the same day (ie, your peer has not timed out the channels) then my hunch says that either they are not sending the ChannelReady/FundingLocked or your node is not handling the message properly. Do you see anything in the logs re FundingLocked? might be good idea to ask your peer the same question to see if the issue is with sending the message

[bota87]

Hi @ellemouton, all it's happened in the same day (batch channels open request and transaction confirmation). On the logs that I've posted there are many FundingLocked messages, I've checked again for ChannelReady but didn't find. I've asked to the peer his logs, waiting for the reply.

[bota87]

Here the logs from the peer, he said that didn't find any ChannelReady message

[ellemouton]

further up in your logs, do you at all see a log starting with Unable to store the forwarding policy?

(also, don't worry about ChannelReady, in 0.16 it will be FundingLocked)

[ellemouton]

another question: when exactly did you upgrade to 0.16? did you initiate the funding and update to 0.16 afterwards or was this all done after upgrading to 0.16?

[bota87]

further up in your logs, do you at all see a log starting with Unable to store the forwarding policy?

no the only messages talking about policy is this one [WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:3, using default values

another question: when exactly did you upgrade to 0.16? did you initiate the funding and update to 0.16 afterwards or was this all done after upgrading to 0.16?

I'm not sure, I migrated from 0.15.4 to 0.16 on the same day but didn't remember if I start the batch channel open before or after the update

[ellemouton]

ok cool - I managed to reproduce your scenario. Thanks for reporting! Working on a fix asap

[bota87]

That's great! The fix will let me use this channels or I have to close and reopen? I have to wait for the fix before opening other channels?

[ellemouton]

I think the fix will let you use the channels as long as the peer doesnt force close before you apply the fix.

It should be safe to open other channels in the mean time - the error resulted from the channel being opened before upgrading to 0.16, then upgrading to 0.16 before channel confirmation.

[bota87]

Updated to 0.16.1.rc1, the channels works! 🥳 Thank's @ellemouton

[saubyk]

Closing this issue. Feel free to reopen if the reported problem has not been addressed.

[dskvr]

Still happens on most recent release.

2023-11-02 14:13:24.024 [ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found

[ralyodio]

i get this opening fee modal in RTL.

500 - {"code":2,"message":"edge Not Found","details":[]}