rpc: got error `unable to find arbitrator` from `PendingChannels`

yyforyongyu commented 2 years ago

From the TODO, the error found in build:

=== RUN   TestLightningNetworkDaemonTemp/tranche03/06-of-99/bitcoind/data_loss_protection
    lnd.go:125: 
            Error Trace:    lnd.go:125
                                        harness_assertion.go:436
                                        wait.go:48
                                        wait.go:24
                                        asm_amd64.s:1571
            Error:          Received unexpected error:
                            rpc error: code = Unknown desc = unable to find arbitrator
            Messages:       failed to PendingChannels for Carol

Here's what happened. In rpcServer, when we do fetchPendingForceCloseChannels,

We use r.server.chanStateDB.FetchClosedChannels fetches the pending force close channel.
After the channels are fetched, we then use r.arbitratorPopulateForceCloseResp which relies on the channel arbitrator to get the report.

Now, between steps 1 and 2, the arbitrator may be deleted due to the force close channel being resolved, which ends in this error unable to find arbitrator.

I think when receiving this error from chain arbitrator, it indicates the channel has finished the closing process and we need to remove it from the response.

Once it's fixed, the retry logic can be removed here.

APBlitz commented 2 years ago

I believe I might be running into this issue:

lncli listchannels does not include the channel
lncli closedchannes does not include the channel

lncli pendingchannels returns [lncli] rpc error: code = Unknown desc = unable to find arbitrator and the logs show:

2022-11-01 22:14:59.181 [DBG] RPCS: [pendingchannels]
2022-11-01 22:14:59.245 [DBG] UTXN: NurseryReport: building nursery report for channel xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:1
2022-11-01 22:14:59.245 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: unable to find arbitrator

I was able to find the channel via chantools dumpchannels --closed with IsPending: (bool) true and see that it was included in block 761127 with CloseType: (string) (len=1) "1".

APBlitz commented 2 years ago

A re-sync seems to have fixed the problem.

TrezorHannes commented 2 years ago

A re-sync seems to have fixed the problem.

@APBlitz What type of re-sync did you do, could you be more specific? Like, rescan of the blockchain, with rescan=1, or rescan of the lnd wallet with lnd --reset-wallet-transactions?

Thanks

APBlitz commented 1 year ago

Hi @TrezorHannes apologies for not getting back to you earlier.

I believe I rescanned using --reset-wallet-transactions. It takes quite a while, but eventually all channels were back in their correct state.

raelschiffler commented 1 year ago

Hi @TrezorHannes apologies for not getting back to you earlier.

I believe I rescanned using --reset-wallet-transactions. It takes quite a while, but eventually all channels were back in their correct state.

Could you please inform the exact command you used to rescan?

guggero commented 1 year ago

Could you please inform the exact command you used to rescan?

https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md#forced-in-place-rescan

SeverinAlexB commented 1 year ago

We just experience this issue on our Blocktank regtest setup and --reset-wallet-transactions was not enough to resolve it. I had to stop lndmon and all services that call pendingchannels. Only then, --reset-wallet-transactions worked. Calling pendingchannels in the reset process would reliably not solve the issue. There must be some race condition.

dzdidi commented 1 month ago

Regtest lnd version 0.17.4-beta commit=v0.17.4-beta just faced the same issue

lightningnetwork / lnd

rpc: got error `unable to find arbitrator` from `PendingChannels` #6787