search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.67k stars 2.07k forks source link

After restarting service now I'm getting "[lncli] rpc error: code = Unknown desc = verification failed: signature mismatch after caveat verification" #6382

Closed joepalermo closed 2 years ago

joepalermo commented 2 years ago

LND was working fine for several weeks. Upon restarting the service sudo systemctl restart lnd.service and unlocking the wallet I am getting [lncli] rpc error: code = Unknown desc = verification failed: signature mismatch after caveat verification after any attempt to run an lncli command, e.g. lncli walletbalance. I find it perplexing since no configuration was changed - this was triggered simply by restarting the service.

Your environment

I would appreciate some debugging advice. Thanks!

Roasbeef commented 2 years ago

Perhaps new set of macaroons were generated? I'd check the timestamps on the files, and also make sure you're using the latest set of macaroons with that command (say if you're hitting it from another machine and copied over the set of macaroons).

joepalermo commented 2 years ago

Thanks for your input @Roasbeef!

Here are the timestamps as of today (Mar 30th), note that this problem occurred yesterday (Mar 29):

Mar 16 11:31 admin.macaroon Mar 16 11:31 chainnotifier.macaroon Mar 29 15:19 channel.backup Mar 16 11:31 invoice.macaroon Mar 16 11:31 invoices.macaroon Mar 29 15:15 macaroons.db Mar 16 11:31 readonly.macaroon Mar 16 11:31 router.macaroon Mar 16 11:31 signer.macaroon Mar 30 09:13 wallet.db Mar 16 11:31 walletkit.macaroon

So it seems that this is not an entirely new set of macaroons, although channel.backup, macaroons.db and wallet.db continue to be updated.

I'm not hitting it from another machine so I don't see how I could be using anything but the latest macaroons.

guggero commented 2 years ago

Did you delete your macaroon.db file by accident yesterday? Or does lncli pick up a different macaroon for some reason? Did you move any files? You should be able to fix this by deleting all *.macaroon (not the macaroons.db) file and restarting/unlocking the node. But very weird that this suddenly happened.

joepalermo commented 2 years ago

Thanks for your input @guggero!

"Did you delete your macaroon.db file by accident yesterday? ... Did you move any files?"

It's possible but I don't think so.

"Or does lncli pick up a different macaroon for some reason?"

It seems to be fetching them from the same directory as before, namely .lnd/data/chain/bitcoin/mainnet/.

Backed up the macaroons and tried rm *.macaroon but then got this after restarting/unlocking the node:

~/.lnd/data/chain/bitcoin/mainnet$ lncli getinfo
[lncli] could not load global options: unable to read macaroon path (check the network setting!): open /home/rt/.lnd/data/chain/bitcoin/mainnet/admin.macaroon: no such file or directory

Also tried running lnd with --no-macaroons and that fixed the error however after that my channels are missing.

guggero commented 2 years ago

It sounds to me that you have different users and things aren't picked up from the correct directory. What user is lnd running under? What does that user's home directory (specifically /home/<thatuser>/.lnd) look like? And what user are you running your commands from? Channels missing also sounds like lnd is just running from an empty directory and you basically created an empty node.

joepalermo commented 2 years ago

Hmmm, there's only one user being utilized for everything.

It turns out that by using the --no-macaroons option the problem is essentially solved, although all of the channels have been automatically closed.

guggero commented 2 years ago

Is it possible that lnd is running as the root user? And therefore has a different home directory?

Running with --no-macaroons is very dangerous and should never be done with mainnet funds on the line! And all channels having been closed is very suspicious as well. I suggest to investigate further. For example, were the channels really closed and you got your funds back? Or is the channel database missing and that's why lnd doesn't know any channels?

joepalermo commented 2 years ago

Hi @guggero

I'm pretty sure it's not running as root, both the systems script and ps claim it's running as my user (and double checking, there isn't a /root/.lnd directory that's popped up).

The channel database was initially missing, so lncli listchannels came up empty. After running with —no-macaroons I tried lncli restorechanbackup --multi_file ~/.lnd/data/chain/bitcoin/mainnet/channel.backup which restored the channel list, but it had them as pending closing. Note that this was probably more than 24 hours after the node went down (because I couldn't bring it back up immediately).

The funds seem to be recovered, at least according to lncli walletbalance.

Any suggestions on how to proceed with using macaroons again? Is it best to start over?

guggero commented 2 years ago

The channel database was initially missing...

But why? That doesn't just go missing.

Using the channel backup will close the channels, that's how they work. So it's good that you have your funds back, but that still doesn't answer the question what exactly happened.

But ultimately it's probably best to start over if you've already closed the channels anyway.

joepalermo commented 2 years ago

Yeah, it did seem odd that it happened like that (both macaroons didn’t work and channels were gone)…Is it possible I’ve configured something in a fundamentally bad way that would explain both of these things?

I noted the Prevent Data Corruption section of the docs you linked and as far as I know, none of the examples should apply (shutdown was clean, disk is local in the desktop and is a reasonably high-end SSD, there is lots of free space on the volume, etc).

My plan to reset is to start by moving funds elsewhere, deleting the contents of ~/.lnd/data/chain/bitcoin/mainnet (or perhaps just all of ~/.lnd/data?), and then create a new wallet to transfer the funds back to. Is this a reasonable approach or is there a better way?

joepalermo commented 2 years ago

tagging @marctyndel who is working on this with me

marctyndel commented 2 years ago

Hello! By the way @guggero, I'm not sure whether you're leaning towards this being a possible bug or just user/configuration/local error, but if you are suspecting the former we'd be happy to try and provide anything that could help you learn more.

guggero commented 2 years ago

I don't see how this could happen by just running lnd normally, so I'm pretty sure the channel DB going missing is not a bug of lnd. So I'm okay with the issue being closed. I was just suggesting to find out what really happened for you to be sure the system isn't compromised or configured incorrectly. Otherwise this could happen again in the future and lead to possible loss of funds.

guggero commented 2 years ago

I'm closing this issue for now since it doesn't look like lnd did something unexpected here.