search

comit-network / xmr-btc-swap

Bitcoin–Monero Cross-chain Atomic Swap
GNU General Public License v3.0
655 stars 83 forks source link

`asb` main Monero wallet and/or swap-specific wallets corrupted after disk ran out of space #1573

Open guardian-beach opened 7 months ago

guardian-beach commented 7 months ago

Managed to get my Monero wallet in a bad state after my asb server ran out of disk space. Or, more exactly, it looks like it isn't so much the main wallet but one of the swap-specific wallets which asb creates.

Restarting monerod and monero-wallet-rpc and asb didn't fix it, neither did using monero-wallet-cli to refresh the wallet. Here's the error:

{"timestamp":"2024-03-10T14:51:08.583530903Z","level":"WARN","fields":{"message":"Failed to make quote: JSON-RPC request failed with code -13: No wallet file: JSON-RPC request failed with code -13: No wallet file","peer":"QmNaHJET..."}}

The wallet exists and I can check the balance with monero-wallet-cli, asb also correctly sees the wallet balance when it starts up, it just gets into trouble once asb is running. First it sees the balance, then it says the wallet already exists, then it says there's no wallet:

{"timestamp":"2024-03-11T09:30:36.130948031Z","level":"DEBUG","fields":{"message":"Opening Monero wallet"}}
{"timestamp":"2024-03-11T09:30:37.650439805Z","level":"DEBUG","fields":{"message":"Opened Monero wallet","monero_wallet_name":"asb-wallet"}}
{"timestamp":"2024-03-11T09:30:37.650753641Z","level":"INFO","fields":{"message":"Monero wallet address","monero_address":"4A..."}}
{"timestamp":"2024-03-11T09:30:37.672508747Z","level":"WARN","fields":{"message":"The Monero balance is 0, make sure to deposit funds at","monero_address":"4Ae..."}}
{"timestamp":"2024-03-11T09:30:37.672590416Z","level":"DEBUG","fields":{"message":"Opening Bitcoin wallet"}}
{"timestamp":"2024-03-11T09:34:07.214667849Z","level":"INFO","fields":{"message":"Bitcoin wallet balance","bitcoin_balance":"0 BTC"}}
{"timestamp":"2024-03-11T09:34:09.171308756Z","level":"INFO","fields":{"message":"Waiting for Monero transaction finality","txid":"1a3...","target_confirmations":"10"},"span":{"id":"37..","name" :"swap"},"spans":[{"id":"377...","name":"swap"}]}
{"timestamp":"2024-03-11T09:34:10.576981242Z","level":"INFO","fields":{"message":"Received new confirmation for Monero lock tx","txid":"1a3...","seen_confirmations":"1910","needed_confirmations":"10"},"span":{"id":"377...","name":"swap"},"spans":[{"id":"377...","name":"swap"}]}
{"timestamp":"2024-03-11T09:34:10.584902104Z","level":"ERROR","fields":{"error":"JSON-RPC request failed with code -1: Wallet already exists."},"span":{"id":"377...","name":"swap"},"spans":[{"id":"377...","name":"swap"}]}
{"timestamp":"2024-03-11T09:34:10.585010672Z","level":"ERROR","fields":{"message":"Swap failed: JSON-RPC request failed with code -1: Wallet already exists.: JSON-RPC request failed with code -1: Wallet already exists.","swap_id":"377..."}}
{"timestamp":"2024-03-11T09:34:10.585365112Z","level":"DEBUG","fields":{"message":"A swap stopped without sending a transfer proof: request channel closed"}}
{"timestamp":"2024-03-11T09:34:12.186901272Z","level":"DEBUG","fields":{"message":"Found relevant Bitcoin transaction","txid":"20...","status":"unseen"}}
{"timestamp":"2024-03-11T09:35:06.58699135Z","level":"WARN","fields":{"message":"Failed to make quote: JSON-RPC request failed with code -13: No wallet file: JSON-RPC request failed with code -13: No wallet file","peer":"QmRRhBuTddbzDb4okhN2BUtpgWxTnVgKYuyuEJUSwtcri4"}}
{"timestamp":"2024-03-11T09:36:06.489813393Z","level":"WARN","fields":{"message":"Failed to make quote: JSON-RPC request failed with code -13: No wallet file: JSON-RPC request failed with code -13: No wallet file","peer":"QmZGEX4HxtsjiEri55dHCsWh2xC9cm54GhdZr91C8mojfR"}}
{"timestamp":"2024-03-11T09:37:06.110557882Z","level":"WARN","fields":{"message":"Failed to make quote: JSON-RPC request failed with code -13: No wallet file: JSON-RPC request failed with code -13: No wallet file","peer":"QmfMaBzDxrDChUiLW9iT2as2AT4DZf4YvW4nzFR5SQ4ZSZ"}}
{"timestamp":"2024-03-11T09:38:06.601929781Z","level":"WARN","fields":{"message":"Failed to make quote: JSON-RPC request failed with code -13: No wallet file: JSON-RPC request failed with code -13: No wallet file","peer":"QmTZLinpwrJqp2cZfxMdbVq8DWdqhekPyEUBr8Q1t4KFCU"}}

The obvious conclusion is that the Monero wallet got corrupted somehow, due to the out of disk error. The solution to this is just rename the old wallet dir and asb-wallet and asb-wallet.keys files, and restart asb, so that it creates a new wallet.

Unfortunately, that isn't enough: after doing this, we still get the same errors! It looks like some of the swap-specific wallets ([swap-id] and [swap-id].keys) are corrupted.

This hypothesis is corroborated by a look at the monero-wallet-rpc.log:

2024-03-11 06:28:33.527 [RPC0]  WARNING wallet.wallet2 src/wallet/wallet2.cpp:2224     Transaction extra has unsupported format: <e1c6f00bar>
2024-03-11 06:29:41.713     7fd620e5d700        WARNING wallet.wallet2 src/wallet/wallet2.cpp:2224     Transaction extra has unsupported format: <e1c6....>
2024-03-11 06:33:18.265 [RPC0]  ERROR   wallet.rpc      src/wallet/wallet_rpc_server.cpp:3633   boost::filesystem::exists(wallet_file, ignored_ec). THROW EXCEPTION: error::file_exists
2024-03-11 08:25:47.588 [RPC0]  ERROR   wallet.wallet2  src/wallet/wallet2.cpp:6105     e || !exists. THROW EXCEPTION: error::file_not_found
2024-03-11 08:30:30.723 [RPC0]  WARNING wallet.wallet2 src/wallet/wallet2.cpp:2224     Transaction extra has unsupported format: <2cd9d...>
2024-03-11 09:07:24.777 [RPC0]  ERROR   wallet.rpc      src/wallet/wallet_rpc_server.cpp:3633   boost::filesystem::exists(wallet_file, ignored_ec). THROW EXCEPTION: error::file_exists

It's making multiple requests and getting both kinds of errors, so there are probably transactions where either the wallet exists (but shouldn't) or where the wallet is missing (but should be there).

However, there is not one monero-wallet-rpc.log entry for each failed request in the asb log. There are many entries in the asb log with these errors but only a few in the monero-wallet-rpc log, so it's not a 1:1 correlation.

Permissions in the swap specific wallets are strange, they have differing permissions though I would expect they should all have the same permissions:

-rw-r----- 1 asb  asb  9611634 31. Okt 00:36 d3ee...
-rw-r----- 1 asb  asb     1662 31. Okt 00:36 d3ee...keys
-rwxrwx--- 1 asb  asb  8193045 22. Jul 2022  d5ce...
-rwxrwx--- 1 asb  asb     1538 22. Jul 2022  d5ce...keys

At the same time, the fact that it is producing "Failed to make quote" errors indicates that the main wallet may also have a problem, despite having been freshly re-created. The make_quote function at swap/src/asb/event_loop.rs:321 only seems to touch the main self.monero_wallet.

I can't possibly see how it is having trouble getting balance in make_quote() when it was previously able to get the correct XMR balance on startup... but that's what's happening.

I opened the sqlite database file in sqlitebrowser to check for corruption. Nothing obvious: the monero_addresses and peer_addresses tables are empty, but that's also true for another asb database I had saved.

I opened the old wallet file in the Monero GUI wallet... everything fine, no sign of corruption.

All the swaps in question have safely terminated (btc is cancelled) so I don't care about them. How do I fix the state of my asb instance so it can continue providing service?

Platform (please complete the following information):

guardian-beach commented 7 months ago

"Resolved" by deleting most of the files in the ASB config file folder, and essentially starting over from scratch.