search

JoinMarket-Org / joinmarket-clientserver

Bitcoin CoinJoin implementation with incentive structure to convince people to take part
GNU General Public License v3.0
701 stars 174 forks source link

Wallet Recovery Problem #372

Closed HamishMacEwan closed 4 years ago

HamishMacEwan commented 5 years ago

First let me say that I have access to the joinmarket internal wallet funds by restoring the 12 word phrase on a Trezor 1, so that is a fallback that works fine.

I lost my working Bitcoin Core & Joinmarket client/server 0.5.4 installation to a malware attack, via Exim.

Recovered the Bitcoin Core wallet and joinmarket config including wallets. Then reinstalled and resynced Bitcoin Core on another PC.

I've done a -resync and -txindex=1 during which BC reported:

2019-07-13T07:36:20Z [default wallet] Keys: 0 plaintext, 1347 encrypted, 60955 w/ metadata, 1347 total. Unknown wallet records: 12

Which I take to indicate the joinmarket records are there.

I then restored the jm wallet with the phrase and the old password.

Presently python wallet-tool.py wallet.jmdat summary is reporting zero balance on all mix levels, which is correct for m0 & m1, but m2-m4 should have positive balances.

Any hints? I tried increasing the gap, though I can't really see that being a problem, there wouldn't have been many unused addresses, and it didn't resolve the problem.

Thanks in advance for any suggestions.

undeath commented 5 years ago

have you done a -rescan after jm importing the addresses to bitcoind?

HamishMacEwan commented 5 years ago

Where I wrote -resync, I meant -rescan, and I'm a bit uncertain about the exact sequence of what I've done, so I will do a -rescan now.

Is there an explicit command for "jm importing the addresses to bitcoind?" I'm assuming not and starting the -rescan.

kristapsk commented 5 years ago

It's parameter you can give to bitcoind/bitcoin-qt at start-up. But you can do bitcoin-cli rescanblockchain instead. It allows you to specify starting blockheight, which allows you to do only partial downloading and rescan of the Bitcoin blockchain, if you know approximate date when wallet was created. Btw, I have written a simple bash script that will return you the last blockheight before certain date/time, might be helpful.

HamishMacEwan commented 5 years ago

Thanks, I'm finally fully resync'd and throughout that process and my -rescan activities there have been numerous AddToWallet notifications which made me think, along with the numbers provided in the initial post, that the BC wallet has all the JM transactions in it, just JM isn't looking/asking for them?

undeath commented 5 years ago

This is a pretty weird problem. I'm not sure what is happening.

Is the wallet of the correct type (segwit/non-segwit)? Are the addresses shown by display/displayall the same as the ones Trezor shows? Is your history empty?

HamishMacEwan commented 5 years ago

After -rescan

Still zero balance on all mix depths & total.

AdamISZ commented 5 years ago

Unlikely here, but worth mentioning: Are you using multiwallet in Bitcoin Core? If so make sure you have the right wallet.dat file specified in the joinmarket.cfg

More general point: please paste your joinmarket.cfg with any sensitive info scrubbed.

Trivial point, but try wallet-tool without 'summary' just in case that is bugged.

Have you tried wallet-tool with a very large gap limit?

You may need to rescan again after doing that but you can use @kristapsk 's methods above to speed that up.

Generally speaking in the worst-case scenario: completely new Bitcoin Core and recovering wallet with no JM wallet file (so no pre-existing information on indexes), with a very highly used JM wallet with lots of txs and lots of gaps, the situation can be very difficult and tedious, sometimes requiring multiple rescans, in the worst case. I started looking into improving this in #359 but I've been busy and not able to complete it yet.

Can't think of anything else right now.

HamishMacEwan commented 5 years ago

I have created a new wallet, and it successfully received and reported the sum I've sent to it using the same JM & BC setup that won't work with the restored from phrase/password and restored from backup wallet.jmdat .

Would the age of the wallet, and the fact it was migrated from the old text wallet.dat format have any impact?

AdamISZ commented 5 years ago

Oh hang on, what do you mean by "old text wallet.dat" .. do you mean old joinmarket wallet format (which by default was wallet.json, but you can name it anything)? If so it might be the cause of the problem. There was a script specifically for converting from the old json plaintext format to the new *.jmdat (encrypted). See https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/docs/wallet-upgrade.md

But having said that, I'm not at all sure I interpreted correctly what you said above (the first sentence). I don't really understand that sentence.

HamishMacEwan commented 5 years ago

My apologies, it was rather condensed.

As noted earlier the Bitcoin Core wallet was restored from a weekly backup and was described as [default wallet] Keys: 0 plaintext, 1347 encrypted, 60955 w/ metadata, 1347 total. Unknown wallet records: 12

I'm testing three joinmarket wallets:

To confirm the Bitcoin Core and JoinMarket set up was functioning, I sent bitcoin to wallet-thinkpad.jmdat and wallet-tool.py successfully reported the amount. From this I conclude that all parts of the system are working correctly.

The setup does not report correctly for either wallet.jmdat nor wallet-recover.jmdat

Since the variable that leads to failure is the original wallet, I wondered about the original wallet's history. There was the original P2PKH wallet.json which I stopped using when I migrated to segwit when that became available (created a new wallet and swept the mix levels to the new addresses). Then there was the move from .json to .jmdat.

I hope that clarifies the situation, essentially on a rebuilt, resynced & rescanned JoinMarket/Bitcoin Core system, on a different PC, with restored wallets from both JoinMarket and Bitcoin, the contents of the JoinMarket wallet are not correctly reported.

HamishMacEwan commented 5 years ago

Sorry Adam I missed your earlier post.

Are you using multiwallet in Bitcoin Core? If so make sure you have the right wallet.dat file specified in the joinmarket.cfg

No multiwallet.

More general point: please paste your joinmarket.cfg with any sensitive info scrubbed.

[BLOCKCHAIN]
blockchain_source = bitcoin-rpc
network = mainnet
rpc_host = localhost
rpc_port = 8332
#rpc_user = bitcoin
#rpc_password = password
rpc_cookie_file = /home/user/Bitcoin/.cookie

[MESSAGING]
host = irc.cyberguerrilla.org, agora.anarplex.net
channel = joinmarket-pit, joinmarket-pit
port = 6697, 14716
usessl = true, true
socks5 = true, true
socks5_host = localhost, localhost
socks5_port = 9050, 9050

[LOGGING]
console_log_level = INFO

[TIMEOUT]
maker_timeout_sec = 45
unconfirm_timeout_sec = 90
confirm_timeout_hours = 6

[POLICY]
merge_algorithm = default
minimum_makers = 2
tx_fees = 100
absurd_fee_per_kb = 200000
tx_broadcast = self
taker_utxo_retries = 3
taker_utxo_age = 5
taker_utxo_amtpercent = 20
accept_commitment_broadcasts = 1
commit_file_location = cmttools/commitments.json

Trivial point, but try wallet-tool without 'summary' just in case that is bugged.

Same result. Zero balance in all mix depths. The derivation paths indicate use (I guess):

  1. m/49'/0'/0'/0/260
  2. m/49'/0'/1'/0/148
  3. m/49'/0'/2'/0/185
  4. m/49'/0'/3'/0/120
  5. m/49'/0'/4'/0/124

Have you tried wallet-tool with a very large gap limit?

Not recently, earlier I tried 60 and 600.

You may need to rescan again after doing that but you can use @kristapsk 's methods above to speed that up.

Generally speaking in the worst-case scenario: completely new Bitcoin Core and recovering wallet with no JM wallet file (so no pre-existing information on indexes), with a very highly used JM wallet with lots of txs and lots of gaps, the situation can be very difficult and tedious, sometimes requiring multiple rescans, in the worst case.

I love a challenge. I do wish though I had more of a clue of what is, or isn't happening.

Can't think of anything else right now.

This should be enough to be going on with...

AdamISZ commented 5 years ago

Sorry I didn't respond today/yesterday, had some stuff going on. Will try to add something more shortly.

undeath commented 5 years ago

Using a very large gap limit and then using -rescan again should solve this issue. Fundamentally it's an issue with how bitcoind works.

When importing an already used wallet joinmarket will import the addresses to bitcoind but bitcoind (and thus joinmarket) will consider all addresses unused until a rescan is performed. Once the rescan is completed bitcoind again only knows about the addresses used in those few imported addresses, which are likely to be empty already in a heavily used wallet. On the next start joinmarket will recognize the used addresses and import another small batch of addresses, which again require a rescan to be recognized. This procedure needs to be performed until all used addresses have been imported to bitcoind (and subsequent rescans).

Using a large gap limit will allow to complete this with far fewer rescans.

That whole mechanism is pretty crude and annoying. A solution might be for joinmarket to have some detection for such a situation (unimported addresses) and retrieve the actual transactions for such addresses and comparing destination addresses with lots of pre-generated addresses in the wallet. This way it could iteratively figure out the indices without requiring dozens of rescans. (this might not be possible at all; depends on what information the bitcoind rpc can offer)

kristapsk commented 5 years ago

It's that way when wallet is recovered from the seed, not when you copy old JM wallet file, with that JM should import all used addresses in a single step after which you restart bitcoind/bitcoin-qt with -rescan or call bitcoin-cli rescanblockchain. As I understood, that was also tried and for some reason didn't work.

AdamISZ commented 5 years ago

There was the original P2PKH wallet.json which I stopped using when I migrated to segwit when that became available (created a new wallet and swept the mix levels to the new addresses). Then there was the move from .json to .jmdat.

@HamishMacEwan Did you set segwit = false in the config? It looks like you didn't.

On the surface, that seems like too bizarre of a mistake, because you would have seen that the addresses are of the wrong type. But unless I'm reading you wrong .. isn't this the case? You're actually trying to recover an old p2pkh wallet?

HamishMacEwan commented 5 years ago

You're actually trying to recover an old p2pkh wallet? @AdamISZ

No. I was describing, again poorly it would seem, the history of the JoinMarket wallets I have used with JoinMarket.

There was the original P2PKH wallet.json which I stopped using when I migrated to segwit when that became available (created a new wallet and swept the mix levels to the new addresses). Then there was the move from .json to .jmdat. @HamishMacEwan

What I am trying to recover is the Segwit wallet that was migrated from .json to .jmdat.

Using a very large gap limit and then using -rescan again should solve this issue. @undeath

OK, well I did a python wallet-tool.py -g 6000 wallet.jmdat and was going to do some fiddling to determine the minimal -rescan using the command @kristapsk recommended, but I've kicked off another full -rescan.

My intention was to use the information gleaned from: bitcoin-cli listtransactions where I found four "joinmarket-wallet-xxxxxx" labels. One I can't identify, the remainder are:

My plan was to identify the earliest block containing a "joinmarket-wallet-d6fef8" labelled transaction and use that blockheight to minimise the -rescan.

HamishMacEwan commented 5 years ago 
Enter wallet decryption passphrase:
2019-07-21 14:35:17,913 [WARNING]  Connection had broken pipe, attempting reconnect.
JM wallet
Balance for mixdepth 0: 0.00000000
Balance for mixdepth 1: 0.00000000
Balance for mixdepth 2: 0.00000000
Balance for mixdepth 3: 0.00000000
Balance for mixdepth 4: 0.00000000
Total balance:  0.00000000

It seems to me the idea is to have joinmarket flag some addresses to Bitcoin Core, and then -rescan is to generate AddToWallet events for transactions that match those addresses, right?

Well it seems from everything I've seen and passed on, those transactions are there in the Bitcoin Core wallet.dat and yet... joinmarket won't see them and repeated gaps and -rescans are not working.

AdamISZ commented 4 years ago

Was this ever solved @HamishMacEwan ?

chris-belcher commented 4 years ago

Try using the new no-history sync which is in the master branch. It allows you to find out the balance of your wallet without rescanning the blockchain, so you can more quickly try different values for gap limit

kristapsk commented 4 years ago

What's the status of this? Can be closed? @HamishMacEwan ?

HamishMacEwan commented 4 years ago

The problem has not resolved in respect to joinmarket-clientserver, if anything it has worsened, but since I'm about to give up on it, it may as well be closed.

I'll try the suggestions above, and re-open if I discover anything useful.

HamishMacEwan commented 4 years ago

OK, I installed the master branch successfully. 0.6.1 threw ValueError: Invalid blockchain source

joinmarket-clientserver/scripts $ python3 wallet-tool.py recover
.
.
File "/opt/mynode/joinmarket-clientserver/jmclient/jmclient/jsonrpc.py", line 91, in queryHTTP
    "authentication for JSON-RPC failed"

I have used the recover method successfully with 0.6.1 using cookie authentication: rpc_cookie_file = /home/bitcoin/.bitcoin/.cookie

Reinstalling 0.6.1 has restored the original and expected behaviour.

pelotius commented 3 years ago

Try using the new no-history sync which is in the master branch. It allows you to find out the balance of your wallet without rescanning the blockchain, so you can more quickly try different values for gap limit

  • set blockchain_source = bitcoin-rpc-no-history in the config file joinmarket.cfg
  • restore your wallet from seed phrase with $ python3 wallet-tool.py recover
  • run $ python3 wallet-tool.py --recoversync -g 1000 newwalletfile.jmdat and see if your money appears. If not try raising the -g value to however many addresses you think you used.

Hope you don't mind me commenting on this old issue, but it seems to be the most relevant place for my situation.

I restored an old segwit wallet and have been able to retrieve the valid current wallet state using your directions here - thanks @chris-belcher

Problem is - when I try to run the yield generator, the yg initiates with an invalid (historic) wallet state making it unusable. Commitments that no longer exist are being added for market making and therefore fail and are blacklisted.

How can I ensure that the current wallet state from --recoversync is used with yg-privacyenhanced? I have tried including the above options in the yg command but this does nothing. Please can you tell me what the recommended way of doing this is

chris-belcher commented 3 years ago

@pelotius that is not possible directly, because yg-privacyenhanced.py and other scripts must know which addresses to use for change outputs, and no-history sync can't provide that information.

What you need to do is get back to a state where you're using blockchain_source = bitcoin-rpc. You use rescanblockchain to rescan and find all your transactions. Because you already know the oldest unspent transaction from the no-history sync you can use that date to start the rescan from.

Another option if that just doesn't work is to create a brand new wallet and then sweep the old wallet into the new (which will work because sweeping doesn't require a change address). That will cost you miner fees though so ideally you'd avoid it.

pelotius commented 3 years ago

@pelotius that is not possible directly, because yg-privacyenhanced.py and other scripts must know which addresses to use for change outputs, and no-history sync can't provide that information.

What you need to do is get back to a state where you're using blockchain_source = bitcoin-rpc. You use rescanblockchain to rescan and find all your transactions. Because you already know the oldest unspent transaction from the no-history sync you can use that date to start the rescan from.

Another option if that just doesn't work is to create a brand new wallet and then sweep the old wallet into the new (which will work because sweeping doesn't require a change address). That will cost you miner fees though so ideally you'd avoid it.

Thanks for getting back to me so quickly @chris-belcher

Yep I'd prefer to avoid sweeping to avoid miner fees but also avoid merging all of my utxos again, even those within mixdepths if at all possible.

I tried a rescan a few days ago but it didn't work - it's possible that I forgot to switch back to blockchain_source = bitcoin-rpc. Would this prevent the rescan from saving the current wallet state in Bitcoin core?

I'll try another rescan from oldest utxo block, ensuring this time that blockchain_source = bitcoin-rpc. Hopefully should sort it. Will report back

AdamISZ commented 3 years ago

@pelotius make sure you run wallet-tool.py with --recoversync, in some circumstances this is required to see all the coins.

pelotius commented 3 years ago

@pelotius make sure you run wallet-tool.py with --recoversync, in some circumstances this is required to see all the coins.

Thanks, in the end I just swept the old segwit wallet