Restore Issues (funds missing)

[raladev]

Describe the bug Sometimes I faced situations when I lose my balance after lnd restoring with backup files or using only seed. Sometimes it works, but sometimes is not, Im tired T_T

To Reproduce A:

1. Perform clean docker simnet installation
2. Trade a little bit (optional)
3. Close channels
4. Wait for closing
5. docker rm -f && docker rmi $(docker images) && sudo rm -rf /.xud-docker/simnet
6. perform new installation and restore wallet w/o backup files

B:

1. Perform clean docker simnet installation
2. Trade a little bit (optional)
3. Close channels
4. Wait for closing
5. docker rm -f && docker rmi $(docker images) && sudo rm -rf /.xud-docker/simnet
6. perform new installation and restore wallet with backup files

Actual result No balance on wallet

Expected result Restored balance on wallet

Data For now there are two seeds. Both installations were performed yesterday. 1) For first I closed all channels (but Im not sure that i waited confirmations of closing) and removed all backup files, but cant restore wallet balance.

Seed:
 1. absent      2. cereal      3. table       4. tribe     
 5. slim        6. certain     7. sheriff     8. mandate   
 9. casual     10. skin       11. save       12. alarm     
13. domain     14. lift       15. supply     16. prefer    
17. fatal      18. item       19. dutch      20. message   
21. scale      22. album      23. zone       24. govern    

Approx balance: BTC - 1.29983313
LTC - 4.98999998

Full output to cli to be sure about seed: full_cli_output.txt

2) I closed all channels and checked confirmed balance of nodes (u can find in in cli_output). I tried restoring with and w/o backup files, but can get my balance back. There is even no info about closed channels when restore from backup file, but addresses and aliases are same.

Seed:
 1. abandon     2. second      3. escape      4. narrow    
 5. reopen      6. way         7. ostrich     8. coin      
 9. asset      10. consider   11. globe      12. student   
13. stage      14. antenna    15. scrap      16. engine    
17. limb       18. uncle      19. ribbon     20. club      
21. upon       22. country    23. broken     24. hammer  

Approx balance: BTC - "confirmed_balance": "189999934", LTC - "confirmed_balance": "506000000",

Full cli output for clean installation - full_cli_output2.txt Archive with backup files - backup_arch.zip

And there is no lnd logs for this cases, sry -_-

[kilrau]

Thanks for opening this! To be sure: Did you do test A (restoring without backup files) & B (restoring with backup files) for both environments or only A for the 1. environement and B for the 2. environment?

Reason why I am asking: https://github.com/ExchangeUnion/xud-docker/files/4651361/backup_arch.zip only contain one set of backup files, I am assuming from environment 2.

[raladev]

Thanks for opening this! To be sure: Did you do test A (restoring without backup files) & B (restoring with backup files) for both environments or only A for the 1. environement and B for the 2. environment?

Reason why I am asking: https://github.com/ExchangeUnion/xud-docker/files/4651361/backup_arch.zip only contain one set of backup files, I am assuming from environment 2.

A for first; A and B for the second;

There is backup files only for second case(abandon) because for the first(absent) i didn't store them.

[kilrau]

Today my restore attempt on testnet:

 Warming up...

Do you want to create a new xud environment or restore an existing one?
1) Create New
2) Restore Existing
Please choose: 2
Please paste the path to your XUD backup to restore your channel balance, your keys and other historical data: /home/kilrau/.xud-docker/testnet
Checking files... Looking good. This will restore xud, lndbtc and lndltc. Do you wish to continue? [Y/n] 
BEWARE: Restoring your environment will close your existing lnd channels and restore channel balance in your wallet. Do you wish to continue? [Y/n] 

You are restoring an xud node key and underlying wallets. All will be secured by
a single password provided below.

Enter your 24 word mnemonic separated by spaces: (node:154) UnhandledPromiseRejectionWarning: Error: EISDIR: illegal operation on a directory, read
(node:154) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:154) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:154) UnhandledPromiseRejectionWarning: Error: EISDIR: illegal operation on a directory, read
(node:154) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
Enter your 24 word mnemonic separated by spaces: absorb mammal snap enrich ski basket sniff super degree quick spread grit rich clutch goose prison scrub menu boost film family exclude faint adjust
Enter a password: 
Re-enter password: 

Do you want to create a new xud environment or restore an existing one?
1) Create New
2) Restore Existing

[raladev]

Today my restore attempt on testnet:

 Warming up...

Do you want to create a new xud environment or restore an existing one?
1) Create New
2) Restore Existing
Please choose: 2
Please paste the path to your XUD backup to restore your channel balance, your keys and other historical data: /home/kilrau/.xud-docker/testnet
Checking files... Looking good. This will restore xud, lndbtc and lndltc. Do you wish to continue? [Y/n] 
BEWARE: Restoring your environment will close your existing lnd channels and restore channel balance in your wallet. Do you wish to continue? [Y/n] 

You are restoring an xud node key and underlying wallets. All will be secured by
a single password provided below.

Enter your 24 word mnemonic separated by spaces: (node:154) UnhandledPromiseRejectionWarning: Error: EISDIR: illegal operation on a directory, read
(node:154) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:154) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:154) UnhandledPromiseRejectionWarning: Error: EISDIR: illegal operation on a directory, read
(node:154) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
Enter your 24 word mnemonic separated by spaces: absorb mammal snap enrich ski basket sniff super degree quick spread grit rich clutch goose prison scrub menu boost film family exclude faint adjust
Enter a password: 
Re-enter password: 

Do you want to create a new xud environment or restore an existing one?
1) Create New
2) Restore Existing

it is because of this one https://github.com/ExchangeUnion/xud-docker/issues/450 Your backup folder contains other folders

[raladev]

Testing results:

1. I got balance back always (after one or after several restore attempts).
2. If u dont get balance back or pending close channel after restore in 5 min, you will not get them for this installation
3. Main assumption: Restore problems connected with time of restore. It seems u will not get back neither wallet balalnce nor channel balance if not enough time passed after your lnd being online. ScreenTable

Reasoning:

1. On simnet instance this problem is synchronous for lndbtc an lndltc. If u dont get lndltc balance, result of lndbtc would be same. (they have same new block creation time)
2. Based on first point we can assume that this problem is not connected with lndbtc or lndltc init sync actions or our race conditions in utils restore flow.
3. Restoring after balance getting was always successful with old backup files and seeds

[kilrau]

Thank you! Main result is: funds can always be restored (just might need to try several times). Your subresult that "the time your lnd being online" seems to matter, hints on yet another Neutrino issue.

Next step: testing restore with a full-node environment

[raladev]

Result of testnet run with native litecoind and bitcoind nodes:

Summary:

1. Balance lose is reproduced, but I have no good explanation for that result.
2. Manual restore through lnd always works
3. Looks like there is connection between lndbtc and lndltc but i cant explain that (based on 3 runs)

Steps:

1. Create env with LTC/BTC wallet balance + LTC/BTC/ETH channel balance 1.1 rm containers and data in testnet folder
2. Restore balance through xud flow -> ETH channel is fine, no BTC/LTC wallet balance, no BTC/LTC channel balance
3. wait ~30mins, check getbalance -> same result 3.1 rm containers and data in testnet folder
4. Try restore balance through xud flow -> same result 4.1 rm containers and data in testnet folder
5. Try restore balance manually for lnd-BTC (do nothing with lnd-LTC) 5.1 lncli --network=testnet --chain=bitcoin create -> wallet balance restored 5.2 lncli --network=testnet --chain=bitcoin restorechanbackup --multi_file=./lnd-BTC -> channel restored 5.3 rm containers and data in testnet folder
6. Try restore balance through xud flow again -> ETH channel is fine, BTC wallet balance and BTC channel balance are fine, LTC wallet balance and LTC channel balance are fine.

[kilrau]

This is indeed hard to explain... oh dear.

I am changing this to P2 since we could verify that in all scenarios wallet and channel balance could eventually be restored. Ty for this extensive testing frenzy!!

[raladev]

restoring with native xud results:

1.2 correct restore and 3 in row lost balance;

2. if u restore wallet and after that have admin.macroon problem (for lncli --network simnet --chain bitcoin walletbalance u get admin.macroon error) u will not get your balance after lnd restart;
3. It seems restore does not start and LightningWallet does not open. Second one is the reason of admin.macroon problem. On screenshot below u can see logs lndbtc: left side is log of lndbtc after restoring through xud with admin.macroon problem, right side is log of lndbtc after manual restore using lncli create --multi_file ./lnd-LTC U can see that left side do not contain restore line and LightningWallet opening line. Screenshot from 2020-07-22 16-44-19

So, for now we need to find our misusing of create/restore flow in xud because manual lnd's create/restore function always works

restore_admin_macroon_xud.log restore_admin_macroon_lndltc.log restore_admin_macroon_lndbtc.log

succ_manual_recovering_lndltc.log succ_manual_recovering_lndbtc.log

Note: we solved admin.macroon problem on sud-docker env by lnd restart but it looks like hack instead of correct solution.

[kilrau]

To make it short: there seems to be a problem with how https://api.exchangeunion.com/#createnode & https://api.exchangeunion.com/#restorenode are implemented (yes both). These calls seem to covertly fail at times with lnd.

@raladev verified that missing funds can be reproduced with a native (non-dockerized) xud (following https://docs.exchangeunion.com/development/developer-guide) using xucli create & xucli restore. Since using a native xud should make debugging much easier (change log level, add additional log statements etc) - would you mind doing a couple of restores with a non-0 lndbtc walletbalance using a native xud and a dockerized lndbtc trying to figure out what's going wrong when the result is "restored balance = 0"? @sangaman

[sangaman]

I have recreated this now twice, and in both cases the missing balance eventually showed up after several hours with no further intervention on my part. I can't see anything wrong in the logs when we restore nor anything wrong with the way we restore an existing wallet, it seems to mirror the lncli create process almost exactly,

@raladev Could you try recreating again and then letting lndbtc run for a few hours and then see if the balance is still missing?