maidsafe / safe_network

72 stars 40 forks source link

bug: safe wallet send debits the wallet if the send fails and uploading seems to hit problems with the wallet #1991

Open happybeing opened 1 month ago

happybeing commented 1 month ago

I've set up two wallets which I use for receiving donations (savings wallet) and one which I use to pay for uploads (uploads wallet). These are on a VPS so I can do uploads from there and eliminate the possibility that my mobile broadband connection is in part causing the issues I'm seeing when uploading to the beta.

Starting with donations of 1650 in my 'savings' wallet, I tried to send 450 to my 'uploads' wallet. That failed to register the transfer on the network (see below), but the wallet was still debited and so only has 1250 left. I sent 450 again, this time successfully, and the wallet then had 750 left.

Here's the transfer which failed, but which still debited the wallet.

safe wallet send 0.000000450 b580cbd0dae688e26976ae67ac57e5682096db86154719a3470655c2cdbdf8454be591ccdbbd2665ba835b527c6cdbaf
Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-07-21_17-06-03"
safe client built with git version: b1e0cc9 / stable-2024-07-08 / b1e0cc9 / 2024-07-08
Instantiating a SAFE client...
Connecting to the network with 50 peers
Failed to send NanoTokens(450) to b580cbd0dae688e26976ae67ac57e5682096db86154719a3470655c2cdbdf8454be591ccdbbd2665ba835b527c6cdbaf due to Wallet(CouldNotSendMoney("The transfer was not successfully registered in the network: CouldNotSendMoney(\"Failed to send spend requests to the network:93b61b7b92b3b8cb6e2c0970528a29812a441d7f65ded12d7641f1342b5a74f2ea09885ae1485670fa1bf217d7fca98d: Network Error GetRecord Query Error QueryTimeout.\")")).
Completed with Err(
   0: Wallet Error Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Failed to send spend requests to the network:93b61b7b92b3b8cb6e2c0970528a29812a441d7f65ded12d7641f1342b5a74f2ea09885ae1485670fa1bf217d7fca98d: Network Error GetRecord Query Error QueryTimeout.").
   1: Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Failed to send spend requests to the network:93b61b7b92b3b8cb6e2c0970528a29812a441d7f65ded12d7641f1342b5a74f2ea09885ae1485670fa1bf217d7fca98d: Network Error GetRecord Query Error QueryTimeout.")

Location:
   sn_cli/src/bin/subcommands/wallet/hot_wallet.rs:297

Wallet status shows:

safe wallet status
Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-07-21_17-34-43"
safe client built with git version: b1e0cc9 / stable-2024-07-08 / b1e0cc9 / 2024-07-08
0.000000750
Unconfirmed spends are:
Available cash notes are:
CashNote { unique_pubkey: b069bd8944a905f15bf50c4c3b0e89c5123ffb0fc281e35559d26be728643a8571802ad6ac7adf1554d121f24bce08f4, parent_spends: {SignedSpend { spend: Spend { unique_pubkey: 99e57b982bbdde62e0951008a23512c2c6a7fd15f6114841fa7b77a0ceaeab51baa4841bd112ff378c9702f7db9b6741 } }, SignedSpend { spend: Spend { unique_pubkey: b31dca1f0395463461ffff0113a215f2376dcb8df7b7d21c19d4b58c7e1e6876883e8fb7b3def924394c57aebbd13707 } }}, main_pubkey: 80fb6b7b366510ed8f210a4b661fd74ba2fcdf2bfd734f24da7597f5ecc19a2e7f63324c94ab7959b6364736343f17fe, derivation_index: 392edf.. }

safe client:

$ safe -V
sn_cli 0.93.9

The VPS is running Ubuntu 20.04 __ Is there a way to recover nanos lost in a failed send? I'm losing lots of nanos this way which is making testing my app very difficult!

Alternatively can Autonomi supply me with more nanos on request. I've just spent two days creating a script to manage wallets and safe client commands on my VPS, to reduce the chances of losing nanos when uploading with my client as happened recently, but now I'm losing nanos when transferring from the 'savings' wallet to the 'uploads' wallet! :man_facepalming: __ @mickvandijke I've only tried one upload so far, starting with 450 nanos in the uploads wallet. This failed when the wallet had dropped to 250 nanos, saying it wanted 317 nanos for a chunk - I think that message may be misleading though. I think it means for all remaining chunks. Also, that command doesn't tell you how many chunks are uploaded or remaining which is unhelpful.

Also, more nanos would help if you can provide them because I'm still losing them due to problems with safe wallet send even now I'm being super careful with the wallets.

Not sure whether to put this in a separate issue, but trying the upload again from the 'uploads' wallet I got a report of "Pre-Unconfirmed transactions" which it seemed to clear. Then warns not to close the client, but I've been staring at this message for over 30 minutes and nothing is happening.

Wallets surely should not be corrupted if a connection is lost, a client crashes etc. Especially if they lock up like this. And this is using safe files upload from a VPS:

Pre-Unconfirmed transactions exist, sending again after 30 seconds...
It's safe to terminate the work, but do remember to retain the unconfirmed_spend file during wallet update.
Otherwise, you are in risk to make the wallet corrupted.
Unconfirmed Spend SpendAddress(928e2c) has no copy in the network yet Network(GetRecordError(RecordNotFound)) !
Parent SpendAddress(413ac4) of unconfirmed Spend SpendAddress(928e2c) is find having at least one copy in the network !
Wallet is now all cleared, OK to progress further.
WARNING: Closing the client now could corrupt the wallet !

Just as I finished typing the above the following appeared in the terminal:

Pre-Unconfirmed transactions exist, sending again after 30 seconds...
It's safe to terminate the work, but do remember to retain the unconfirmed_spend file during wallet update.
Otherwise, you are in risk to make the wallet corrupted.
Unconfirmed Spend SpendAddress(385518) is find at least one copy in the network !
Unconfirmed Spend SpendAddress(b0a76c) is find at least one copy in the network !
Wallet is now all cleared, OK to progress further.
WARNING: Closing the client now could corrupt the wallet !

And now I'm waiting again ... and after another 20-30+ minutes it has errored out with this:

Completed with Err(
   0: Failed to upload chunk batch: The upload failed with maximum repayments reached for multiple items: [8418f0(10000100)..] Summary: UploadSummary { storage_cost: NanoTokens(660), royalty_fees: NanoTokens(68), final_balance: NanoTokens(18), uploaded_addresses: {NetworkAddress::ChunkAddress(0889d2 - b31f6bc70efcd29a3b17f1ff91462d07a8a81c4b76a0e0c8155e6e689476ae25), NetworkAddress::ChunkAddress(0ab023 - ed212b9d90e1f079c84f1c35d9242be4194b368be9340f839093c47d02539b81), NetworkAddress::ChunkAddress(11247c - ab81edd84bae8d7a461f55c5b8eddf478b51e336ec30600d44a4cb1e809e4f1f), NetworkAddress::ChunkAddress(120f8c - c1db9c406ff4d57ac0fbbab62fd78d4912ebe366e7e47e315b42d7e0ab8780a4), NetworkAddress::ChunkAddress(18f392 - 051600d6dc13bf299533e7dee0d0a109aec75ad2895d6b11af81fc8bf651b4f4), NetworkAddress::ChunkAddress(1a9b7f - c5a7bc8fe4446ce373d864e298ce863c0a23d25a91f003ac2d21a734e2075fca), NetworkAddress::ChunkAddress(1f7436 - 935ef24704beb7d205917692057abb65361f6160c53c3142771ef6ae70f7298a), NetworkAddress::ChunkAddress(2391bd - fbe4090d96c1fb2c5cfdfbf4836f76567cd3e69570961bbda826beffc322b70c), NetworkAddress::ChunkAddress(2393fc - b66f81a988e7145dfd2d78b3bf1e75ca94d5edc51d89f069faa23fd909be70e3), NetworkAddress::ChunkAddress(2f4185 - 4bdc3431601bc4d4dca2ba2c255813fb7b6c21462522659b9081b45ae0b06856), NetworkAddress::ChunkAddress(32752b - 1cb3ecd2be5ca9728d9464e099b89b036e9e4270325b55389c9780685ac3abbd), NetworkAddress::ChunkAddress(3de742 - 4cacb765b3aafbe156f964914256b72a16928b3809c340ed7731d5e8760fb575), NetworkAddress::ChunkAddress(421103 - 1bd594f812b73366e6da6790d8d5585c21bd6d6cdeae3f826ef1a2af94100949), NetworkAddress::ChunkAddress(440110 - 1143ec2f1c65b3f28238b9038b71550a890f896adfc8272b08dcb8fb30615a29), NetworkAddress::ChunkAddress(4aea4a - 6295986cc9e6d468cded553ce2bb9d0eae96856014502283e69fd781fc85f7c6), NetworkAddress::ChunkAddress(4e9dee - aca6ee15ff7d3be60d7f67ae72c1fb4a49c13f976fd15edcc674cca7ba40720b), NetworkAddress::ChunkAddress(4f9dee - 48592237da0a8bf354f8312257291c96c49a156d5aa0a3072a3ec4733b88e452), NetworkAddress::ChunkAddress(542a09 - f85bfd8c7cf1006b422d3f7b504ff0d97b04c9f54682dea5ea906cc439f24c60), NetworkAddress::ChunkAddress(5466ea - 2c9af2c697bcfaa6cd7126067f71b6f3ef0f3b7c0c8e3887f52b8df3b9b0204c), NetworkAddress::ChunkAddress(610bac - ad84bec82f32c62deeb7e274cae8b069e83d31f31b4453eee831b0ab9232593d), NetworkAddress::ChunkAddress(67f548 - 0ae76c621e4c74d1e768bc1f5df7537b9b48df7ef823b729a92bb793863a63a9), NetworkAddress::ChunkAddress(7107df - 05c5410c610be9489bd5c8b454f1579f82f53044adf76376dddec39c12d3f4d2), NetworkAddress::ChunkAddress(79ac97 - c4851009718407f6e97f1e0d21afd042c36652d89716b8007023cec22bbad909), NetworkAddress::ChunkAddress(7db292 - 2ddedee2b08f5e3bc55e22773d566e065132a28f0b84fe7a1d2cf164ba6c8e2d), NetworkAddress::ChunkAddress(7f6bed - ee8e6fa644bc5469410ef3a2c605f222bc799f3a0f9552911a20b0e154f5c087), NetworkAddress::ChunkAddress(83d14b - 17a18712aa185459b2f39e11fda13ef8845817c294ce05307757e424ce1fcfe0), NetworkAddress::ChunkAddress(8d795a - 89c061b650234f0faa51e3a7e466b53a2ca9b288eb96b6f83be35608d2ac78c1), NetworkAddress::ChunkAddress(971591 - 69a10941522db2f033882d4eb49d6eb77b2b2ac426e060a3fb15ecdb9ba67c88), NetworkAddress::ChunkAddress(9a2d2f - 9433350842d3446bf4bb25cb07f20d8631ea6ede829d87a374ac40614dd8e0c8), NetworkAddress::ChunkAddress(a0ec3f - 0dded95ffb3f15ec5ca797a2a029fa23638c3d86c68742d3e4a0d26fce913b7e), NetworkAddress::ChunkAddress(a3f2ad - e9946c400a222078e53548b421db613089ea83bd41e74582ff7bf8453a5e6e44), NetworkAddress::ChunkAddress(a98847 - 794a8741c10459ed7b12a546f8b593980cd9af8b117e9f81127fa0073831c086), NetworkAddress::ChunkAddress(aed019 - 381952f0514f1bce11e8e4456c1edc5abb34cb862559a2d0e58277a778784444), NetworkAddress::ChunkAddress(aedb3c - a2f22e622503a3ab220bd5d92e35399168d203c976fa3ea37a3d946e11b7c45a), NetworkAddress::ChunkAddress(b3c30b - 45ba61b4c3c7e4689771533013229cb611124da10c4613a9f1601298e7b51c35), NetworkAddress::ChunkAddress(b5b641 - 467b37fd27077ac33cadd19b06f1aa220e2dc65edcfbf088ecd9d9f57ac6812f), NetworkAddress::ChunkAddress(be64f2 - ec43a426075a42060de67e0443f58911bb93be1cb9190a7566d8964251d076fe), NetworkAddress::ChunkAddress(c37458 - c326fb4b03d2c2b57fa18ce1db69afb758737a267bd2f7b4396a789c4856491b), NetworkAddress::ChunkAddress(cbb599 - 6c0e44f6b834bd725a6ec0710b8204f6cd7a842f799ec7d837995690b22c8b5f), NetworkAddress::ChunkAddress(d24c90 - b835b92be1c1afcb1dcd43d6e89e49d6336fb48e10488a60fdab155e938cc57a), NetworkAddress::ChunkAddress(e6c5f1 - 07322643e89c8616b259ff43916bedeea9a64c531314ff1a0076f057b43dadde), NetworkAddress::ChunkAddress(e88192 - 44a03a50ab6bf66a0f84d07ddf75d05a6e5542b6dda0d7ae5b870ac0a97aa222), NetworkAddress::ChunkAddress(e9f09b - e4e3f059c1bce9ac45d436e9b38e12adda2a79e34f4bc09718fc3afaa1f82b80), NetworkAddress::ChunkAddress(efe694 - 2c93857e885e4daa8093dc4462ce377158b83193c541c6d05f40e733babd4cba), NetworkAddress::ChunkAddress(f0c1ff - 107c50b060d9e660b0dd7afdf0621dbcff374cfaa00c98038b997d49673d0640), NetworkAddress::ChunkAddress(f3c12b - 2368f5f478c21339ecbc7b452c1d51457fd2baa8df9ba82430c484f2b536b161), NetworkAddress::ChunkAddress(f45867 - d58aacc9727be0bc101945b0da4c75ffdad55445470542d1f0fa0a1e2e18878a), NetworkAddress::ChunkAddress(f540d2 - dbd3bc3744c9718bb357375807f4ae12906e2e7ab92be6b98a165c6ca712d853), NetworkAddress::ChunkAddress(f851c9 - ba76a9bd21676baaab5959a70c4dd12756ccdf7e127bc0cefc8a45bc65b2d911), NetworkAddress::ChunkAddress(f9e1e2 - f6cad81f9570708e4172e5139db47a1357b0aed60d5946e42a5bd6f27c65241f)}, uploaded_registers: {}, uploaded_count: 43, skipped_count: 7 }

Location:
   /home/runner/work/safe_network/safe_network/sn_cli/src/files/files_uploader.rs:171

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.) of execute "Files(Upload { file_path: \"/tmp/vps-wallet-tmpdir/v1-awv-type-site/\", batch_size: 16, make_data_public: true, retry_strategy: Balanced })"

I've now only 18 nanos left (of 1650 donated) by a kind community member so cannot do more testing for now. Send more please!

happybeing commented 1 month ago

I have 7 of 76 chunks to go from the upload I was testing but only 18 nanos left. 450 were eaten by the network though, so 69 chunks cost 1200 nanos or 18 per chunk which seems very reasonable.

This confirms the earlier bugfix :pray:

happybeing commented 1 month ago

Just to confirm I'm seeing repeated loss of nanos due to a safe wallet send command not registering the transfer on the network.

This morning I sent 300, and then 300 again and both failed and 600 nanos were deducted from the sending wallet, and of course nothing was received:

safe wallet send 0.000000300 b580cbd0dae688e26976ae67ac57e5682096db86154719a3470655c2cdbdf8454be591ccdbbd2665ba835b527c6cdbaf
Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-07-23_09-31-05"
safe client built with git version: b1e0cc9 / stable-2024-07-08 / b1e0cc9 / 2024-07-08
Instantiating a SAFE client...
Connecting to the network with 50 peers
đź”— Connected to the Network                                                                                                                  Failed to send NanoTokens(300) to b580cbd0dae688e26976ae67ac57e5682096db86154719a3470655c2cdbdf8454be591ccdbbd2665ba835b527c6cdbaf due to Wallet(CouldNotSendMoney("The transfer was not successfully registered in the network: CouldNotSendMoney(\"Failed to send spend requests to the network:93f97c5f2c44e8d71e9ac12f333e9d9a9f9ba1391ed7b8dec81830394a4d3ed436f61f7f4f0f2ef87aaec6b43f0e2602: Network Error Record not stored by nodes, it could be invalid, else you should retry: NetworkAddress::RecordKey(f3f366 - 616124732374eb6e8490a2456e26fecb88fe693abcfb2c017ed061cf8316ce0f).\")")).
Completed with Err(
   0: Wallet Error Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Failed to send spend requests to the network:93f97c5f2c44e8d71e9ac12f333e9d9a9f9ba1391ed7b8dec81830394a4d3ed436f61f7f4f0f2ef87aaec6b43f0e2602: Network Error Record not stored by nodes, it could be invalid, else you should retry: NetworkAddress::RecordKey(f3f366 - 616124732374eb6e8490a2456e26fecb88fe693abcfb2c017ed061cf8316ce0f).").
   1: Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Failed to send spend requests to the network:93f97c5f2c44e8d71e9ac12f333e9d9a9f9ba1391ed7b8dec81830394a4d3ed436f61f7f4f0f2ef87aaec6b43f0e2602: Network Error Record not stored by nodes, it could be invalid, else you should retry: NetworkAddress::RecordKey(f3f366 - 616124732374eb6e8490a2456e26fecb88fe693abcfb2c017ed061cf8316ce0f).")

### Balance after:
 safe wallet balance
Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-07-23_09-34-37"
safe client built with git version: b1e0cc9 / stable-2024-07-08 / b1e0cc9 / 2024-07-08
0.000000400

### Try again immediately after it fails again:
$ safe wallet send 0.000000300 b580cbd0dae688e26976ae67ac57e5682096db86154719a3470655c2cdbdf8454be591ccdbbd2665ba835b527c6cdbaf
Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-07-23_09-34-44"
safe client built with git version: b1e0cc9 / stable-2024-07-08 / b1e0cc9 / 2024-07-08
Instantiating a SAFE client...
Connecting to the network with 50 peers
đź”— Connected to the Network                                                                                                                  Failed to send NanoTokens(300) to b580cbd0dae688e26976ae67ac57e5682096db86154719a3470655c2cdbdf8454be591ccdbbd2665ba835b527c6cdbaf due to Wallet(CouldNotSendMoney("The transfer was not successfully registered in the network: CouldNotSendMoney(\"Failed to send spend requests to the network:98235edecdce847837c9a5ce346b77af80af0acd0668f56ad47f532ea0832233101c019d92b5f829e798a7c2b12e487c: Network Error Record not stored by nodes, it could be invalid, else you should retry: NetworkAddress::RecordKey(471aff - e30009c5c914a9da8350b82a32b8011028a7c63619ead9aff89e5ebbe71a7673).\")")).
Completed with Err(
   0: Wallet Error Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Failed to send spend requests to the network:98235edecdce847837c9a5ce346b77af80af0acd0668f56ad47f532ea0832233101c019d92b5f829e798a7c2b12e487c: Network Error Record not stored by nodes, it could be invalid, else you should retry: NetworkAddress::RecordKey(471aff - e30009c5c914a9da8350b82a32b8011028a7c63619ead9aff89e5ebbe71a7673).").
   1: Failed to send tokens due to The transfer was not successfully registered in the network: CouldNotSendMoney("Failed to send spend requests to the network:98235edecdce847837c9a5ce346b77af80af0acd0668f56ad47f532ea0832233101c019d92b5f829e798a7c2b12e487c: Network Error Record not stored by nodes, it could be invalid, else you should retry: NetworkAddress::RecordKey(471aff - e30009c5c914a9da8350b82a32b8011028a7c63619ead9aff89e5ebbe71a7673).")

Location:
   sn_cli/src/bin/subcommands/wallet/hot_wallet.rs:297

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.) of execute "Wallet(Send { amount: \"0.000000300\", to: \"b580cbd0dae688e26976ae67ac57e5682096db86154719a3470655c2cdbdf8454be591ccdbbd2665ba835b527c6cdbaf\" })"

$ safe wallet balance
Logging to directory: "/home/safe/.local/share/safe/client/logs/log_2024-07-23_09-38-38"
safe client built with git version: b1e0cc9 / stable-2024-07-08 / b1e0cc9 / 2024-07-08
0.000000100
happybeing commented 1 month ago

Others have been sending to me no problem, so I wondered if this was because the VPS I was doing the send on was overloaded with nodes which might affect connectivity.

So I shut them all down as a test and sent 10. No problem! Sent 100. No problem!

:thinking: ahah, problem solved…

Sent 200. It snaffled them (same error) so that’s 800 down today :man_facepalming:. So it isn’t just the VPS was overloaded with nodes. It’s like there’s a malevolent force in this beta.

It isn't a very powerful VPS but maybe there is some other issue with that particular system that makes it more prone to this.

However, the bug remains that if there's a failure to register a transfer, nanos are being lost and should be recoverable.