Account balance corruption. Staking bug?

[HiHat]

Note The problem occurred on the second day after the node was launched and has not been resolved since than (see what were the steps to do it bellow). I can provide full access to the node. Description When staking after a day or so the sum of "balance" plus "immature_balance" on "getwalletinfo" will be less than the real sum on the node, which you can see on blockchain explorer. The difference will never go away but rather will stay for another day or so and then will increase. The range of the difference might vary from few coins up to four digit sum value. When occurred it might stop the staking (though most of the times it doesn't), once I even suspect it resulted to node stopping. The only way to "treat it" is to reload the node. Steps to solve the problem Originally the node was installed on Ubuntu 22.04.01 LTS. First step was to launch it in Docker (it didn't help). Second step was to use 1(one) processor core instead of 8 since the node is installed on VPS and there could be some "virtual issues" with processor (8 vCPU Cores of AMD EPYC™ 7282 processor 30 GB RAM 400 GB NVMe) (it didn't help). Third step was to creat a new wallet.dat (it didn't help).

To Reproduce Steps to reproduce the behavior: Just wait a couple of days with staking active. I will provide access to the problem VPS (please ask for credentials).

Expected behavior The sum of "balance" plus "immature_balance" will be less than the sum on blockchain explorer. The difference will never go away but rather will stay for another day or so and then will increase.

VPS

• 8 vCPU Cores of AMD EPYC™ 7282 processor 30 GB RAM 400 GB NVMe
• OS: Ubuntu 22.04.01 LTS

[andyoknen]

Thank you for your response! I will think about possible steps to get more details. Access to yours is not needed, I think we can reproduce the situation on the test node. Do not give anyone access to your node with coins!

[HiHat]

All coins withdrawn now. You can perform all tests on it (using testnet)!

[HiHat]

Since the issue seems to deal with the virtualization, the access to the VPS in order to find it out might be vital. My partner (who rents the VPS) is going ask the provider to reassemble (or reconfigure) the VPS on different cluster. BUT HELPING BASTYON CODE TO BECOME BETTER IS MORE IMPORTANT to us and we are ready to wait as long as it takes to test it out.

WE PLAN TO RECONFIGURE IT on December the 6-th (Tuesday). If by that date you won't let know that you need the access to the VPS (or you say PRIOR to the date that you DON'T need it), we initiate the VPS reconfiguration and the issue might get gone (and not be discovered). If you let know that you'd like to try it on the current VPS node, you'll be provided with the access until you say "that's it".

At the moment the problem node is running without coins on it. All the coins are staking at another node!

Thank you and waiting for response!

[andyoknen]

I observe the same problem on our test nodes, OS features or server hardware are not related to this problem.

I haven't delved into the question yet, but I'm almost sure that if you run a node with the argument -zapwallettxes=1, then the difference in your balances will disappear.

[HiHat]

Node was operated with zapwallettxes=1 parameter whole time!

[andyoknen]

after restarting the node, the balance does not match? on my test node, the balance matched after restarting

[HiHat]

After restarting the node, the balance matches. That's the reason to restart it every second day or so.

[andyoknen]

I'm clarifying this because it gives a direction to think about exactly where the problem is - I think the problem is in the staking mechanisms, namely the distribution of coins inside the wallet. There has long been a problem with "rejected" bets that are not accepted by the network, but the wallet ceases to take them into account in the balance. This may not be the only problem, but if the recalculation of the wallet corrects the situation, then perhaps the problem is not very serious.

You need to dig deep inside the wallet while debugging the node - it will take some time, I will keep you informed.

[HiHat]

Ок, do we need to switch to the testnet for wallet debugging? Looking forward to your instructions!

[andyoknen]

@HiHat Hi! I am currently searching for a problem, working with the new version of node 0.21-beta and I don't see a problem yet. In this version, the node code was updated (to version 0.21 bitcoin), where, among other things, there were many improvements in the operation of the wallet.

Since the zapwallettxes method is removed in more recent versions of bitcoin, I suggest you try this sequence to try to reproduce the problem:

• Restart node with zapwallettxes argument on version 0.20.29 and stop after full launch (before first staking block)
• Install 0.21.0-beta9
• Start and wait staking

0.21.0-beta9 available here: https://github.com/pocketnetteam/pocketnet.core/actions/runs/3656434804 https://github.com/pocketnetteam/pocketnet.core/suites/9767857087/artifacts/469349412

[andyoknen]

Although I found the problem on the test node, but my actions were quite chaotic and are not subject to investigation - we need clearer steps to reproduce the problem

[andyoknen]

I also want to note that the node wallet always operates with several addresses - the listaddressgroupings method will show all addresses in the wallet with their confirmed balances

[HiHat]

After switching to testnet and long waiting for blockchain sync (last 5000 blocks was synced in 3 days) bug was reproduced on 0.20.29 and then on 0.21! The sum of "balance" plus "immature_balance" from getwalletinfo became less than the real sum on the node (on blockchain explorer); however, sum calculated by following command is always correct:

pocketcoin-cli -testnet listaddresses | jq '.[].balance' | awk '{n += $1}; END{printf "%.0f\n", n}' | sed 's/........$/.&/'

I can't repair wallet balance on 0.21 by rescanblockchain command (due to removed zapwallettxes option). Only rollback to 0.20.29 repairs the wallet balance for some time with further corruption.

[andyoknen]

The problem is fixed in version 0.21.0-beta10 (available via link below or through docker pocketnetteam/pocketnet.core:0.21.0-beta10) https://github.com/pocketnetteam/pocketnet.core/actions/runs/3694145539

Perhaps after the first launch, it will be necessary to remove previously suspended transactions from the wallet - this can be done via GUI or in the terminal using the following commands:

# Get all transactions for removing
pocketcoin-cli potencialabandontransactions
# Clear suspended transactions
pocketcoin-cli abandontransaction <TX>