[Bug] [1.3.0] Wallet stop syncing

[Ealrann]

Hello,

After migrating my setup to 1.3, I have some trouble to have my wallet on sync (so my plotnft isn't synced neither). It appears that the wallet sync normally at the beginning, but when approaching the lasts blocks, the sync process either slow down (Sync status: Syncing... forever), or stop totally (Sync status: Not synced). Even if I let it run 24h, there is no progress anymore, but CPU usage remains high.

I did it on two different computers (one on my farmer, and one on another computer wallet only, both synchronized from scratch), but it appears they all struggle to get in sync. On the other side, I have a newer wallet that keep in sync perfectly. The two wallets report to be connected to peers, and show some bandwidth usage.

My wallet is maybe a bit specific, since I received lots of 1 mojo transactions (the dust storm I guess...). Due to that, it's really long to synchronize, even with the new light wallet.

For now, the main errors I see are something like "ERROR Exception while adding state: XXXXXXXXXX". The XXX message can vary, I'll try to list some of them here:

No Title

2022-03-09T17:10:00.174 wallet chia.wallet.wallet_node    : WARNING  Validate false 1
2022-03-09T17:10:00.237 wallet chia.wallet.wallet_node    : ERROR    Exception while adding state:  Traceback (most recent call last):
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 619, in receive_and_validate
    valid_states = [
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 622, in <listcomp>
    if await self.validate_received_state_from_peer(inner_state, peer, cache, fork_height)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 1189, in validate_received_state_from_peer
    validated = await self.validate_block_inclusion(state_block, peer, peer_request_cache)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 1294, in validate_block_inclusion
    assert res_ses is not None
AssertionError

2022-03-09T17:10:10.181 wallet chia.wallet.wallet_node    : WARNING  Validate false 1
2022-03-09T17:10:10.187 wallet chia.wallet.wallet_node    : WARNING  Validate false 1
2022-03-09T17:10:10.188 wallet chia.wallet.wallet_node    : WARNING  Validate false 1

Was not able to obtain solution

2022-03-09T08:29:29.023 wallet wallet_server              : DEBUG    Time for request request_puzzle_solution: {'host': '185.112.225.80', 'port': 8444} = 8.082389831542969e-05, None? True
2022-03-09T08:29:29.024 wallet chia.wallet.wallet_node    : ERROR    Exception while adding state: Was not able to obtain solution None Traceback (most recent call last):
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 635, in receive_and_validate
    await self.wallet_state_manager.new_coin_state(valid_states, peer, fork_height)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_state_manager.py", line 813, in new_coin_state
    cs: CoinSpend = await self.wallet_node.fetch_puzzle_solution(
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 1375, in fetch_puzzle_solution
2022-03-09T08:29:30.739 wallet chia.wallet.wallet_node    : DEBUG    Address manager query count: 0. Query limit: 25

Error fetching blocks from peer

2022-03-08T21:29:55.154 wallet chia.wallet.wallet_node    : ERROR    Exception while adding state: Error fetching blocks from peer None Traceback (most recent call last):
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 635, in receive_and_validate
    await self.wallet_state_manager.new_coin_state(valid_states, peer, fork_height)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_state_manager.py", line 676, in new_coin_state
    await self.coin_added(coin_state.coin, coin_state.created_height, all_txs, wallet_id, wallet_type)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_state_manager.py", line 1004, in coin_added
    timestamp = await self.wallet_node.get_timestamp_for_height(height)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 820, in get_timestamp_for_height
    raise ValueError(f"Error fetching blocks from peer {peer.get_peer_info()}")
ValueError: Error fetching blocks from peer None

Cannot fetch timestamp

2022-03-02T01:51:09.756 wallet chia.wallet.wallet_node    : ERROR    Exception while adding state: Cannot fetch timestamp, no peers Traceback (most recent call last):
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 637, in receive_and_validate
    await self.wallet_state_manager.new_coin_state(valid_states, peer, fork_height)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_state_manager.py", line 665, in new_coin_state
    await self.coin_added(coin_state.coin, coin_state.created_height, all_txs, wallet_id, wallet_type)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_state_manager.py", line 991, in coin_added
    timestamp = await self.wallet_node.get_timestamp_for_height(height)
  File "/home/chia/git/chia-blockchain/chia/wallet/wallet_node.py", line 787, in get_timestamp_for_height
    raise ValueError("Cannot fetch timestamp, no peers")
ValueError: Cannot fetch timestamp, no peers

Other errors

2022-03-09T17:28:34.839 wallet chia.wallet.wallet_node    : ERROR    Error fetching blocks 1581746 1582100
2022-03-09T17:28:34.840 wallet chia.wallet.wallet_node    : ERROR    Error fetching blocks 1581717 1582100
2022-03-09T17:28:34.840 wallet chia.wallet.wallet_node    : ERROR    Error fetching blocks 1581712 1582100
2022-03-09T17:28:34.841 wallet chia.wallet.wallet_node    : ERROR    Error fetching blocks 1581778 1582100

Version

1.3.0

What platform are you using?

Linux

What ui mode are you using?

CLI

[mariano54]

Can you try removing the wallet DB and retrying? Sometimes it might take a while to sync, especially if you have many transactions. You should be patient and let it run.

If you are connected to a trusted node (farmer mode) then it will sync much more quickly.

[Ealrann]

I did it multiple time (and once again since yesterday). Totally erased the folder .chia and .chia_keyring, do a chia init, reenter the passphrase... Both computer are powerfull enough (recent intel i7 and i3), 16GB of ram, disks are far from full.

Now both wallets show Sync status: Syncing... forever, and cannot reach the last block. If I restart the wallet (chia start -r wallet), it could (sometime) burst sync to the head and show the balance, but unfortunately, on one wallet, the balance is wrong... Maybe #10646

Since the sync process is long (something like 20 hours?), I cannot be sure, but it looks like the errors arrive after a specific bloc. Maybe the light wallet cannot handle correctly some blocks ? Or maybe the thousand, dust storm, 1 mojo transactions, are not well supported by the light wallet ?

I didn't have any problem with the full wallet.

[mariano54]

Do you see your height moving up while you are syncing? Can you give me your log file to sorgente711 on keybase?

[Ealrann]

It's random: sometime it remains stuck on the same block for hours, sometime it's nor fat from the top height, but still stay on "syncing...". Ok, I go to keybase.

[Ealrann]

I tried again today with the main branch, but still, this wallet is almost impossible to get on sync. After 2-3 chia wallet start -r (because of Syncing... forever without any real cpu activity), I got the wallet on sync, but the balance is wrong...

If needed, I'd be glad to help testing, even on an unstable branch.

By the way, do you have any advice to help keeping my main setup (farmer) on sync ? I'm a bit worried that farming with a plotnft not synced prevents me to win blocks...

[Ealrann]

1.3.4 appears to fix the issue. My two wallets now synchronize correctly. Thank you Chia team ;)