search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[Bug] Chia 1.7.0 only syncs CATs as they are received #14610

Closed trgarrett closed 1 year ago

trgarrett commented 1 year ago

What happened?

I have a wallet with an effective derivation index of right around 7,200. With 1.7.0 and a fresh sync, I noticed all my CAT balances were starting out 0. Only as new CATS (such as the daily deposits of MZ token) came in, were they counted. Also, of 4 NFTs in one DID, I saw 2, 3, or 4 of them on different fresh syncs. To me, this points to a problem handling derivation indexes consistently.

image

Version

1.7.0

What platform are you using?

macOS

What ui mode are you using?

GUI

Relevant log output

2023-01-11T19:18:24.328 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Error fetching timestamp from all peers
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1114, in new_coin_state
  File "chia/wallet/wallet_node.py", line 999, in get_timestamp_for_height
chia.wallet.util.wallet_sync_utils.PeerRequestException: Error fetching timestamp from all peers
2023-01-11T19:18:24.332 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Error fetching timestamp from all peers
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1114, in new_coin_state
  File "chia/wallet/wallet_node.py", line 999, in get_timestamp_for_height
chia.wallet.util.wallet_sync_utils.PeerRequestException: Error fetching timestamp from all peers
2023-01-11T19:18:24.336 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Was not able to get states for [b'\xedK\x1b\r\xd1H\x8c\xa5\xa7\\\x80#ZhPl\xa7\xabb\xfc\xe1\x1a\x88\x1fV\xf8\x99\x029\x9c@;']
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1034, in new_coin_state
  File "chia/wallet/wallet_state_manager.py", line 615, in determine_coin_type
  File "chia/wallet/wallet_node.py", line 1618, in get_coin_state
chia.wallet.util.wallet_sync_utils.PeerRequestException: Was not able to get states for [b'\xedK\x1b\r\xd1H\x8c\xa5\xa7\\\x80#ZhPl\xa7\xabb\xfc\xe1\x1a\x88\x1fV\xf8\x99\x029\x9c@;']
2023-01-11T19:18:24.342 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Error fetching timestamp from all peers
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1114, in new_coin_state
  File "chia/wallet/wallet_node.py", line 999, in get_timestamp_for_height
chia.wallet.util.wallet_sync_utils.PeerRequestException: Error fetching timestamp from all peers
2023-01-11T19:18:24.344 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Was not able to get states for [b"^n\xc4\xb1\xa6\xe9$k\x04\xac#\xa4'O\x90\xf3R5\xb5\xdd\x91@W/O{\x9e\xe0\x86\\\xb7\x80"]
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1034, in new_coin_state
  File "chia/wallet/wallet_state_manager.py", line 615, in determine_coin_type
  File "chia/wallet/wallet_node.py", line 1618, in get_coin_state
chia.wallet.util.wallet_sync_utils.PeerRequestException: Was not able to get states for [b"^n\xc4\xb1\xa6\xe9$k\x04\xac#\xa4'O\x90\xf3R5\xb5\xdd\x91@W/O{\x9e\xe0\x86\\\xb7\x80"]
2023-01-11T19:18:24.348 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Error fetching timestamp from all peers
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1114, in new_coin_state
  File "chia/wallet/wallet_node.py", line 999, in get_timestamp_for_height
chia.wallet.util.wallet_sync_utils.PeerRequestException: Error fetching timestamp from all peers
2023-01-11T19:18:24.349 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Was not able to get states for [b"]\xe8J\xc3ZF\x9d\xfa\xa7\xc5\x08\x03\x1e*@K\x95\x97\xd1\\\xc4(^\xb7]\xae'NR}@\xac"]
Traceback (most recent call last):
"~/.chia/mainnet/log/debug.log" [readonly] 39658L, 6554177B
  File "chia/wallet/wallet_state_manager.py", line 1749, in convert_puzzle_hash
KeyError: 24
2023-02-16T22:18:17.672 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... Couldn't generate child puzzle for NFT
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1078, in new_coin_state
  File "chia/wallet/wallet_state_manager.py", line 1531, in coin_added
  File "chia/wallet/nft_wallet/nft_wallet.py", line 175, in coin_added
  File "chia/wallet/nft_wallet/nft_wallet.py", line 242, in puzzle_solution_received
ValueError: Couldn't generate child puzzle for NFT
2023-02-16T22:20:26.945 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:21:10.833 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:21:11.167 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... 37
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1134, in new_coin_state
  File "chia/wallet/wallet_state_manager.py", line 1749, in convert_puzzle_hash
KeyError: 37
2023-02-16T22:21:11.209 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... 37
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1134, in new_coin_state
  File "chia/wallet/wallet_state_manager.py", line 1749, in convert_puzzle_hash
KeyError: 37
2023-02-16T22:21:11.340 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... 37
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1134, in new_coin_state
  File "chia/wallet/wallet_state_manager.py", line 1749, in convert_puzzle_hash
KeyError: 37
2023-02-16T22:21:20.306 wallet chia.wallet.wallet_state_manager: ERROR    Error adding state... 39
Traceback (most recent call last):
  File "chia/wallet/wallet_state_manager.py", line 1134, in new_coin_state
  File "chia/wallet/wallet_state_manager.py", line 1749, in convert_puzzle_hash
KeyError: 39
2023-02-16T22:21:37.243 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:21:39.135 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:21:39.881 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:22:27.537 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:23:03.502 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:24:07.706 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:25:18.918 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:29:00.702 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:29:05.475 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:29:22.881 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-16T22:32:09.909 wallet chia.pools.pool_puzzles    : ERROR    Unexpected return from PoolWallet Smart Contract code cannot unpack non-iterable int object
2023-02-17T00:09:50.214 wallet chia.rpc.rpc_server        : WARNING  Error while handling message: Traceback (most recent call last):
  File "chia/rpc/rpc_server.py", line 331, in safe_handle
  File "chia/rpc/rpc_server.py", line 322, in ws_api
  File "chia/rpc/wallet_rpc_api.py", line 1548, in get_offer_summary
  File "chia/wallet/trading/offer.py", line 611, in from_bech32
ValueError: Invalid Offer

2023-02-17T08:30:19.208 wallet chia.rpc.wallet_rpc_api    : WARNING  NFT wallet 20 has DID 669014c72aff553bbebcb2224d1f49c24a83736a797b92498d6cf9a52817248b but no DID wallet
2023-02-17T08:30:19.209 wallet chia.rpc.wallet_rpc_api    : WARNING  NFT wallet 34 has DID aac3540ad58de32a65ade61118303f90f44fc3e681d1bf3e1b0ae4b5f862d954 but no DID wallet
2023-02-17T08:30:19.210 wallet chia.rpc.wallet_rpc_api    : WARNING  NFT wallet 48 has DID 8ec8c193d7d8753707af7fc1936056eea8a3589c91250ce03f464f8d506b6fea but no DID wallet
trgarrett commented 1 year ago

Couple of additional data points. The issue is definitely intermittent, and it is not new to 1.7.0. I went down to 1.6.2 to try to clean up my state with another fresh sync. Some of the coins from some of my derivation indexes were again not synced. I have now also seen it affect the main XCH balance in both 1.6.2 and 1.7.0.

So, after about 5 full syncs in the last 2 days (with ~3 GB of data transfer each), I haven't had any that found all the coins but I have been able to move some coins to the primary observable address as I had syncs that found them.

trgarrett commented 1 year ago

Behavior continues with 1.8.0-b1.

trgarrett commented 1 year ago

Tried several things. After enabling unknown CAT loading (automatically_add_unknown_cats: true), I am now seemingly able to see all my CAT balances, although I have a lot more wallets than I would prefer to be syncing. I don't remember it behaving this way before. If it's meant to be an improvement rather than a regression, I could see some benefits. Just looking for clarity now.

trgarrett commented 1 year ago

Okay, after a full new sync with 1.8.0-b1, my CAT balances look much improved. I am missing several DID coins now though, and hundreds of NFTs.

I am using a trusted local node for syncing. Should I explore restoring it from an earlier backup?

trgarrett commented 1 year ago

Okay, with 1.8.0-b1 sync stopped and showed Synced at derivation index 2496. Going to manually update that and see if that will get the remaining DIDs and NFTs in place.

Edit: It did not. Trying a sync with untrusted nodes to eliminate possibilities.

Edit: A sync with my own full node in untrusted mode did not work either.

johnathon-b commented 1 year ago

Have you tried to use 1.6.2?

trgarrett commented 1 year ago

Have you tried to use 1.6.2?

Yes. 1.6.2 did not give a clean sync either. I can work backward to 1.6.1, 1.6.0 and find the last version that worked correctly, but I am unable to accept the vast majority of offers made with 1.7.0 if I do that.

johnathon-b commented 1 year ago

I'm seeing the same issues as well. Thank you for confirming it's not just me.

MumfMeisterT commented 1 year ago

Could try running main and run a chia wallet check chia wallet check --db-path ~/.chia/mainnet/wallet/db/blockchain_wallet_v2_r1_mainnet_<your_fingerprint>.sqlite?

trgarrett commented 1 year ago

Nice. A new command! Here's my output @MumfMeisterT:

`Reading /Users/tgarrett/.chia/mainnet/wallet/db/blockchain_wallet_v2_r1_mainnet_2544449244.sqlite No errors found.

Reading /Users/tgarrett/.chia/mainnet/wallet/db/blockchain_wallet_v2_r1_mainnet_1307711849.sqlite

---- Errors Found ----

Missing Wallet IDs: [5 to 9, 12, 14, 18 to 23, 33 to 34, 41, 82, 89]`

aqk commented 1 year ago

Hi there, thanks for using the tool!

This tools automates integrity checking of the wallet DB, but can't tell you why an issue happened.

Missing Wallet IDs can happen for two reasons that I know of:

One, we created a wallet and later deleted it. This is okay in normal operations, as we don't reuse wallet IDs.

Two, there is a bug in the wallet code we are tracking down that can cause these to appear. This bug does not cause any loss on-chain, but can make the wallet's local idea of your balances incorrect.

I'll update the output to be more understandable in the Missing Wallet ID case.

aqk commented 1 year ago

Hello again - the improved messages should hit main soon.

https://github.com/Chia-Network/chia-blockchain/pull/14688

trgarrett commented 1 year ago

@aqk On 1.7.1, the command "chia wallet check" is not recognized. Is this expected to come back in the future?