search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

Wallet not synced stuck on 721957[Bug] #9672

Closed feccie closed 2 years ago

feccie commented 2 years ago

What happened?

Can't sync a wallet although the blockchain itself is synced. Have tried to delete both DBs - wallet and blockchain, and resynced again. Tried deleting the wallet and importing it back.

I try both on 1.2.11 and 1.2.7 and both mode GUI and CLR are same results. Please help advice

Version

1.2.11 and 1.2.7

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

2021-12-26T00:22:01.341 wallet chia.wallet.wallet_node    : INFO     Requesting blocks 721955-721987
2021-12-26T00:22:01.838 wallet chia.wallet.wallet_state_manager: INFO     Adding coin: {'amount': 117,
 'parent_coin_info': '0xed17c426dacc4ea9a2c3b0c2f6329d6f868638a22760dfbba3300013288f91e5',
 'puzzle_hash': '0x9f1d2df27bc1784cc50bd4b2e257db130a32685af658c55267c208bf36afd1ec'} at 721957
2021-12-26T00:22:01.840 wallet chia.wallet.wallet_state_manager: INFO     Coins removed [Coin(parent_coin_info=<bytes32: 1a9c358095a4eb88e7b17cf76993a48858d2cac5610d8fc32298e55ed658b34b>, puzzle_hash=<bytes32: a58af351e4d45a265eb6fa4ac583dfc0a5e7fd9a4d7be8b74f085e07a767acad>, amount=118)] at height: 721957
2021-12-26T00:22:01.842 wallet chia.wallet.wallet_state_manager: INFO     Found created launcher. Creating pool wallet
2021-12-26T00:22:01.909 wallet chia.wallet.wallet_blockchain: ERROR    Error during db transaction: 'NoneType' object is not iterable
2021-12-26T00:22:01.913 full_node full_node_server        : INFO     Connection closed: 127.0.0.1, node id: 10c75f460903db542a91a097bc2f51052cba726729f7375c2404adf1eba5bd86
2021-12-26T00:22:01.913 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '127.0.0.1', 'port': 8449}
2021-12-26T00:22:02.139 full_node chia.full_node.mempool_manager: INFO     It took 0.002931356430053711 to pre validate transaction
2021-12-26T00:22:02.149 full_node chia.full_node.mempool_manager: INFO     add_spendbundle took 0.00879359245300293 seconds, cost 34952103 (0.318%)
2021-12-26T00:22:01.913 wallet wallet_server              : INFO     Connection closed: 127.0.0.1, node id: fad55ce599c7997f7fa8122ae05ade336b4d155fee60499089cacd55c5f84c85
2021-12-26T00:22:02.915 wallet chia.wallet.wallet_node    : ERROR    Error while trying to fetch from peer:'NoneType' object is not iterable Traceback (most recent call last):
  File "chia\wallet\wallet_node.py", line 555, in batch_sync_to_peak
  File "chia\wallet\wallet_node.py", line 736, in fetch_blocks_and_validate
  File "chia\wallet\wallet_blockchain.py", line 237, in receive_block
  File "chia\wallet\wallet_blockchain.py", line 349, in _reconsider_peak
  File "chia\wallet\wallet_state_manager.py", line 616, in new_transaction_block_callback
  File "chia\pools\pool_wallet.py", line 359, in create
  File "chia\pools\pool_wallet.py", line 232, in update_pool_config
  File "chia\pools\pool_config.py", line 45, in load_pool_config
TypeError: 'NoneType' object is not iterable

2021-12-26T00:22:02.916 wallet chia.wallet.wallet_node    : ERROR    Loop exception in sync Was not able to add blocks 721955-721987. Traceback (most recent call last):
  File "chia\wallet\wallet_node.py", line 608, in sync_job
  File "chia\wallet\wallet_node.py", line 668, in _sync
  File "chia\wallet\wallet_node.py", line 565, in batch_sync_to_peak
RuntimeError: Was not able to add blocks 721955-721987

2021-12-26T00:22:02.916 wallet chia.wallet.wallet_node    : INFO     Loop end in sync job
2021-12-26T00:22:02.916 wallet chia.wallet.wallet_node    : INFO     Loop start in sync job
2021-12-26T00:22:02.917 wallet chia.wallet.wallet_node    : INFO     Have collected 46 potential peaks
2021-12-26T00:22:02.917 wallet chia.wallet.wallet_node    : INFO     No peers to sync to
2021-12-26T00:22:02.918 wallet chia.wallet.wallet_node    : INFO     Loop end in sync job
2021-12-26T00:22:02.918 wallet chia.wallet.wallet_node    : INFO     Loop start in sync job
RY1979 commented 2 years ago

Same here any one know what is going on?? Full node overview: Synced, earning beer money but Wallet Status: Syncing, not yet getting beer money (1,335,846)

Cheers

rmaroun commented 2 years ago

Same pb stuck at 1,341,170 on my raspberrypi farmer

Jacek-ghub commented 2 years ago

Can you do a screenshot of your Connections tab on Full Node panel, or CLI "chia show -c" output?

rmaroun commented 2 years ago

image

Jacek-ghub commented 2 years ago

Usually, when you are fully synced (on the blockchain side), most of your peers end up with the current height. Basically, the number of good peers should reflect the state of the network. If there is no dust storm going one, you should not have that many lagging peers.

Kind of a test would be to drop one of those lagging peers, and monitor the new one, whether it will show height, and it will start moving Up/Down MB counters. If it will not, most likely, your node is close to be overwhelmed. I assume that those are all peers that you have (i.e., you have already limited them to 10). Maybe you could try to further drop the peer count down to 5 and try to see whether that would help.

As far as your RPi, how much memory you have, where is your swap file located (SSD / HD), does blockchain / wallet dbs sit on an SD, SSD, HD?

rmaroun commented 2 years ago

Removed some of the lagging peers and still nothing changed it is stucked at the same high peak. I have 8Gb memory and the chia GUI and wallet and db all sit on a 256Gb micro SD card.

Jacek-ghub commented 2 years ago

Again, the only lagging node is most likely yours, so removing peers that you feel like lagging is not really changing anything. The UI provides states, but without the time stamps, these are rather worthless to depend on.

  1. How fast is your SD card? It would be better to move your dbs to HD (one of your plots holding HD). Maybe even better would be to move blockchain db to one drive, and your wallet db to another one, thus r/w will not interfere with each other. Of course, adding an SSD, even via USB to hold your db (at least your blockchain) would be the best. (Of course, you can also symlink those folder - maybe a cleaner solution.) 
    full_node:
...
database_path: /mnt/hd1/blockchain/blockchain_v1_CHALLENGE.sqlite
peer_db_path:  /mnt/hd1/blockchain/peer_table_node.sqlite
...
wallet:
...
database_path:     /mnt:/hd2/wallet/blockchain_wallet_v1_CHALLENGE_KEY.sqlite
wallet_peers_path: /mnt:/hd2/wallet/wallet_peers.sqlite
  2. Change your peer count in config.yaml as follow (that will further reduce the r/w load on your blockchain db): 
    full_node:
...
target_outbound_peer_count: 10
target_peer_count: 10
feccie commented 2 years ago

I keep all dB and wallet path on my SSD, and try to change peer count that your mention or remove all peers but wallet still does not sync.

Jacek-ghub commented 2 years ago

@feccie looking at the logs you provided, it looks (to me) that your full node drops connections to your wallet. That is not related to problems with number of peers (potentially, that was relevant for RPi).

2021-12-26T00:22:01.909 wallet chia.wallet.wallet_blockchain: ERROR    Error during db transaction: 'NoneType' object is not iterable
2021-12-26T00:22:01.913 full_node full_node_server        : INFO     Connection closed: 127.0.0.1, node id: 10c75f460903db542a91a097bc2f51052cba726729f7375c2404adf1eba5bd86
2021-12-26T00:22:01.913 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '127.0.0.1', 'port': 8449}
...
2021-12-26T00:22:01.913 wallet wallet_server              : INFO     Connection closed: 127.0.0.1, node id: fad55ce599c7997f7fa8122ae05ade336b4d155fee60499089cacd55c5f84c85

Somehow your full node is deciding to drop that connection. The wallet error right before that disconnection rather suggests that it was a serious problem on the wallet side ("'NoneType' object is not iterable"). Unfortunately, there is nothing in the log that would suggest that it was really the cause for that disconnection, and/or whether that was a catastrophic error. Although, the last few lines coming from your wallet suggest that your wallet really went South, whatever that means (e.g., started blindly looping, ...).

Also, if you check the screenshot that @rmaroun provided, you should see your wallet (127.0.0.1) there 24/7. In my case, the Up/Down value right now is at 30.1/0.1. In my case that indicates that my wallet is pushing some data to the full node. I am not sure whether those numbers are significant, as in that screenshot both of them are at 0.

So, sorry but I don't know how to help you. I would hope that some dev would chime in, as the signature of that error is clearly code screwup, and this issue is already marked as a bug. Maybe you can also repost it on chiaforum.com, as over there I saw few people that have much better understanding of wallet issues that I have.

On the other hand, I saw some people suggesting to delete "unconfirmed transactions" as that sometimes helps with wallet issues. Here is the screenshot (click on those dots): image I am not sure if you need to delete the wallet db after deleting those unconfirmed transactions. I would give it some time to sync first.

By the way, check that thread: https://chiaforum.com/t/wallet-cant-connect-sync-to-fullnode-on-local-network/12006/6

feccie commented 2 years ago

Hi All,

Thank you @Jacek-ghub for advice,

I found a thread on ChiaForum with same issue and suggest rollback to version1.1.7 and it works for me.

https://chiaforum.com/t/wallet-not-synced-even-though-node-is-fully-synced/1973/7

Jacek-ghub commented 2 years ago

Great that you got unblocked! Again, sorry that I couldn't help much.

Also, I was running v1.2.6 up to two weeks ago. At that time I tried to update directly to v1.2.11 but couldn't make it work. I found someone else who also couldn't do it from the same version, but v1.2.10 worked for him. I tried that, and it worked right away. I left it like that for a couple of weeks, and just the other day tried upgrading to v1.2.11. It worked this time.

Maybe there is some quirk that prevents upgrades directly from v1.2.6/7 (maybe the last two good versions) to v1.2.11, and first chia needs to be upgraded to v1.2.10?

emlowe commented 2 years ago

Can you post a snippet of your config.yaml file that starts with pool: it will look something like this:

  logging: *id001
  network_overrides: *id002
  pool_list:
  - authentication_public_key:

I believe the problem might be that in config.yaml you have the pool_list: item, but it is empty. Deleting the pool_list line or setting it to pool_list: [] might help with later versions. This also helps to explain why 1.1.7 version works since it doesn't know anything about pools.

RY1979 commented 2 years ago

Mine worked after I updated to latest chia application

On Wed, Dec 29, 2021, 1:55 AM Ralph @.***> wrote:

Same pb stuck at 1,341,170 on my raspberrypi farmer

— Reply to this email directly, view it on GitHub https://github.com/Chia-Network/chia-blockchain/issues/9672#issuecomment-1002143406, or unsubscribe https://github.com/notifications/unsubscribe-auth/AV36H6E3TTAWGFXJDOJAAGLUTHFVRANCNFSM5KX76MHA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>