search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[Bug] GUI Full Node Connects, Begins Syncing and then Stops Syncing Until Restarted #16697

Closed strikeoncmputrz closed 11 months ago

strikeoncmputrz commented 11 months ago

What happened?

Chia connects to peers and begins syncing. After around 5-10 minutes it stops syncing, and displays the last block it processed.

I've let it sit for over 24 hours. My network connectivity is low latency high bandwidth. I have a confirmed port forward to my full node's TCP 8444 port.

I am running Chia Blockchain as a full node in GUI mode. I recently upgraded to the v2 database. To do so, I stopped Chia, downloaded a copy of the V2 database, ran chia db upgrade, which completed successfully, and executed Chia via its Ubuntu application icon.

After I realized it wasn't syncing* I immediately assumed I had a corrupt DB, but validate doesn't indicate that. 

# Output from Validate
$ chia db validate
opening file for reading: /home/REDACTED/.chia/mainnet/db/blockchain_v2_mainnet.sqlite
peak hash: 64e7afc2e1f0a229a7e601f8f2575dbd7572a1cef7c0852e32b2ff7fd9e9d1ae
peak height: 4311580
traversing the full chain
0 orphaned blocks: 4584       
4584 orphaned blocks

DATABASE IS VALID: /home/REDACTED/.chia/mainnet/db/blockchain_v2_mainnet.sqlite

If I stop Chia and restart it, the same behavior repeats. It begins syncing, syncs a few hundred blocks, and then stops syncing. It's definitely aware that there are more blocks. It's currently displaying the following in Synching Status.

image

After I see the first full_node chia.full_node.full_node: ERROR sync from fork point failed: KeyError: in the log the Full Node stops syncing. This exception appears 30 seconds after the KeyError: exception=OperationalError('too many SQL variables')>

More logs below.

Version

2.1.1

What platform are you using?

Linux

What ui mode are you using?

GUI

Relevant log output

2023-10-25T20:43:35.068 full_node full_node_server        : INFO     Connected with full_node 
PeerInfo(_ip=IPv4Address('218.80.68.13'), _port=8444)
2023-10-25T20:44:04.773 full_node chia.full_node.full_node_api: WARNING  Received unsolicited/
late block from peer PeerInfo(_ip=IPv4Address('218.80.68.13'), _port=8444)
2023-10-25T20:44:05.191 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424039 ti
mestamp: 1698281015
2023-10-25T20:44:05.192 wallet chia.wallet.wallet_node    : INFO     Finished processing new p
eak of 4424039
2023-10-25T20:44:07.344 full_node chia.full_node.full_node: INFO     Added blocks 574392 to 57
4424
2023-10-25T20:44:10.677 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424040 ti
mestamp: 1698281015
2023-10-25T20:44:10.678 wallet chia.wallet.wallet_node    : INFO     Finished processing new p
eak of 4424040
2023-10-25T20:44:18.628 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424041 ti
mestamp: 1698281015
2023-10-25T20:44:18.630 wallet chia.wallet.wallet_node    : INFO     Finished processing new p
eak of 4424041
2023-10-25T20:44:20.710 full_node full_node_server        : INFO     Cannot connect to host 28
03:9800:a801:8051:b0ca:9ada:f899:b036:8444 ssl:<ssl.SSLContext object at 0x7f358615d4c0> [Netw
ork is unreachable]
2023-10-25T20:44:25.328 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: e
vent started, loaded 0, removed 0, processed 0, remaining 91, duration: 0.00 seconds, total pl
ots: 91
2023-10-25T20:44:25.344 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: e
vent done, loaded 0, removed 0, processed 91, remaining 0, duration: 0.01 seconds, total plots
: 91
2023-10-25T20:44:26.009 full_node full_node_server        : INFO     Connected with full_node 
PeerInfo(_ip=IPv4Address('203.86.203.131'), _port=8444)
2023-10-25T20:44:32.592 full_node full_node_server        : INFO     Connected with full_node PeerInfo(_ip=IPv4Address('75.52.82.170'), _port=8444)
2023-10-25T20:44:32.594 full_node full_node_server        : INFO     Connection closed: 75.52.82.170, node id: 33f505b367b4a794271dab74c6db0757eee3351fe95f8b5410e3d32d8134330f
2023-10-25T20:44:32.594 full_node chia.full_node.full_node: INFO     peer disconnected PeerInfo(_ip=IPv4Address('75.52.82.170'), _port=8444)
2023-10-25T20:44:38.340 full_node full_node_server        : INFO     Connected with full_node PeerInfo(_ip=IPv4Address('95.165.14.21'), _port=8444)
2023-10-25T20:44:38.343 full_node full_node_server        : INFO     Connection closed: 95.165.14.21, node id: a6237d8f0e30dd91f54b278590b66b5790a08e7ffad084b0f64b120d1b6d9b58
2023-10-25T20:44:38.343 full_node chia.full_node.full_node: INFO     peer disconnected PeerInfo(_ip=IPv4Address('95.165.14.21'), _port=8444)
2023-10-25T20:44:43.942 full_node chia.full_node.full_node: INFO     Added blocks 574424 to 574456
2023-10-25T20:45:01.917 full_node full_node_server        : INFO     Cannot connect to host 112.21.98.137:8444 ssl:<ssl.SSLContext object at 0x7f358615d4c0> [Connect call failed ('112.21.98.137', 8444)]
2023-10-25T20:45:02.176 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424042 timestamp: 1698281076
2023-10-25T20:45:02.177 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424042
2023-10-25T20:45:04.717 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424043 timestamp: 1698281076
2023-10-25T20:45:04.718 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424043
2023-10-25T20:45:04.896 full_node full_node_server        : INFO     Cannot connect to host 2a01:e0a:4c1:7fa0:c1ab:6512:3c49:721e:8444 ssl:<ssl.SSLContext object at 0x7f358615d4c0> [Network is unreachable]
2023-10-25T20:45:14.068 full_node chia.full_node.full_node: INFO     Added blocks 574456 to 574488
2023-10-25T20:45:17.883 full_node full_node_server        : INFO     Cannot connect to host 2001:b011:400b:9f48:c11f:1255:5890:eb78:8444 ssl:<ssl.SSLContext object at 0x7f358615d4c0> [Network is unreachable]
2023-10-25T20:45:19.482 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424044 timestamp: 1698281076
2023-10-25T20:45:19.483 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424044
2023-10-25T20:45:27.597 full_node full_node_server        : INFO     Cannot connect to host 161.97.94.190:8444 ssl:<ssl.SSLContext object at 0x7f358615d4c0> [Connect call failed ('161.97.94.190', 8444)]
2023-10-25T20:45:27.710 full_node chia.full_node.full_node: ERROR    sync from fork point fail
ed: KeyError: <bytes32: 8cd563feced9fa172cd6049f43683a58d43d2b1e4f9e805207e266d434152a91>
Traceback (most recent call last):
  File "chia/util/log_exceptions.py", line 20, in log_exceptions
  File "chia/full_node/full_node.py", line 1129, in sync_from_fork_point
  File "chia/full_node/full_node.py", line 1098, in validate_block_batches
  File "chia/full_node/full_node.py", line 1259, in add_block_batch
  File "chia/full_node/weight_proof.py", line 233, in create_prev_sub_epoch_segments
  File "chia/consensus/blockchain.py", line 696, in height_to_block_record
  File "chia/consensus/blockchain.py", line 689, in block_record
KeyError: <bytes32: 8cd563feced9fa172cd6049f43683a58d43d2b1e4f9e805207e266d434152a91>

2023-10-25T20:45:27.710 full_node chia.full_node.full_node: INFO     long sync done
2023-10-25T20:45:27.710 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 4312716, weight 9061550064, hh f2ca765c3978ec9d95decb40471c52270a3a9f5cc5c55eb817f0f7ba56d44046, forked at 4312715, rh: a60553d8ab601ffe90f0a874ad113c1e9ec6e0d52706ca1d31a714cd3b2f59ff, total iters: 18295049948763, overflow: True, deficit: 16, difficulty: 2656, sub slot iters: 147849216, Generator size: No tx, Generator ref list size: No tx
2023-10-25T20:45:27.714 full_node chia.full_node.weight_proof: INFO     create weight proof peak f2ca765c3978ec9d95decb40471c52270a3a9f5cc5c55eb817f0f7ba56d44046 4312716
2023-10-25T20:45:33.492 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424045 timestamp: 1698281076
2023-10-25T20:45:33.494 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424045
2023-10-25T20:45:35.687 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424046 timestamp: 1698281076
2023-10-25T20:45:35.688 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424046
2023-10-25T20:45:37.431 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424047 timestamp: 1698281076
2023-10-25T20:45:37.432 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424047
2023-10-25T20:45:46.529 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424048 timestamp: 1698281110
2023-10-25T20:45:46.531 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424048
2023-10-25T20:45:59.521 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2023-10-25T20:45:59.522 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2023-10-25T20:45:59.522 full_node chia.full_node.full_node: INFO     Collected a total of 6 peaks.
2023-10-25T20:45:59.523 full_node chia.full_node.full_node: INFO     Selected peak Peak(header_hash=<bytes32: 85729bdb77b8ee9e22dd2664a39aacb3129a36c27bfd4614d9a7e35c662f29a8>, height=4424048, weight=9397495632)
2023-10-25T20:45:59.694 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-1382' coro=<FullNode._sync() done, defined at chia/full_node/full_node.py:903> exception=OperationalError('too many SQL variables')>
Traceback (most recent call last):
  File "chia/full_node/full_node.py", line 983, in _sync
  File "chia/full_node/full_node.py", line 1294, in _finish_sync
  File "chia/full_node/weight_proof.py", line 89, in get_proof_of_weight
  File "chia/full_node/weight_proof.py", line 117, in _create_proof_of_weight
  File "chia/full_node/weight_proof.py", line 193, in _get_recent_chain
  File "chia/consensus/blockchain.py", line 784, in get_header_blocks_in_range
  File "chia/full_node/block_store.py", line 388, in get_blocks_by_hash
  File "aiosqlite/context.py", line 39, in __aenter__
  File "aiosqlite/core.py", line 190, in execute
  File "aiosqlite/core.py", line 133, in _execute
  File "aiosqlite/core.py", line 106, in run
sqlite3.OperationalError: too many SQL variables
2023-10-25T20:46:00.368 full_node chia.full_node.full_node: INFO     Total of 6 peers with peak 4424048
2023-10-25T20:46:00.369 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 51.154.74.190 up to height 4424048
2023-10-25T20:46:00.517 full_node full_node_server        : INFO     Connected with full_node PeerInfo(_ip=IPv4Address('203.206.232.104'), _port=8444)
2023-10-25T20:46:00.519 full_node full_node_server        : INFO     Connection closed: 203.206.232.104, node id: b29a63702c6c19b092e38d5eeb9368bcaa732d73a16a6eb136ce258ffbee6cee
2023-10-25T20:46:00.519 full_node chia.full_node.full_node: INFO     peer disconnected PeerInfo(_ip=IPv4Address('203.206.232.104'), _port=8444)
2023-10-25T20:46:12.925 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
task: <Task pending name='Task-1618' coro=<FullNode.sync_from_fork_point.<locals>.fetch_block_batches() running at chia/full_node/full_node.py:1084> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at asyncio/tasks.py:720]>
2023-10-25T20:46:14.051 full_node chia.full_node.weight_proof: INFO     validating 11520 sub epochs
2023-10-25T20:46:14.093 full_node chia.full_node.weight_proof: INFO     validate weight proof peak height 4424048
2023-10-25T20:46:14.317 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424049 timestamp: 1698281153
2023-10-25T20:46:14.318 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4424049
2023-10-25T20:46:21.576 full_node full_node_server        : INFO     Cannot connect to host 86.125.61.34:8444 ssl:<ssl.SSLContext object at 0x7f358615d4c0> [Connect call failed ('86.125.61.34', 8444)]
2023-10-25T20:46:22.670 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4424050 timestamp: 1698281153
strikeoncmputrz commented 11 months ago

Original poster here. Does anyone have a work around so I can rejoin the network? I haven't been farming for over a week. This seems like a significant bug. There's nothing custom about my setup. I installed the deb package and have updated every time an update is available.

wjblanke commented 11 months ago

Wow thats some crazy logs. What version of linux and distribution are you running? The SQL error is usually due to an older sqlite version but our installer should have a high limit. very interested in figuring this out. The fastest workaround would be to download the torrent db and go from there. i think your db is just in an odd state. we'd love to debug it if u can provide it though. uploading it will be a drag. let us know.

mstevensm2 commented 11 months ago

Original poster here. Does anyone have a work around so I can rejoin the network? I haven't been farming for over a week. This seems like a significant bug. There's nothing custom about my setup. I installed the deb package and have updated every time an update is available.

I have the exact same issue in windows

mstevensm2 commented 11 months ago

Wow thats some crazy logs. What version of linux and distribution are you running? The SQL error is usually due to an older sqlite version but our installer should have a high limit. very interested in figuring this out. The fastest workaround would be to download the torrent db and go from there. i think your db is just in an odd state. we'd love to debug it if u can provide it though. uploading it will be a drag. let us know.

Mine did this when I downloaded your October 1 database torrent of version two because of the upcoming lack of support for version one soft fork

Elyasin commented 11 months ago

I believe that I have the same issue. I downloaded the October Snapshot DB (because I did not have enough space to convert mine from v1 to v2). The suggested workaround cannot be applied since this issue is with a snapshot db version.

In my opinion this is a serious bug. I cannot use Chia because of this. I regret having updated to 2.1.1. Lesson learnt.

strikeoncmputrz commented 11 months ago

Wow thats some crazy logs. What version of linux and distribution are you running? The SQL error is usually due to an older sqlite version but our installer should have a high limit. very interested in figuring this out. The fastest workaround would be to download the torrent db and go from there. i think your db is just in an odd state. we'd love to debug it if u can provide it though. uploading it will be a drag. let us know.

Thanks. I ended up redownloading the db via torrent, purging and re-installing chia via apt, and the waiting for sync. I was down for days before I went nuclear but I didn't want to wait as noone proposed a solution.

wjblanke commented 11 months ago

please open another issue since the original poster has resolved with a workaround if u have issues

mstevensm2 commented 11 months ago

How can you close the issue when the issue has not been resolved? The issue is still unresolved. I will not reopen a new issue and you will see that many other people are experiencing this as well because we all are supposed to upgrade before the soft fork. Do you actually believe it is just three people who have this problem?

mstevensm2 commented 11 months ago

This is a totally unacceptable way to close an unresolved issue

On Wed, Nov 8, 2023 at 11:08 AM wjblanke @.***> wrote:

Closed #16697 https://github.com/Chia-Network/chia-blockchain/issues/16697 as completed.

— Reply to this email directly, view it on GitHub https://github.com/Chia-Network/chia-blockchain/issues/16697#event-10903639189, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUQ2TP6HYERTFAAP3SAQPOTYDO4BLAVCNFSM6AAAAAA6QI4YPWVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJQHEYDGNRTHEYTQOI . You are receiving this because you commented.Message ID: @.*** com>

mstevensm2 commented 11 months ago

I got help on Discord from clyd3wallace22 , and here is the solution it seems to be working: Hey, can you try:

  1. Stop chia
  2. Delete the height-to-hash and sub-epoch-summary files in the wallet DB folder ~\.chia\mainnet\db\. NOTE- also make sure the DB file is still the same size as when you moved it there. If it is not let me know as there are some more steps.
  3. Start chia and give it upwards of 30 minutes to start services and begin syncing
Elyasin commented 11 months ago

Maybe the deletion of the height-to-hash and sub-epoch files is what I was missing. This information should be communicated more clearly. It's too late to find out now.

I decided to sync from scratch since I did not trust much what was mentioned as workaround/solution out there. Most people seem to have decided to sync from scratch is my impression.

Regarding closing this issue, I agree with mstevensm2 - the problem is not resolved. It's ok if you want to do that, but it is not living up to a high standard. Workarounds do not solve the source of the problem.