search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[Bug] uncaught KeyError exception in block_record #9677

Closed durdin85 closed 2 years ago

durdin85 commented 2 years ago

What happened?

I am not sure if this would be possible to reproduce, but I was running chia on raspberry (4GB RAM) for a like half a year, when in October the entire FS crashed, and I had to reinstall and unfortunately sync from scratch. After like 2 months of checking and regular restarts of chia daemon it got into near sync state of blockchain (wallet was full sync, but chain last day missing, this was somewhere between 21-22nd December). But when I've checked it after Christmas, the log is full of this:

2021-12-27T13:59:13.294 full_node full_node_server        : WARNING  Banning 99.152.32.71 for 10 seconds
2021-12-27T13:59:13.296 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '99.152.32.71', 'port': 8444}
2021-12-27T13:59:13.298 full_node full_node_server        : ERROR    Exception b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2', exception Stack: Traceback (most recent call last):
  File "chia/server/server.py", line 437, in start_client
  File "chia/server/server.py", line 335, in connection_added
  File "chia/full_node/full_node.py", line 619, in on_connect
  File "chia/full_node/full_node.py", line 585, in synced
  File "chia/consensus/blockchain.py", line 157, in get_peak
  File "chia/consensus/blockchain.py", line 674, in height_to_block_record
  File "chia/consensus/blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'

2021-12-27T13:59:13.301 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-291378' coro=<RpcServer._state_changed() done, defined at chia/rpc/rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia/rpc/rpc_server.py", line 51, in _state_changed
  File "chia/rpc/full_node_rpc_api.py", line 61, in _state_changed
  File "chia/rpc/full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia/consensus/blockchain.py", line 157, in get_peak
  File "chia/consensus/blockchain.py", line 674, in height_to_block_record
  File "chia/consensus/blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-27T13:59:14.603 full_node full_node_server        : INFO     Cannot connect to host 112.25.210.130:8444 ssl:<ssl.SSLContext object at 0xffff377097c0> [Connect call failed ('112.25.210.130', 8444)]

running eg. chia show -s results in similar message:

Exception from 'show' Traceback (most recent call last):
  File "chia/cmds/show.py", line 41, in show_async
  File "chia/rpc/full_node_rpc_client.py", line 27, in get_blockchain_state
  File "chia/rpc/rpc_client.py", line 49, in fetch
ValueError: {'error': "b'\\xd7\\x80\\xd2,z\\x87\\xc9\\xe0\\x1d\\x98\\xb4\\x9a\\t\\x10\\xf6p\\x1c;\\x95\\x01WA1k?\\xda\\x04.]{\\x81\\xd2'", 'success': False}

but chia wallet show is ok, so seems only main daemon is somehow stuck:

chia wallet show
Wallet height: 1338206
Sync status: Synced
...

Version

1.2.11

What platform are you using?

Pi4

What ui mode are you using?

CLI

Relevant log output

No response

gigilio commented 2 years ago

Screenshot_20211228-061915_AnyDesk I have the same problem...

PoolPirate commented 2 years ago

Same problem, same header hash. Looking at the db I found this:

The block with is_peak set to true has height 0. The header has also is the header hash from the error message:

EDIT: The block with height 0 is actually correct, it's the block with height 0. It's just the is_peak not being correct.

I have manually set is_peak to true on the block with the highest height and removed it on block 0, let's see if that helps

EDIT: With that change in the database the sync continued to work, the node is fully synced now

durdin85 commented 2 years ago

Same problem, same header hash. Looking at the db I found this: ... EDIT: With that change in the database the sync continued to work, the node is fully synced now

I've just checked my node, and you are right, the block 0 is actually set as peak: 

sqlite> select header_hash, prev_hash, height, is_peak, is_block from block_records where is_peak == 1 and is_block == 1;

header_hash|prev_hash|height|is_peak|is_block
d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2|ccd5bb71183532bff220ba46c268991a3ff07eb358e8255a65c30a2dce0e5fbb|0|1|1

sqlite> select header_hash, prev_hash, height, is_peak, is_block from block_records where height == (select max(height) from block_records);

header_hash|prev_hash|height|is_peak|is_block
3c61c9e9730722c45f738cd5cb3e3203428683679e966cc52dbfc78a623e69d5|356586bc3f5fa2e3671a1498e8d875e7892a59dccb40cd5cfdd369329132d561|1313895|0|1

So I've tried swapping peak on block 0 vs last block as suggested:

sqlite> update block_records set is_peak = 1 where height == (select max(height) from block_records);
sqlite> update block_records set is_peak = 0 where height == 0;
sqlite> select header_hash, prev_hash, height, is_peak, is_block from block_records where is_peak == 1 and is_block == 1;

header_hash|prev_hash|height|is_peak|is_block
3c61c9e9730722c45f738cd5cb3e3203428683679e966cc52dbfc78a623e69d5|356586bc3f5fa2e3671a1498e8d875e7892a59dccb40cd5cfdd369329132d561|1313895|1|1

and yes, it solves the problem, chia is again syncing. Question still remains how this could happen? Some remote trigger? ( / attacker?) Client bug? Protocol bug?

dcow commented 2 years ago

I had the same symptoms happen: block 0 got is_peak set to 1 while my highest block (1307016) had is_peak 0. My error message was different, pointing to some failed uniqueness constraint rather than a key error...

2021-12-29T08:35:05.592 full_node chia.full_node.full_node: INFO     Start syncing from fork point at 0 up to 1347496
2021-12-29T08:35:07.661 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name

Manually fixing the records in the DB using @durdin85's example also works in my case and the node is syncing again.

mariano54 commented 2 years ago

I'm looking at the recent changes in the way we store block records in memory, I think this might have something to do with it.

mariano54 commented 2 years ago

I think the current peak somehow gets set to None. Then the node tries to start syncing, gets the first block, adds it, and now the blockchain peak is block 0, but the coin store will still have the real coins. That would explain "UNIQUE constraint failed: coin_record.coin_name".

Maybe the cause is related to BlockHeightMap not having the correct value for height 0, or maybe for the final block?

mariano54 commented 2 years ago

The only way to set the peak to 0 is by this line getting executed: https://github.com/Chia-Network/chia-blockchain/blob/main/chia/consensus/blockchain.py#L367 This means that get_peak returned None, which means that somehow when initializing we could not get the peak from the DB (nothing was set as the peak).

This seems really weird, as we update the peak atomically by unsetting the previous one and setting the new one

mariano54 commented 2 years ago

What version of the software are you all running? Main or 1.2.11? 1.2.11 was released on Nov 4th and does not include most of the recent block_store changes. Maybe it's related to this? https://github.com/Chia-Network/chia-blockchain/commit/e4b9db08cd53e8a03b2dfb5abe55242795e424f8#diff-93a564af2275d87306b87d322d695e40d1c11b0628606e56954c4c207ecead47 @arvidn

zEdS15B3GCwq commented 2 years ago

Having the same issue here. On latest as per install instructions (git clone with -b latest on Ubuntu).

In my case, the db row with max(height) did NOT have is_block == 1. I'm assuming I should not set it to be the peak, instead, set peak to the row that has max(height) where is_block == 1, right? Something like this: update block_records set is_peak = 1 where height == (select max(height) from block_records where is_block == 1);.

mariano54 commented 2 years ago

Which issue are you having @zEdS15B3GCwq , can you post the error message? You should not change the peak manually, that is pretty dangerous. You might be missing some coins and your node will experience consensus issues in the future. You can try, but if it fails again, you probably need to re sync or start from a backup

mariano54 commented 2 years ago

from dcow: Interestingly, I have two entries for the block before the last block in my db:

66220d00919a373fb7f27b0583f5f52a1e3f89208662e48af1cd9edfcfe0b52a|26a480a84cd73256d4ddbaabf1001a30effbff9d4acaf502247ec2f4fa08c467|1307015|0|1
84b8b671b838e6ac2eb7d8db79fbfd454385b3e6623548bdcd11c7d6f3e2f39c|26a480a84cd73256d4ddbaabf1001a30effbff9d4acaf502247ec2f4fa08c467|1307015|0|1
zEdS15B3GCwq commented 2 years ago

Which issue are you having @zEdS15B3GCwq , can you post the error message? You should not change the peak manually, that is pretty dangerous. You might be missing some coins and your node will experience consensus issues in the future. You can try, but if it fails again, you probably need to re sync or start from a backup

Pretty much as in OP. chia show -s exits with an error, logfile shows below error multiple times.

2021-12-24T17:34:04.440 full_node full_node_server        : ERROR    Exception b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA>
  File "/home/t/chia-blockchain/chia/server/server.py", line 437, in start_client
    await self.connection_added(connection, on_connect)
  File "/home/t/chia-blockchain/chia/server/server.py", line 335, in connection_added
    await on_connect(connection)
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 619, in on_connect
    synced = await self.synced()
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 585, in synced
    curr: Optional[BlockRecord] = self.blockchain.get_peak()
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 157, in get_peak
    return self.height_to_block_record(self._peak_height)
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 674, in height_to_block_record
    return self.block_record(header_hash)
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 670, in block_record
    return self.__block_records[header_hash]
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'

Also, similarly to the posts looking at the db, the block with height 0 has is_peak==1. In my case, the block with height==max(height) did NOT have is_block==1, the meaning of which I don't know.

I did try to move the peak to the row with max(height), and also to the highest row that still had is_block==1, but both didn't work. Syncing was stuck at the same block ID and the logs repeatedly showed this error:

2022-01-06T01:52:44.377 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed

So you're saying that I might need to sync from zero, but as chia is now crashing and not syncing at all, unless you give me another way to fix it, starting from zero seems to be the only option.

Edit: I wrote that I was on latest, but that was incorrect. I noticed the issue after my latest fetch yesterday but the issue occurred prior to that so I did have several missing commits.

mariano54 commented 2 years ago

Thank you for the information, fixing this issue is my #1 priority right now. Do you happen to have the logs before this error happened? I want to find out how this database sets the peak to height 0.

mariano54 commented 2 years ago

Does anyone else have the log file for the first time the error happened? If so please send it to me on keybase (@sorgente711)

dcow commented 2 years ago

@mariano54 just spitballing haven't looked deeply at the code, maybe a spurious read error on the DB query/transaction that looks for the peak that would causing the node to think there is none and then initialize the mempool incorrectly? Although my node was running for a long time I don't think I restarted it or anything but I can't recall for sure.

dcow commented 2 years ago

Looking at my logs, my node had just restarted twice immediately prior to seeing the first UNIQUE constraint ERROR. This is the first one:

2021-12-29T08:07:40.968 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-29T08:07:41.121 full_node chia.full_node.full_node: INFO     long sync done
2021-12-29T08:07:41.123 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 0, weight 7, hh d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2, forked at 0, rh: bdde7b5b2bc6025c07a9f5233d8eae167bea654146b272652262b362524c3e85, total iters: 11798946, overflow: False, deficit: 15, difficulty: 7, sub slot iters: 134217728, Generator size: No tx, Generator ref list size: No tx
2021-12-29T08:07:41.124 full_node chia.full_node.mempool_manager: INFO     Size of mempool: 0 spends, cost: 0 minimum fee to get in: 0

The surrounding logs mostly involve finding plots (so lots of I/O). Here's a restart from just before the error where the node says it's initialized an empty chain:

2021-12-29T08:03:11.229 daemon chia.daemon.server         : INFO     Starting Daemon Server
2021-12-29T08:03:11.232 daemon chia.daemon.server         : INFO     Waiting Daemon WebSocketServer closure
2021-12-29T08:03:13.782 harvester harvester               : INFO     Started harvester service on network_id: mainnet
2021-12-29T08:03:13.782 harvester harvester               : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8447}
2021-12-29T08:03:13.786 harvester harvester_server        : INFO     Cannot connect to host 127.0.0.1:8447 ssl:<ssl.SSLContext object at 0x7f8
dfff62940> [Connect call failed ('127.0.0.1', 8447)]
2021-12-29T08:03:13.792 daemon chia.daemon.server         : INFO     Register service {'service': 'chia_harvester'}
2021-12-29T08:03:13.792 daemon chia.daemon.server         : INFO     registered for service chia_harvester
2021-12-29T08:03:13.792 daemon chia.daemon.server         : INFO     {'success': True}
2021-12-29T08:03:13.996 full_node chia.full_node.full_node: INFO     Initializing blockchain from disk
2021-12-29T08:03:13.998 full_node chia.consensus.blockchain: INFO     Started 59 processes for block validation
2021-12-29T08:03:13.999 full_node chia.full_node.full_node: INFO     Initialized with empty blockchain time taken: 0s
2021-12-29T08:03:14.003 full_node full_node_server        : INFO     Started listening on port: 8444
2021-12-29T08:03:14.003 full_node full_node               : INFO     Started full_node service on network_id: mainnet
2021-12-29T08:03:14.013 daemon chia.daemon.server         : INFO     Register service {'service': 'chia_full_node'}
2021-12-29T08:03:14.013 daemon chia.daemon.server         : INFO     registered for service chia_full_node
2021-12-29T08:03:14.013 daemon chia.daemon.server         : INFO     {'success': True}
2021-12-29T08:03:15.032 farmer farmer_server              : INFO     Started listening on port: 8447
2021-12-29T08:03:15.033 farmer farmer                     : INFO     Started farmer service on network_id: mainnet
mariano54 commented 2 years ago

Just to confirm, this was before the issues started happening right? The node was synching fine before this error?

mariano54 commented 2 years ago

If peak is set to height 0, the UNIQUE constraint makes sense because the full node will have no blocks but still will have all the coins. So when it tries to add the first coins it wlil fail

mariano54 commented 2 years ago

Another interesting point is the log message "Initialized with empty blockchain time taken". This indicates that there was no row with is_peak=1. If there was, then you would see initialized to height 0 or something. So it seems like some issue caused the is_peak to be set to 0 for all blocks.

zEdS15B3GCwq commented 2 years ago

Got the log files, found the one back from the end of Dec where it started showing. Didn't notice then as I forgot about checking on chia around Xmas; tried to start up again yesterday; fail. How can I share it with you, if not on keybase? Any way to attach a 890k zip file here?

A few things that popped up in the log:

First occurrence of a database malformed error. Before that, nothing for hours that seems relevant (many block validation time warnings; failed to fetch block exceptions but the last one came 6+ hours prior to the error below):

2021-12-18T13:09:52.569 full_node full_node_server        : ERROR    Exception: database disk image is malformed, {'host': '121.99.176.158', 'port': 8444}. Traceback (most recent call last):
  File "/home/t/chia-blockchain/chia/server/server.py", line 566, in wrapped_coroutine
    result = await coroutine
  File "/home/t/chia-blockchain/chia/full_node/full_node_api.py", line 124, in new_peak
    return await self.full_node.new_peak(request, peer)
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 509, in new_peak
    if await self.short_sync_backtrack(
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 443, in short_sync_backtrack
    raise e
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 440, in short_sync_backtrack
    await self.respond_block(response, peer)
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 1440, in respond_block
    added, error_code, fork_height, coin_changes = await self.blockchain.receive_block(
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 274, in receive_block
    fork_height, peak_height, records, (coin_record_change, hint_changes) = await self._reconsider_peak(
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 414, in _reconsider_peak
    added_rec = await self.coin_store.new_block(
  File "/home/t/chia-blockchain/chia/full_node/coin_store.py", line 115, in new_block
    await self._add_coin_records(additions)
  File "/home/t/chia-blockchain/chia/full_node/coin_store.py", line 415, in _add_coin_records
    cursor = await self.coin_record_db.executemany(
  File "/home/t/chia-blockchain/venv/lib/python3.9/site-packages/aiosqlite/core.py", line 210, in executemany
    cursor = await self._execute(self._conn.executemany, sql, parameters)
  File "/home/t/chia-blockchain/venv/lib/python3.9/site-packages/aiosqlite/core.py", line 129, in _execute
    return await future
  File "/home/t/chia-blockchain/venv/lib/python3.9/site-packages/aiosqlite/core.py", line 102, in run
    result = function()
sqlite3.DatabaseError: database disk image is malformed

A bit later started to get a large number of these errors (but the previous exceptions disappeared):

2021-12-18T15:01:17.366 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed

Then something else (also in large number), and again, previous errors stopped:

2021-12-23T17:12:55.252 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name

Later, the UNIQUE errors stopped for one similar in the OP. Pasting here with a little context:

2021-12-24T03:02:06.259 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: 195.211.160.112

2021-12-24T03:06:23.959 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:09:46.607 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:13:31.659 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:17:48.257 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:21:41.456 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:25:28.737 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:28:54.235 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:33:00.918 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:36:26.547 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:40:08.941 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:43:41.348 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:46:55.021 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:50:52.770 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:54:29.271 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T03:58:57.639 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:02:27.066 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:05:46.878 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:09:21.303 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:12:45.871 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:13:24.336 full_node chia.full_node.full_node: WARNING  Received unsolicited/late block from peer {'host': '116.62.192.206', 'port': 8444}
2021-12-24T04:16:47.275 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:20:24.733 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:24:24.964 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:28:12.734 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:32:47.892 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:36:19.940 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:39:46.532 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:43:16.271 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:46:48.448 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:50:17.702 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:54:04.808 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T04:57:55.688 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:01:35.348 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:05:26.020 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:09:34.129 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:16:29.645 full_node full_node_server        : WARNING  Banning [2806:2f0:1060:1b41:49e3:2378:31a8:3992] for 600 seconds
2021-12-24T05:16:29.648 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: [2806:2f0:1060:1b41:49e3:2378:31a8:3992]

2021-12-24T05:19:51.073 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:23:33.517 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:27:39.484 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:31:28.933 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:35:49.895 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:40:04.924 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:43:32.791 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:47:31.172 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:51:07.645 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:54:29.882 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T05:58:23.951 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T06:02:00.251 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T06:05:27.845 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2021-12-24T06:05:47.506 full_node chia.full_node.full_node: WARNING  Received unsolicited/late block from peer {'host': '91.117.219.126', 'port': 8444}
2021-12-24T06:05:55.522 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: 96.250.85.237

2021-12-24T06:21:35.704 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T06:21:35.713 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-267' coro=<FullNode._sync() done, defined at /home/t/chia-blockchain/chia/full_node/full_node.py:716> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 847, in _sync
    await self._finish_sync()
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 1081, in _finish_sync
    peak: Optional[BlockRecord] = self.blockchain.get_peak()
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 157, in get_peak
    return self.height_to_block_record(self._peak_height)
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 674, in height_to_block_record
    return self.block_record(header_hash)
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 670, in block_record
    return self.__block_records[header_hash]
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T06:21:35.847 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-3241' coro=<RpcServer._state_changed() done, defined at /home/t/chia-blockchain/chia/rpc/rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "/home/t/chia-blockchain/chia/rpc/rpc_server.py", line 51, in _state_changed
    payloads: List[Dict] = await self.rpc_api._state_changed(*args)
  File "/home/t/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 61, in _state_changed
    data = await self.get_blockchain_state({})
  File "/home/t/chia-blockchain/chia/rpc/full_node_rpc_api.py", line 102, in get_blockchain_state
    peak: Optional[BlockRecord] = self.service.blockchain.get_peak()
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 157, in get_peak
    return self.height_to_block_record(self._peak_height)
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 674, in height_to_block_record
    return self.block_record(header_hash)
  File "/home/t/chia-blockchain/chia/consensus/blockchain.py", line 670, in block_record
    return self.__block_records[header_hash]
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2

This error continued then on and on for 6 more log files. I'd be happy to share them just reluctant to create yet another new account on keybase.

shared my log file a few messages below

zEdS15B3GCwq commented 2 years ago

I can't find any "Initialized with empty blockchain time taken" lines in the logs, perhaps because my chia was running continuously through the evolution of this error?

mariano54 commented 2 years ago

Yeah if you didn't restart you would not see the empty blockchian issue. Did you ever run out of disk space before these issues started happening?

zEdS15B3GCwq commented 2 years ago

Yeah if you didn't restart you would not see the empty blockchian issue. Did you ever run out of disk space before these issues started happening?

nope ... that is, never noticed it. Had around 60 GB free on the flash, so unless the db was copied several times, don't think the disk space could've been an issue.

Update: FYI I added a mega link to the log file at the end of my long message above; just in case you didn't notice the edit

mariano54 commented 2 years ago

Disk image malformed started on this date: 2021-12-18T13:09:52.569 Did you have a power loss or restart the node somehow on that day?

zEdS15B3GCwq commented 2 years ago

Looking more carefully at the log file, even the first error is related to the new_peak routine. Might be relevant?

2021-10-18T00:30:16.889 full_node full_node_server        : WARNING  Banning 89.219.17.9 for 10 seconds
2021-10-18T00:30:16.905 full_node full_node_server        : ERROR    Exception: Failed to fetch block 1010969 from {'host': '186.138.64.95', 'port': 8444}, timed out, {'host': '186.138.64.95', 'port': 8444}. Traceback (most recent call last):
  File "/home/t/chia-blockchain/chia/server/server.py", line 553, in wrapped_coroutine
    result = await coroutine
  File "/home/t/chia-blockchain/chia/full_node/full_node_api.py", line 106, in new_peak
    return await self.full_node.new_peak(request, peer)
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 398, in new_peak
    if await self.short_sync_backtrack(
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 332, in short_sync_backtrack
    raise e
  File "/home/t/chia-blockchain/chia/full_node/full_node.py", line 317, in short_sync_backtrack
    raise ValueError(f"Failed to fetch block {curr_height} from {peer.get_peer_logging()}, timed out")
ValueError: Failed to fetch block 1010969 from {'host': '186.138.64.95', 'port': 8444}, timed out

Sharing older logfile - see a few messages below

dcow commented 2 years ago

@zEdS15B3GCwq FYI GitHub lets you attach .zip/gz and .log files to an issue (among others). Just drag and drop or click the Attach files ... panel under the editor.

mariano54 commented 2 years ago

This is a nomal error, it happens sometimes. The real issue is the database malformed. Before then, everything seems fine. You did restart the node at some point betwen Dec 18th and Dec 24th right?

zEdS15B3GCwq commented 2 years ago

Disk image malformed started on this date: 2021-12-18T13:09:52.569 Did you have a power loss or restart the node somehow on that day?

Probably no restart or power loss. Not that I remember that particular day, but the log file has entries pretty much consistently throughout the day, which you can also see. Check my logs to see if you can spot other signs of a possible restart. On the other hand, the network the Pi which drives the chia is on is quite unreliable, but it's been dropping in and out for months now without db issues.

zEdS15B3GCwq commented 2 years ago

@zEdS15B3GCwq FYI GitHub lets you attach .zip/gz and .log files to an issue (among others). Just drag and drop or click the Attach files ... panel under the editor.

okies, thanks debuglog7.zip

debuglog6.zip

zEdS15B3GCwq commented 2 years ago

This is a nomal error, it happens sometimes. The real issue is the database malformed. Before then, everything seems fine. You did restart the node at some point betwen Dec 18th and Dec 24th right?

Hmm, maybe. That part is kind of blurry. I don't remember checking on it until yesterday.

Update: actually, I don't think it was restarted, at least not based on what I can see in the log - lots of entries spread out between Dec 18 and 24. Shouldn't be there some kind of log message specific to a restart?

When I checked on it yesterday, chia was not running. I can check the system logs about a reboot, but I don't think I touched the Pi around those days.

mariano54 commented 2 years ago

Thanks for the help. I will try to fix the issue but I don't think i can recover your DB to a good state. You will have to resync or grab the DB from someone else.

zEdS15B3GCwq commented 2 years ago

Thanks for the help. I will try to fix the issue but I don't think i can recover your DB to a good state. You will have to resync or grab the DB from someone else.

OK, no problem, hope you can fix this. I'm wondering though how others (posts above) seem to have been able to fix their DBs? What was the difference?

mariano54 commented 2 years ago

I'm still looking into that. However your database might be more messed up than the other users on this issue.

mariano54 commented 2 years ago

@durdin85, @poolpirate, @gigilio do you have the log file of when the issue started occurring for you?

dcow commented 2 years ago

Thanks for the help. I will try to fix the issue but I don't think i can recover your DB to a good state. You will have to resync or grab the DB from someone else.

It seems like a command that re-verifies (or re-builds) the (or a new) chain (optionally from a specified height) first consulting the existing local DB for each block and then pulling blocks from peers as needed when the local record is corrupt or incorrect would be rather useful.

durdin85 commented 2 years ago

@durdin85, @PoolPirate, @gigilio do you have the log file of when the issue started occurring for you?

Unfortunately no, I forgot to save them and they are gone. I have only logs from the time it was already happening for couple of days, but not before or on the first occurrence.

arvidn commented 2 years ago

did anyone experiencing this problem change the db_sync setting?

arvidn commented 2 years ago

these patches aim to help track this down:

https://github.com/Chia-Network/chia-blockchain/pull/9780 https://github.com/Chia-Network/chia-blockchain/pull/9781

PoolPirate commented 2 years ago

@durdin85, @PoolPirate, @gigilio do you have the log file of when the issue started occurring for you?

Sorry for the late reply, did not check my messages in a while.

I actually do still have the logs from back then. https://s.playwo.de/upload/data/find/2bBiK

I narrowed it down to the probably relevant section. Below that it just continues to spam the same error message over and over.

barneyjackson commented 2 years ago

@mariano54 if it helps I did have a power outage and then started experiencing this issue (although it took me a few days to notice...). I don't have the logs I'm afraid.

Now syncing again thanks to @durdin85 's db fixing SQL.

jeffj55374 commented 2 years ago

@mariano54 - This happened to me on Jan 18th. I have the log files logging at INFO and I have the DB.

I've attached the logs. debug.log.2022.01.18.gz

I believe things started to go badly around 2022-01-18T18:01:46.273 I've filtered out the full_node logs that I think might be helpful below

Version 1.2.11 What platform are you using? Pi4 - 8GB, plenty of disk space, Ubuntu (venv) ubuntu@chia-1:~/chia-blockchain$ uname -a Linux chia-1 5.11.0-1027-raspi #30-Ubuntu SMP PREEMPT Thu Jan 13 12:52:48 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

What ui mode are you using? CLI

My Experience January 18, 2022 Things were running fine when I noticed that the partials submitted by my farmer had declined. I noticed the issue about 3 hours after the problem likely started. (No power failure, no restarts, etc.)

Shortly after the harvester stopped finding partials, the full_node logged

2022-01-18T18:01:46.273 full_node chia.full_node.full_node: INFO     Added unfinished_block 665cfacb30b2ea637a3bd22d1f7dde4bea04fab35bd8bef053d4e335f2676bc4, not farmed by us, SP: 6 farmer re
sponse time: 9.8785, Pool pk xch1f0ryxk6qn096hefcwrdwpuph2hm24w69jnzezhkfswk0z2jar7aq5zzpfj, validation time: 0.4313 seconds, pre_validation time 0.6284, cost: 3260441268, percent full: 29.64
%
2022-01-18T18:01:52.683 full_node chia.full_node.full_node_store: INFO     Don't have rc hash b731a1fce84db8ba2779f55c5531781c211668ccf55f3c276a3fdfc92d004fed. caching signage point 12.
2022-01-18T18:01:54.854 full_node full_node_server        : INFO     Connection closed: 80.7.129.72, node id: b9ff8f2545a9d7d2787a2b0cfe0bc2cde2baf4d4cf4d6b0964c5ddd3be8367ce
2022-01-18T18:01:54.855 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '80.7.129.72', 'port': 8444}

2022-01-18T18:02:01.051 full_node full_node_server        : WARNING  Cannot write to closing transport 76.68.80.193

2022-01-18T18:02:01.052 full_node full_node_server        : INFO     Connection closed: 76.68.80.193, node id: e852c7baca90662f81240673fc8caaf360191a741030993116bef642796d90cc
2022-01-18T18:02:01.053 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '76.68.80.193', 'port': 8444}
2022-01-18T18:02:02.614 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 605d578392e4b7a54ec169c8fe38a004db9da29e641fcb0a46e462dcf230446c. caching signage point 13.
2022-01-18T18:02:12.249 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 9c373eec48549043e8f3320b22ff4ecd4ccee5ae33f7014ab93319db684930d9. caching signage point 14.
2022-01-18T18:02:21.884 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8fd247b679b0405ddab6e1ed1b21be9609ca44c8e49891ed5668fad5ed7f1fa5. caching signage point 15.
2022-01-18T18:02:28.513 full_node full_node_server        : INFO     Connection closed: 94.182.190.244, node id: e91edb1c2dfc3a2ab5c52e03932cb7d6c3f4c4f7ceece19971e8ba3aeeb6ac16
2022-01-18T18:02:28.515 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '94.182.190.244', 'port': 8444}
2022-01-18T18:02:33.812 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 16.
2022-01-18T18:02:42.179 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 17.
2022-01-18T18:02:51.073 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 18.
2022-01-18T18:02:59.754 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 19.
2022-01-18T18:03:08.467 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 20.
2022-01-18T18:03:17.217 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 21.
2022-01-18T18:03:25.903 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 22.
2022-01-18T18:03:34.564 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 23.
2022-01-18T18:03:43.730 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e817e77362522888e9927a8a307ea629d48138af47b941bbd53fe0e6298354a1. caching signage point 24.
2022-01-18T18:03:54.880 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 2d82b587a81f01fd7a66063c126aeee3030984f85f777e077dc0aa04e6c22852. caching signage point 25.
2022-01-18T18:04:03.478 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 2d82b587a81f01fd7a66063c126aeee3030984f85f777e077dc0aa04e6c22852. caching signage point 26.
2022-01-18T18:04:12.201 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 2d82b587a81f01fd7a66063c126aeee3030984f85f777e077dc0aa04e6c22852. caching signage point 27.
2022-01-18T18:04:23.352 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 399f9e00649ec744e23f1ae6812243b102064b9929ee9ad510c0950cf8ea7da9. caching signage point 28.
2022-01-18T18:04:31.739 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 399f9e00649ec744e23f1ae6812243b102064b9929ee9ad510c0950cf8ea7da9. caching signage point 29.
2022-01-18T18:04:38.783 full_node full_node_server        : INFO     Connection closed: 184.105.247.154, node id: 6e864025c410b8020c52efd65b2d80170c6716afa0b999bc1be2d28b76fbf167
2022-01-18T18:04:38.785 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '184.105.247.154', 'port': 8444}
2022-01-18T18:04:40.361 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 399f9e00649ec744e23f1ae6812243b102064b9929ee9ad510c0950cf8ea7da9. caching signage point 30.
2022-01-18T18:04:49.072 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 399f9e00649ec744e23f1ae6812243b102064b9929ee9ad510c0950cf8ea7da9. caching signage point 31.
2022-01-18T18:04:58.028 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 399f9e00649ec744e23f1ae6812243b102064b9929ee9ad510c0950cf8ea7da9. caching signage point 32.
2022-01-18T18:05:07.834 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 3528d4fd79b6a5f1a53d2797b4a9fa5adb148c370dc32e9a03a691b521dc5dfb. caching signage point 33.
2022-01-18T18:05:16.437 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 3528d4fd79b6a5f1a53d2797b4a9fa5adb148c370dc32e9a03a691b521dc5dfb. caching signage point 34.
2022-01-18T18:05:26.342 full_node chia.full_node.full_node_store: INFO     Don't have rc hash c94cd8e011f0f896fecafcedb3cd363ef8fe857543d318a766b8887d9798be1e. caching signage point 35.
2022-01-18T18:05:34.988 full_node chia.full_node.full_node_store: INFO     Don't have rc hash c94cd8e011f0f896fecafcedb3cd363ef8fe857543d318a766b8887d9798be1e. caching signage point 36.
2022-01-18T18:05:43.720 full_node chia.full_node.full_node_store: INFO     Don't have rc hash c94cd8e011f0f896fecafcedb3cd363ef8fe857543d318a766b8887d9798be1e. caching signage point 37.
2022-01-18T18:05:52.516 full_node chia.full_node.full_node_store: INFO     Don't have rc hash c94cd8e011f0f896fecafcedb3cd363ef8fe857543d318a766b8887d9798be1e. caching signage point 38.
2022-01-18T18:05:53.689 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '92.255.210.125', 'port': 8444}.Inbound limit reached.
2022-01-18T18:05:53.885 full_node full_node_server        : INFO     Connection closed: 92.255.210.125, node id: c12e672c760024049c0391f3643e4381d5beeabbba9cbfe6654f658833768836
2022-01-18T18:05:53.887 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '92.255.210.125', 'port': 8444}
2022-01-18T18:06:02.354 full_node chia.full_node.full_node_store: INFO     Don't have rc hash a2e08d63893098c85b080c434e4af307512495d531cf5563e0ee7f892328ae0f. caching signage point 39.
2022-01-18T18:06:09.497 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '182.253.178.65', 'port': 8444}.Inbound limit reached.
2022-01-18T18:06:09.718 full_node full_node_server        : INFO     Connection closed: 182.253.178.65, node id: 320675706e8c2231cf14773d0138d0c846c8146392d4bf9a839db7754c02a5cb
2022-01-18T18:06:09.719 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '182.253.178.65', 'port': 8444}
2022-01-18T18:06:12.495 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 080441ba2ad98c41efde992f604aa836c2c2692741f3f4a34d30a8f1f3cac8a9. caching signage point 40.
2022-01-18T18:06:26.057 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d3c1846bd7c940b3842cc9f061977f54035bd71c1f934fca1dd3759ac0e993d8. caching signage point 41.
2022-01-18T18:06:31.207 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d3c1846bd7c940b3842cc9f061977f54035bd71c1f934fca1dd3759ac0e993d8. caching signage point 42.
2022-01-18T18:06:34.462 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '31.223.79.11', 'port': 8444}.Inbound limit reached.
2022-01-18T18:06:34.676 full_node full_node_server        : INFO     Connection closed: 31.223.79.11, node id: 69c02974081f9892c5736aeb2e05a6100cd03da9e6f87823ae2cab1bd73904c3
2022-01-18T18:06:34.678 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '31.223.79.11', 'port': 8444}
2022-01-18T18:06:39.984 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d3c1846bd7c940b3842cc9f061977f54035bd71c1f934fca1dd3759ac0e993d8. caching signage point 43.
2022-01-18T18:06:48.785 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d3c1846bd7c940b3842cc9f061977f54035bd71c1f934fca1dd3759ac0e993d8. caching signage point 44.
2022-01-18T18:06:49.177 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '91.23.124.222', 'port': 8444}.Inbound limit reached.
2022-01-18T18:06:49.356 full_node full_node_server        : INFO     Connection closed: 91.23.124.222, node id: e3d9da4044c1762ffc2a0a843bdfc7780943278b46e321582504833df323534c
2022-01-18T18:06:49.357 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '91.23.124.222', 'port': 8444}
2022-01-18T18:06:52.360 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '157.25.176.70', 'port': 8444}.Inbound limit reached.
2022-01-18T18:06:52.562 full_node full_node_server        : INFO     Connection closed: 157.25.176.70, node id: 815a3fe989ccf3a95a1d02cd7b00caf191b9ab527bd4c4f15a9e5a424fbd2a68
2022-01-18T18:06:52.564 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '157.25.176.70', 'port': 8444}
2022-01-18T18:06:57.495 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d3c1846bd7c940b3842cc9f061977f54035bd71c1f934fca1dd3759ac0e993d8. caching signage point 45.
2022-01-18T18:07:07.768 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 456c5fdfb9d024b0593ba53af1776cac9f7cd5b296621aaa48b22da544ac23f3. caching signage point 46.
2022-01-18T18:07:12.407 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '84.92.90.205', 'port': 8444}.Inbound limit reached.
2022-01-18T18:07:12.563 full_node full_node_server        : INFO     Connection closed: 84.92.90.205, node id: 7b5c615c94e8b90eb5a15607c869dcf6c2a0bcf079115ce1a03dac16d8fd2a44
2022-01-18T18:07:12.565 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '84.92.90.205', 'port': 8444}
2022-01-18T18:07:15.757 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 456c5fdfb9d024b0593ba53af1776cac9f7cd5b296621aaa48b22da544ac23f3. caching signage point 47.
2022-01-18T18:07:24.712 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 456c5fdfb9d024b0593ba53af1776cac9f7cd5b296621aaa48b22da544ac23f3. caching signage point 48.
2022-01-18T18:07:34.441 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8d300d1b4119c78f10281e539f8726d6564b30feea0d3a2cd48493558b09b6c7. caching signage point 49.
2022-01-18T18:07:42.840 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8d300d1b4119c78f10281e539f8726d6564b30feea0d3a2cd48493558b09b6c7. caching signage point 50.
2022-01-18T18:07:51.445 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8d300d1b4119c78f10281e539f8726d6564b30feea0d3a2cd48493558b09b6c7. caching signage point 51.
2022-01-18T18:07:59.120 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '39.170.218.224', 'port': 8444}.Inbound limit reached.
2022-01-18T18:07:59.385 full_node full_node_server        : INFO     Connection closed: 39.170.218.224, node id: e1a1b87ffb068b0a0838783c67e6d0896eb6f3b92b52f8c138db03b59be7ef83
2022-01-18T18:07:59.387 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '39.170.218.224', 'port': 8444}
2022-01-18T18:08:00.425 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8d300d1b4119c78f10281e539f8726d6564b30feea0d3a2cd48493558b09b6c7. caching signage point 52.
2022-01-18T18:08:08.918 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8d300d1b4119c78f10281e539f8726d6564b30feea0d3a2cd48493558b09b6c7. caching signage point 53.
2022-01-18T18:08:17.828 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8d300d1b4119c78f10281e539f8726d6564b30feea0d3a2cd48493558b09b6c7. caching signage point 54.
2022-01-18T18:08:26.655 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8d300d1b4119c78f10281e539f8726d6564b30feea0d3a2cd48493558b09b6c7. caching signage point 55.
2022-01-18T18:08:37.501 full_node chia.full_node.full_node_store: INFO     Don't have rc hash bfbad0b76e3ae73fe638a4db4ac4eb0738e09e6c29ba8f94101f860fdbc2e924. caching signage point 56.
2022-01-18T18:08:46.174 full_node chia.full_node.full_node_store: INFO     Don't have rc hash bfbad0b76e3ae73fe638a4db4ac4eb0738e09e6c29ba8f94101f860fdbc2e924. caching signage point 57.
2022-01-18T18:08:47.975 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '100.34.189.80', 'port': 8444}.Inbound limit reached.
2022-01-18T18:08:48.061 full_node full_node_server        : INFO     Connection closed: 100.34.189.80, node id: 50fc32e4ae0890d77e601d79ddc67462fac580837a4678f4b537f860a0f91295
2022-01-18T18:08:48.063 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '100.34.189.80', 'port': 8444}
2022-01-18T18:08:54.680 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '142.114.201.14', 'port': 8444}.Inbound limit reached.
2022-01-18T18:08:54.812 full_node full_node_server        : INFO     Connection closed: 142.114.201.14, node id: f4c658f390a5d15a125e90f082774dba4858f4d5f8f01cae2b537a2203cebb4e
2022-01-18T18:08:54.813 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '142.114.201.14', 'port': 8444}
2022-01-18T18:08:55.961 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 2200d9e22fed3cd4750bf380b8ad40b80f7d0c0d4a9391e059c5abba46f56a5a. caching signage point 58.
2022-01-18T18:09:04.485 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 2200d9e22fed3cd4750bf380b8ad40b80f7d0c0d4a9391e059c5abba46f56a5a. caching signage point 59.
2022-01-18T18:09:13.304 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 2200d9e22fed3cd4750bf380b8ad40b80f7d0c0d4a9391e059c5abba46f56a5a. caching signage point 60.
2022-01-18T18:09:22.089 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 2200d9e22fed3cd4750bf380b8ad40b80f7d0c0d4a9391e059c5abba46f56a5a. caching signage point 61.
2022-01-18T18:09:33.395 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93. caching signage point 62.
2022-01-18T18:09:40.586 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93. caching signage point 63.
2022-01-18T18:09:49.568 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.569 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.655 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.656 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.664 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.665 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.704 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.705 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.722 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.724 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.728 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.729 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.755 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.755 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.820 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.821 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.826 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.827 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.831 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.832 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:09:49.838 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:09:49.839 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8```

Full node continues to log messages like this until I stopped it.

```2022-01-18T18:27:17.995 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:27:17.995 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:28:01.124 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '96.2.162.91', 'port': 8444}.Inbound limit reached.
2022-01-18T18:28:01.487 full_node full_node_server        : INFO     Connection closed: 96.2.162.91, node id: 5597b6f9401a0a83bc0c6584f7c4e0646b8930b0fbd531d8983a062b9cbbd75d
2022-01-18T18:28:01.489 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '96.2.162.91', 'port': 8444}
2022-01-18T18:28:32.414 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash d6ec983d6440fd3e230879ade5ea491fed5d5aa697d702090e7a7b5722b61d93, caching EOS
2022-01-18T18:28:32.416 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 8b25bfad1293b996336cb7c0960f3bd9aa7f71415f8a1dc5e01ab36b9d26f7b8
2022-01-18T18:29:04.947 full_node full_node_server        : INFO     Connection closed: 183.66.203.206, node id: ebe24a047f17c7aa591768567602fd8b0e6080460fe03731413a41ce61582e1b
2022-01-18T18:29:04.948 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '183.66.203.206', 'port': 8444}
2022-01-18T18:30:53.145 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '184.105.247.154', 'port': 8444}.Inbound limit reached.
2022-01-18T18:30:53.187 full_node full_node_server        : INFO     Connection closed: 184.105.247.154, node id: 6e864025c410b8020c52efd65b2d80170c6716afa0b999bc1be2d28b76fbf167
2022-01-18T18:30:53.189 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '184.105.247.154', 'port': 8444}
2022-01-18T18:32:09.680 full_node full_node_server        : INFO     Cannot connect to host 120.235.75.118:8444 ssl:<ssl.SSLContext object at 0xffff9ff725c0> [Connect call failed ('120.235.75.118', 8444)]

2022-01-18T18:48:03.483 full_node full_node_server        : INFO     Cannot connect to host 2a00:23c7:5102:8b01:4a4a:757:ebdc:d70e:8444 ssl:<ssl.SSLContext object at 0xffffaf057cc0> [Network is unreachable]```

stop everything with chia stop -d all

```2022-01-18T22:10:09.306 full_node full_node_server        : INFO     Connection closed: 185.22.10.233, node id: 0be0c2aa02bcc63e69192fbc248eebbb018784d359137382ebdf81158cd3a329
2022-01-18T22:10:09.308 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '185.22.10.233', 'port': 8444}
2022-01-18T22:10:09.311 full_node full_node_server        : INFO     Connection closed: 223.166.20.217, node id: 05aba85108c5f4436546916cff9f4fc66f3692ff5049158dd62b2b24998f9449
2022-01-18T22:10:09.312 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '223.166.20.217', 'port': 8444}
2022-01-18T22:10:10.453 full_node full_node               : INFO     Waiting for RPC server
2022-01-18T22:10:10.454 full_node full_node               : INFO     Closed RPC server
2022-01-18T22:10:10.455 full_node full_node               : INFO     Waiting for service _await_closed callback
2022-01-18T22:10:13.075 full_node full_node               : INFO     Service full_node at port 8444 fully closed
2022-01-18T22:10:13.076 full_node full_node_server        : INFO     Connection closed: 115.93.222.234, node id: ce4bbd745b5197e3a4387d40f1cb4baf9710793884f89c480e29b93f98c3869e
2022-01-18T22:10:13.078 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '115.93.222.234', 'port': 8444}
2022-01-18T22:10:13.080 full_node full_node_server        : INFO     Connection closed: 47.242.128.84, node id: 3fdf9a68acce0ce35d0b3273b489c90eb41c8915c461d811a494f2cdc83b62c3
2022-01-18T22:10:13.082 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '47.242.128.84', 'port': 8444}

full_node logs after restarting

2022-01-18T22:11:54.085 full_node chia.consensus.blockchain: INFO     Started 2 processes for block validation
2022-01-18T22:11:54.092 full_node chia.full_node.full_node: INFO     Initialized with empty blockchain time taken: 0s
2022-01-18T22:11:54.128 full_node full_node_server        : INFO     Started listening on port: 8444
2022-01-18T22:11:54.129 full_node full_node               : INFO     Started full_node service on network_id: mainnet
2022-01-18T22:11:54.687 farmer farmer_server              : INFO     Connected with full_node {'host': '127.0.0.1', 'port': 8444}
2022-01-18T22:11:55.566 full_node chia.full_node.full_node: INFO     Received 32 peers from DNS seeder, using rdtype = A.
2022-01-18T22:11:55.707 full_node chia.full_node.full_node: INFO     Received 32 peers from DNS seeder, using rdtype = AAAA.
2022-01-18T22:11:56.547 full_node full_node_server        : INFO     Connected with full_node {'host': '94.21.120.129', 'port': 8444}
2022-01-18T22:11:56.667 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2022-01-18T22:11:56.668 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2022-01-18T22:11:57.414 full_node full_node_server        : INFO     Connected with full_node {'host': '5.206.102.224', 'port': 8444}
2022-01-18T22:11:57.510 full_node full_node_server        : INFO     Connected with full_node {'host': '220.100.19.173', 'port': 8444}
2022-01-18T22:11:57.621 full_node full_node_server        : INFO     Connection closed: 5.206.102.224, node id: 161b26226d566b5ba20bd6497b5bfb3704d60b537f5ae7d229b137033d988771
2022-01-18T22:11:57.623 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '5.206.102.224', 'port': 8444}
2022-01-18T22:11:57.626 full_node full_node_server        : INFO     Connection closed: 220.100.19.173, node id: cc100dcb143ac871b9638db3de54e1fb49b7217684abf2ca19b6d2fc14da5930
2022-01-18T22:11:57.627 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '220.100.19.173', 'port': 8444}
2022-01-18T22:11:58.280 full_node full_node_server        : INFO     Connected with full_node {'host': '203.106.157.229', 'port': 8444}
2022-01-18T22:12:05.672 full_node full_node_server        : INFO     Connected with full_node {'host': '107.190.8.97', 'port': 8444}
2022-01-18T22:12:06.007 full_node chia.full_node.full_node: INFO     Collected a total of 3 peaks.
2022-01-18T22:12:06.008 full_node chia.full_node.full_node: INFO     Selected peak 1441282, e5fcbfd3a4db823266bc9e0153580a11118e4d9607cfcceea8e9782e3481d977
2022-01-18T22:12:06.651 full_node chia.full_node.full_node: INFO     Total of 3 peers with peak 1441282
2022-01-18T22:12:06.652 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 203.106.157.229 up to height 1441282
2022-01-18T22:12:07.605 full_node full_node_server        : INFO     Cannot connect to host 142.154.44.191:8444 ssl:<ssl.SSLContext object at 0xffff952fe8c0> [Connect call failed ('142.154.44.191', 8444)]
2022-01-18T22:12:20.440 full_node chia.full_node.weight_proof: INFO     validate weight proof peak height 1441282
2022-01-18T22:12:30.974 full_node chia.full_node.weight_proof: INFO     validating 3753 sub epochs
2022-01-18T22:12:40.807 full_node full_node_server        : INFO     Cannot connect to host 2001:8003:d53b:7601:e10a:2d32:f51b:6f7e:8444 ssl:<ssl.SSLContext object at 0xffff952fe640> [Network is unreachable]
2022-01-18T22:12:53.584 full_node full_node_server        : INFO     Cannot connect to host 2607:f2c0:e016:96:e1fb:72db:e344:5bd5:8444 ssl:<ssl.SSLContext object at 0xffff7e70f7c0> [Network is unreachable]
2022-01-18T22:12:57.430 full_node full_node_server        : INFO     Connected with full_node {'host': '193.92.234.220', 'port': 8444}
2022-01-18T22:13:14.378 full_node full_node_server        : INFO     Connected with full_node {'host': '80.208.96.138', 'port': 8444}
2022-01-18T22:13:14.463 full_node full_node_server        : INFO     Connection closed: 80.208.96.138, node id: 6a222d6400970a44d8deab119ebec342a3c74a097d636d1e1a340666a6673bc5
2022-01-18T22:13:14.465 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '80.208.96.138', 'port': 8444}
2022-01-18T22:13:21.739 full_node full_node_server        : INFO     Cannot connect to host 84.74.240.92:8444 ssl:<ssl.SSLContext object at 0xffff97e2d740> [Connect call failed ('84.74.240.92', 8444)]
2022-01-18T22:13:48.763 full_node full_node_server        : INFO     Cannot connect to host 46.205.201.20:8444 ssl:<ssl.SSLContext object at 0xffff7e733940> [Connect call failed ('46.205.201.20', 8444)]
2022-01-18T22:13:56.828 full_node full_node_server        : INFO     Cannot connect to host 184.82.65.175:8444 ssl:<ssl.SSLContext object at 0xffff97de8ec0> [Connect call failed ('184.82.65.175', 8444)]
2022-01-18T22:14:26.638 wallet wallet_server              : INFO     Connected with full_node {'host': '127.0.0.1', 'port': 8444}
2022-01-18T22:14:35.974 full_node full_node_server        : INFO     Connected with full_node {'host': '68.186.66.198', 'port': 8444}
2022-01-18T22:14:36.068 full_node full_node_server        : INFO     Connection closed: 68.186.66.198, node id: e5369004d8e518c2e87373ed91e802b2d1c33b1c766b9454eb7fd3ed34ebeb50
2022-01-18T22:14:36.069 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '68.186.66.198', 'port': 8444}
2022-01-18T22:14:57.731 full_node full_node_server        : INFO     Cannot connect to host 198.54.134.87:57452 ssl:<ssl.SSLContext object at 0xffff7e733a40> [Connect call failed ('198.54.134.87', 57452)]
2022-01-18T22:15:20.705 full_node full_node_server        : INFO     Cannot connect to host 2001:e68:540a:89e3:fd30:86bb:7d1e:aca9:8444 ssl:<ssl.SSLContext object at 0xffff953761c0> [Network is unreachable]
2022-01-18T22:15:29.821 full_node chia.full_node.full_node: INFO     Re-checked peers: total of 3 peers with peak 1441282
2022-01-18T22:15:29.825 full_node chia.full_node.full_node: INFO     Start syncing from fork point at 0 up to 1441282
2022-01-18T22:15:38.639 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2022-01-18T22:15:38.641 full_node chia.full_node.full_node: INFO     long sync done
2022-01-18T22:15:38.648 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 0, weight 7, hh d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2, forked at 0, rh: bdde7b5b2bc6025c07a9f5233d8eae167bea654146b272652262b362524c3e85, total iters: 11798946, overflow: False, deficit: 15, difficulty: 7, sub slot iters: 134217728, Generator size: No tx, Generator ref list size: No tx
2022-01-18T22:15:38.651 full_node chia.full_node.mempool_manager: INFO     Size of mempool: 0 spends, cost: 0 minimum fee to get in: 0
2022-01-18T22:15:44.953 full_node full_node_server        : INFO     Connected with full_node {'host': '85.7.111.129', 'port': 8444}
2022-01-18T22:15:45.085 full_node full_node_server        : INFO     Connection closed: 85.7.111.129, node id: dec5ca10562fed35e735425315b4fd46b66bccd5a6bc0cc2ac10f8291b5eb2a5
2022-01-18T22:15:45.087 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '85.7.111.129', 'port': 8444}
2022-01-18T22:15:47.270 full_node full_node_server        : INFO     Cannot connect to host 2a02:908:1060:b8c0:60c8:597d:aab7:a688:8444 ssl:<ssl.SSLContext object at 0xffff7d69a540> [Network is unreachable]
2022-01-18T22:15:58.261 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2022-01-18T22:15:58.263 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2022-01-18T22:15:59.080 full_node chia.full_node.full_node: INFO     Collected a total of 4 peaks.
2022-01-18T22:15:59.082 full_node chia.full_node.full_node: INFO     Selected peak 1441290, 3d05d93ed89f67f84fc5e1a5d78e4d174737b783e43494807d6c8a16af88c382
2022-01-18T22:15:59.451 full_node chia.full_node.full_node: INFO     Total of 4 peers with peak 1441290
2022-01-18T22:15:59.452 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 94.21.120.129 up to height 1441290
2022-01-18T22:16:03.951 full_node full_node_server        : INFO     Connected with full_node {'host': '192.168.0.102', 'port': 8444}
2022-01-18T22:16:16.760 full_node full_node_server        : INFO     Cannot connect to host 117.171.188.70:8444 ssl:<ssl.SSLContext object at 0xffff953766c0> [Connect call failed ('117.171.188.70', 8444)]

2022-01-18T22:16:25.095 full_node asyncio                 : ERROR    Task was destroyed but it is pending!

2022-01-18T22:16:32.117 full_node chia.full_node.weight_proof: INFO     validate weight proof peak height 1441290
2022-01-18T22:16:32.696 full_node chia.full_node.weight_proof: INFO     validating 3753 sub epochs
2022-01-18T22:16:35.794 full_node full_node_server        : INFO     Connection closed: 183.143.129.118, node id: 0d1e79d52a1628bd76ba92a606c9047fd10015d87e3e6a1fc60479255ab8eca4
2022-01-18T22:16:35.796 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '183.143.129.118', 'port': 8444}
2022-01-18T22:16:40.364 full_node full_node_server        : INFO     Connected with full_node {'host': '67.177.33.63', 'port': 8444}
2022-01-18T22:16:40.450 full_node full_node_server        : INFO     Connection closed: 67.177.33.63, node id: 427650022adec5c02a7dbdbd350e3256e7c7fa537b9368550c7c2483ef4f6819
2022-01-18T22:16:40.452 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '67.177.33.63', 'port': 8444}
2022-01-18T22:16:44.209 full_node full_node_server        : INFO     Connected with full_node {'host': '66.169.36.23', 'port': 8444}
2022-01-18T22:16:44.309 full_node full_node_server        : INFO     Connection closed: 66.169.36.23, node id: 610251d758c046669510983d6bc7fb5edfd0124685d724af17702e72e75766f0
2022-01-18T22:16:44.310 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '66.169.36.23', 'port': 8444}
2022-01-18T22:17:03.486 full_node full_node_server        : INFO     Connection closed: 188.190.153.27, node id: 4057c397cb678738cc19f7d87062005f9b26d8af1ade8d1e276a0a546827b5ce
2022-01-18T22:17:03.488 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '188.190.153.27', 'port': 8444}
2022-01-18T22:17:30.403 full_node full_node_server        : INFO     Cannot connect to host 110.191.209.211:8444 ssl:<ssl.SSLContext object at 0xffff7d6a3f40> [Connect call failed ('110.191.209.211', 8444)]
2022-01-18T22:17:35.184 full_node full_node_server        : INFO     Cannot connect to host 2600:1700:90:8030:dda8:165f:96ac:af2d:8444 ssl:<ssl.SSLContext object at 0xffff7eaa6ec0> [Network is unreachable]
2022-01-18T22:17:42.695 full_node full_node_server        : INFO     Cannot connect to host 2601:409:8404:2e30:6547:aaa5:c2b9:5646:8444 ssl:<ssl.SSLContext object at 0xffff7d69acc0> [Network is unreachable]
2022-01-18T22:17:48.516 full_node chia.full_node.full_node: INFO     Re-checked peers: total of 4 peers with peak 1441290
2022-01-18T22:17:48.525 full_node chia.full_node.full_node: INFO     Start syncing from fork point at 0 up to 1441290

2022-01-18T22:17:51.766 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name

2022-01-18T22:17:51.767 full_node chia.full_node.full_node: INFO     long sync done
2022-01-18T22:17:51.772 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 0, weight 7, hh d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2, forked at 0, rh: bdde7b5b2bc6025c07a9f5233d8eae167bea654146b272652262b362524c3e85, total iters: 11798946, overflow: False, deficit: 15, difficulty: 7, sub slot iters: 134217728, Generator size: No tx, Generator ref list size: No tx
2022-01-18T22:18:01.881 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2022-01-18T22:18:01.883 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2022-01-18T22:18:02.090 full_node chia.full_node.full_node: INFO     Collected a total of 3 peaks.
2022-01-18T22:18:02.091 full_node chia.full_node.full_node: INFO     Selected peak 1441301, 2a8a0929a7f45830806364a83a9b6fa6035ab7c3a6a75cd8b87b8917c4c231c7
2022-01-18T22:18:02.320 full_node chia.full_node.full_node: INFO     Total of 4 peers with peak 1441301
2022-01-18T22:18:02.321 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 203.106.157.229 up to height 1441301
2022-01-18T22:18:16.421 full_node chia.full_node.weight_proof: INFO     validate weight proof peak height 1441301
2022-01-18T22:18:16.763 full_node chia.full_node.weight_proof: INFO     validating 3753 sub epochs
2022-01-18T22:18:24.235 full_node full_node_server        : INFO     Cannot connect to host 240e:37a:e59:8300:ec35:4fcd:cdcf:5218:8444 ssl:<ssl.SSLContext object at 0xffff7ed691c0> [Network is unreachable]
2022-01-18T22:18:28.330 full_node full_node_server        : INFO     Connected with full_node {'host': '104.48.64.115', 'port': 8444}
2022-01-18T22:18:28.444 full_node full_node_server        : INFO     Connection closed: 104.48.64.115, node id: 827cf833622152d04fa7860508999ab9a94d138a69c3542bc4ec37508c467f55
2022-01-18T22:18:28.445 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '104.48.64.115', 'port': 8444}
2022-01-18T22:18:32.974 full_node full_node_server        : INFO     Connected with full_node {'host': '85.126.26.27', 'port': 8444}
2022-01-18T22:18:33.055 full_node full_node_server        : INFO     Connection closed: 85.126.26.27, node id: 7b6fdb1e43ec433d06c902d1167f902a0531c62f5f65b714e62d994c166db1ad
2022-01-18T22:18:33.057 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '85.126.26.27', 'port': 8444}
2022-01-18T22:18:33.079 full_node full_node_server        : INFO     Connection closed: 184.105.247.154, node id: 56bc59d4a90fa6b34a917e2d395acbbd45c2cbec8185a2dacda0e4f5c95d6b93
2022-01-18T22:18:33.081 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '184.105.247.154', 'port': 8444}

2022-01-18T22:18:45.446 full_node full_node_server        : INFO     Connection closed: 84.92.90.205, node id: 7b5c615c94e8b90eb5a15607c869dcf6c2a0bcf079115ce1a03dac16d8fd2a44
2022-01-18T22:18:45.447 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '84.92.90.205', 'port': 8444}
2022-01-18T22:18:57.550 full_node full_node_server        : INFO     Connected with full_node {'host': '195.189.60.232', 'port': 8444}
2022-01-18T22:19:06.414 full_node full_node_server        : INFO     Cannot connect to host 89.64.80.172:8444 ssl:<ssl.SSLContext object at 0xffff7eaec5c0> [Connect call failed ('89.64.80.172', 8444)]
2022-01-18T22:19:14.755 full_node full_node_server        : INFO     Connected with full_node {'host': '120.78.134.20', 'port': 8444}
2022-01-18T22:19:14.846 full_node full_node_server        : INFO     Connection closed: 120.78.134.20, node id: b14a4d4dedea5ae3322bfe45d4c588f76a01a48814796e50be46d1a0fb9ddcd1

After restarting with chia start farmer the logs eventually displayed the errors that started this bug report. The full_node appeared to read from the DB for hours. Eventually the the full_node consumed over 6GB of RAM and essentially used all available memory and the process was killed by the OS.

Functionality was restored by copying the blockchain db from my other full node.

Let me know if you want the logs and / or the DB.

vernzimm commented 2 years ago

debug.log.log

2022-01-17T13:52:47.643 full_node chia.full_node.full_node: ERROR got weight proof request for unknown peak d5bf820d0240699b3ef390a3f5f3496c2c4c2812c9609a2eeb6152f53717f0e6 2022-01-17T14:35:24.361 full_node chia.full_node.full_node: ERROR sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name 2022-01-17T14:35:55.202 full_node asyncio : ERROR Task was destroyed but it is pending! task: <Task pending name='Task-1054' coro=<FullNode.sync_from_fork_point..fetch_block_batches() running at chia\full_node\full_node.py:893> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x000002C180D66A60>()]> cb=[gather.._done_callback() at asyncio\tasks.py:766]> 2022-01-17T14:37:45.362 full_node chia.full_node.full_node: ERROR sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name 2022-01-17T14:39:04.141 full_node full_node_server : WARNING Banning 106.115.114.82 for 600 seconds 2022-01-17T14:39:04.143 full_node chia.full_node.full_node: ERROR Error with syncing: <class 'ValueError'>Traceback (most recent call last): File "chia\full_node\full_node.py", line 823, in _sync File "chia\full_node\weight_proof.py", line 606, in validate_weight_proof concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "chia\full_node\full_node.py", line 826, in _sync ValueError: Weight proof validation threw an error A process in the process pool was terminated abruptly while the future was running or pending.

2022-01-17T14:49:20.766 full_node full_node_server : WARNING Invalid handshake with peer {'host': '67.169.137.61', 'port': 8744}. Maybe the peer is running old software. 2022-01-17T14:52:25.581 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2' 2022-01-17T14:52:25.582 full_node asyncio : ERROR Task exception was never retrieved future: <Task finished name='Task-150' coro=<FullNode._sync() done, defined at chia\full_node\full_node.py:716> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')> Traceback (most recent call last): File "chia\full_node\full_node.py", line 847, in _sync File "chia\full_node\full_node.py", line 1081, in _finish_sync File "chia\consensus\blockchain.py", line 157, in get_peak File "chia\consensus\blockchain.py", line 674, in height_to_block_record File "chia\consensus\blockchain.py", line 670, in block_record KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2' 2022-01-17T14:52:25.602 full_node asyncio : ERROR Task exception was never retrieved future: <Task finished name='Task-3070' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')> Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 51, in _state_changed File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state File "chia\consensus\blockchain.py", line 157, in get_peak File "chia\consensus\blockchain.py", line 674, in height_to_block_record File "chia\consensus\blockchain.py", line 670, in block_record KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2' 2022-01-17T14:52:27.126 full_node asyncio : ERROR Task exception was never retrieved future: <Task finished name='Task-3080' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>

vernzimm commented 2 years ago

Note: my mention of this in my other report 9970 is only potentially related. I didn't realize it would tag it here like this.

mariano54 commented 2 years ago

Can anyone that experienced the issue answer these questions:

  1. What OS and OS version are you running?
  2. What CPU do you have?
  3. How much RAM does your system have? did you ever get close to running out of RAM?
  4. Are you running with default settings or did you change the config ?
jeffj55374 commented 2 years ago

Can anyone that experienced the issue answer these questions:

  1. What OS and OS version are you running? Ubuntu 21.04 (venv) ubuntu@chia-1:~/.chia/mainnet/db$ uname -a Linux chia-1 5.11.0-1027-raspi #30-Ubuntu SMP PREEMPT Thu Jan 13 12:52:48 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

  2. What CPU do you have? Raspberry Pi 4

  3. How much RAM does your system have? did you ever get close to running out of RAM? 8GB When I noticed that I was no longer finding partials I wasn't having any memory or CPU usage issues. (Full logs posted earlier) After I stopped farming and restarted, The full_node never synced and after 3 - 4 hours it consumed over 6GB of RAM and continued until memory was exhausted and the OS killed it. Seems the out of memory was a side effect after restarting. Suspect the DB was corrupted. I duplicated this 4 times before copying a new DB from my other full node on a different system.

  4. Are you running with default settings or did you change the config ? Default config other than turning on INFO logging and target_peer_count=40

alx741 commented 2 years ago

Can anyone that experienced the issue answer these questions:

  1. What OS and OS version are you running?

Archlinux 5.16.1 x86_64

2. What CPU do you have?

Intel i7-4770

3. How much RAM does your system have? did you ever get close to running out of RAM?

6GB usage hangs around 40%

4. Are you running with default settings or did you change the config ?

Default + INFO logging

vernzimm commented 2 years ago

What OS and OS version are you running? Win10 Home 21H2

What CPU do you have? Intel I5-2500K

How much RAM does your system have? did you ever get close to running out of RAM? 32GB... nah

Are you running with default settings or did you change the config ? Default as far as I know.

grocheireland commented 2 years ago

sorry for late reply.... i dont have the machine to hand, but its Windows i7 with alot of RAM, so dont think it was running low.....

I know when i started it up it was not sync'ing properly, see attached(this screen shot i took from another user who put it up on keybase

I am not sure if this is relevant or not, but when i had the above problem i used the menu Force Reload..a few times and also swapped keys on the GUI. I was getting desparate to get it to sync Then i started to get the error with b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x in a pop up

Probably not much help.....

Sorry i dont have the logs either

Also I never changed the configs on that machine.... so it was default

durdin85 commented 2 years ago

Can anyone that experienced the issue answer these questions: yes.

What OS and OS version are you running? Ubuntu 20.04.3 LTS

What CPU do you have? arm64 / Raspberry PI4b

How much RAM does your system have? 4GB

did you ever get close to running out of RAM? yes it actually did a couple of times, and it was swapping to ssd many times (I allowed memory overcommitment, so OOM does not act), and the consumption of the chia processes sometimes grown to around 6GB of (RAM+SWAP) after couple of days, stopped syncing and I had to restart chia or entire machine to get it to "normal".

Are you running with default settings or did you change the config ? Couple things changed in config (upnp, amount of allowed peers, plot locations, logging level)

tomachalek commented 2 years ago

What OS and OS version are you running?

Ubuntu 20.04.3 LTS

What CPU do you have?

Intel Core i5-9600K (slightly underclocked)

How much RAM does your system have? did you ever get close to running out of RAM?

8GB, yes (about 7GB consumed; unfortunately I don't have a measurement or log for this)

Are you running with default settings or did you change the config ?

Mostly default, except for custom plot directories and a configuration for a remote harvester (on the same LAN, 1 GigE)

Just for interest, here is a chart of my blockchain hdd reads/writes (measured per 5 second period) during those error events:

chia_sync_fail