search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[Bug] FullNode suddenly crashes with lot of "Don't have rc hash ...." #9333

Closed mahaupt closed 2 years ago

mahaupt commented 2 years ago

What happened?

Hi all Today, the full node suddenly stopps working with the attached log messages... The command "chia farm summary" takes really long and results in an exception from the full_node When doing a full restart, the full_node stopps working again after a few minutes

cheers

Version

1.2.11

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

2021-11-21T08:06:04.930 harvester chia.harvester.harvester: INFO     8 plots were eligible for farming 62caf25038... Found 0 proofs. Time: 1.02751 s. Total 4744 plots
2021-11-21T08:06:09.734 full_node chia.full_node.full_node: INFO     Added unfinished_block 2805e53407d83809404a72bb244d6a637eb486b5ece610d623fb30a0f33abda1, not farmed by us, SP: 31 farmer response time: 5.6249, Pool pk xch1jp6frj3ecddur7dxak3n7lq0j75ltquh2zyd44epdu0d6704y2hqyky5hf, validation time: 0.1296 seconds, pre_validation time 0.0776, cost: 1110754701, percent full: 10.098%
2021-11-21T08:06:14.969 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8b809b0c73a073214a5fef4ff10cf3fd593dd0aa56ac6a403e8537f8d812a0b6. caching signage point 32.
2021-11-21T08:06:22.918 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8b809b0c73a073214a5fef4ff10cf3fd593dd0aa56ac6a403e8537f8d812a0b6. caching signage point 33.
2021-11-21T08:06:31.743 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 8b809b0c73a073214a5fef4ff10cf3fd593dd0aa56ac6a403e8537f8d812a0b6. caching signage point 34.
2021-11-21T08:06:42.770 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 47e2c7aab39cce3e39b53d6d5bff0e307bb958904c23dfa5ea1626457d4b8f52. caching signage point 35.
2021-11-21T08:06:51.148 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 47e2c7aab39cce3e39b53d6d5bff0e307bb958904c23dfa5ea1626457d4b8f52. caching signage point 36.
2021-11-21T08:07:00.136 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 47e2c7aab39cce3e39b53d6d5bff0e307bb958904c23dfa5ea1626457d4b8f52. caching signage point 37.
2021-11-21T08:07:06.357 full_node full_node_server        : INFO     Connection closed: 77.91.196.54, node id: 00bf62b5d2b24409decb561f09e850af207832284cc9afbdf37fd87064508ead
2021-11-21T08:07:06.358 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '77.91.196.54', 'port': 8449}
2021-11-21T08:07:09.025 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 47e2c7aab39cce3e39b53d6d5bff0e307bb958904c23dfa5ea1626457d4b8f52. caching signage point 38.
2021-11-21T08:07:17.955 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 47e2c7aab39cce3e39b53d6d5bff0e307bb958904c23dfa5ea1626457d4b8f52. caching signage point 39.
2021-11-21T08:07:29.057 full_node chia.full_node.full_node_store: INFO     Don't have rc hash c5ecff8325d09d363140bb7d3c59a00c41d9544cd2e910f391c3ac3115618a15. caching signage point 40.
2021-11-21T08:07:30.468 full_node full_node_server        : INFO     Connection closed: 160.16.67.30, node id: e419f64edba5bd71346988639bdb338cc0943174684d3af991c57d7d37507036
2021-11-21T08:07:30.469 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '160.16.67.30', 'port': 8444}
2021-11-21T08:07:38.693 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 063b4b9a4fd94676a2479444f1741ee8dca23772e1387ce307f269fb753da028. caching signage point 41.
2021-11-21T08:07:47.359 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 063b4b9a4fd94676a2479444f1741ee8dca23772e1387ce307f269fb753da028. caching signage point 42.
2021-11-21T08:07:57.435 harvester chia.plotting.util      : WARNING  Directory: /mnt/plots/38 does not exist.
2021-11-21T08:07:57.442 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 4745, duration: 0.00 seconds
2021-11-21T08:07:57.756 farmer chia.farmer.farmer         : INFO     GET /farmer response: {'authentication_public_key': '0xb58ebff9ca789b27064484e39e2246b8b01c6574beebf9be585f37815c09db38b42002c08d2261d2cbd9817a7fe5fff5', 'payout_instructions': '8bc168b514360856e2ea6a589f28cfeea84141b5e855ed364f6325bddae4200d', 'current_difficulty': 19, 'current_points': 1234}
2021-11-21T08:07:58.010 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 43.
2021-11-21T08:07:58.209 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4445, duration: 0.06 seconds
2021-11-21T08:07:58.276 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4145, duration: 0.06 seconds
2021-11-21T08:07:58.338 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3845, duration: 0.06 seconds
2021-11-21T08:07:58.502 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3545, duration: 0.16 seconds
2021-11-21T08:07:58.571 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3245, duration: 0.07 seconds
2021-11-21T08:07:58.648 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2945, duration: 0.07 seconds
2021-11-21T08:07:58.720 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2645, duration: 0.07 seconds
2021-11-21T08:07:58.793 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2345, duration: 0.07 seconds
2021-11-21T08:07:58.842 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2045, duration: 0.05 seconds
2021-11-21T08:07:58.912 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1745, duration: 0.07 seconds
2021-11-21T08:07:58.973 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1445, duration: 0.06 seconds
2021-11-21T08:07:59.037 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1145, duration: 0.06 seconds
2021-11-21T08:07:59.102 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 845, duration: 0.06 seconds
2021-11-21T08:07:59.165 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 545, duration: 0.06 seconds
2021-11-21T08:07:59.222 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 245, duration: 0.05 seconds
2021-11-21T08:07:59.279 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 245, remaining 0, duration: 0.05 seconds
2021-11-21T08:07:59.280 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 0, removed 0, processed 4745, remaining 0, duration: 1.10 seconds
2021-11-21T08:08:06.335 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 44.
2021-11-21T08:08:15.251 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 45.
2021-11-21T08:08:23.880 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 46.
2021-11-21T08:08:32.659 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 47.
2021-11-21T08:08:41.354 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 48.
2021-11-21T08:08:50.769 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 49.
2021-11-21T08:08:59.194 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 50.
2021-11-21T08:09:08.033 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 95ccc10ceafbc086f77069dc04ad6878c40a64f8c7146b9686a693c78143aea5. caching signage point 51.
mahaupt commented 2 years ago

Update: After a few times restarting all services, chia caught itself and continued farming. 🤖

emlowe commented 2 years ago

I am curious why you mentioned crash in the title, since your log snippet doesn't indicate any crashes. However, caching this many SPs would seem to indicate you are missing some blocks. In any event, closing issue per last comment

mahaupt commented 2 years ago

By „crash“ I mean stopped farming and could not resync without intervention

emlowe commented 2 years ago

Thanks, I re-opened this as there are apparently some other related reports - so we continue to investigate

peter92427 commented 2 years ago

This happens with GUI too. Similar logs. Full Node Overview Status says Not Synced. Restart helps.

Ealrann commented 2 years ago

Same here, on a VPS (4 cores, 8GB ram, 1Gbps internet, 80 peers connected) :

2021-12-09T10:02:35.644 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 5a1a2b81520c5a2fb6973844ef188613bd1db46ba5b9c3df15174953821fb31f. caching signage point 61.
2021-12-09T10:03:57.251 full_node chia.full_node.full_node_store: INFO     Don't have rc hash c9785c398212a179b61bb6cb45b8f62a651bf948b5b915250718e758e59ac8e5. caching signage point 9.
2021-12-09T10:12:59.214 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d38d5943976270e3c8495956f5936ccb9112f19c8fbae67ae472c8841ee6bab8. caching signage point 63.
2021-12-09T10:17:34.258 full_node chia.full_node.full_node_store: INFO     Don't have rc hash a79e5028e8c23b842575fa3af2aff84b33751901025dac9381087c58deb4ae13. caching signage point 33.
2021-12-09T10:22:32.393 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 01452f39b6a036c1ce200661a86123a40bf69305c6b2b109647b2d5d5247ec82. caching signage point 61.
2021-12-09T10:22:43.016 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e64f3a246e6f1c480dcde2ac3a65b8caa117227730f7b47842f1702d7020c5c3. caching signage point 62.
2021-12-09T10:22:52.470 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 7a43add912df1c7dce2bba3f2fdc5e79cd5c556ca40e934a0456a3471d125305. caching signage point 63.
2021-12-09T10:28:47.491 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 57c1d9e14824b18d5f3421263b085c8ddde8efd02067c7f6b9c4654eb616a699. caching signage point 41.
2021-12-09T10:32:22.428 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 1eccbebc589981acadf23c4b3a1b0b85d753e2917d779264da7675f779762b13. caching signage point 61.
2021-12-09T10:32:31.274 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 1eccbebc589981acadf23c4b3a1b0b85d753e2917d779264da7675f779762b13. caching signage point 62.
2021-12-09T10:42:30.672 full_node chia.full_node.full_node_store: INFO     Don't have rc hash e2584bfd941ddc5e39e86674cc6a2b3b42256db14fe2401c67dbe52e38c6ac8a. caching signage point 1.
2021-12-09T10:42:30.967 full_node chia.full_node.full_node_store: INFO     Don't have rc hash cf3079a7ed7868cf9b6b824ac8c3d5b0c8765ef3915cff877b25abac28dd62d2. caching signage point 61.
2021-12-09T10:46:15.418 full_node chia.full_node.full_node_store: INFO     Don't have rc hash d61eb72c4f136534bda4005d219d648f06a63e5a9bf9416b6c8dd4b0627fce9a. caching signage point 25.
2021-12-09T10:52:19.065 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 9f84426d90901992dfa74b03096dc2602fcbf7efd81c58970e12cee058d01f7f. caching signage point 61.
2021-12-09T11:02:28.955 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 1e7a3f416677e3b30f6f47108e3f99fb010502de9c7fe29ea33a49813a0a2df1. caching signage point 62.
2021-12-09T11:02:36.178 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 1e7a3f416677e3b30f6f47108e3f99fb010502de9c7fe29ea33a49813a0a2df1. caching signage point 63.
2021-12-09T11:12:28.355 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 24c5acee5c7ad550f0e100c42db66c5e51eaae76b2c91eb40f8c0a9443af4f60. caching signage point 62.
2021-12-09T11:12:32.642 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 24c5acee5c7ad550f0e100c42db66c5e51eaae76b2c91eb40f8c0a9443af4f60. caching signage point 63.
2021-12-09T11:22:13.505 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 18ad88504ec46d0b3f8967d50f872b104c6b004e3f9ffc0c67c2f958547e2a40. caching signage point 61.
2021-12-09T11:22:15.973 full_node chia.full_node.full_node_store: INFO     Don't have rc hash a0de404af511e8673fa0f7272cf7684ff5d9a8628f21b9f3f7695025fd20ed3b. caching signage point 62.
2021-12-09T11:32:28.100 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 905e40586aac80e423f5f4f01a3096306ae75e92a174685cf70a40f57d1d7e1c. caching signage point 62.
2021-12-09T11:32:32.346 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 5816357d03fc99ef2fa897c9c662cbc6a9c290c03fdbdde2f4ac89dc8fb80138. caching signage point 2.
2021-12-09T11:42:11.429 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 1730d9521627eb0745182cdc459ead15f75fc937f54f1f5352c5e0f254016bd2. caching signage point 61.
2021-12-09T11:42:31.435 full_node chia.full_node.full_node_store: INFO     Don't have rc hash ac3c1640cdf142c05c8c053b5fecbb9e60b13571881017e23acfd86adcf57c44. caching signage point 63.

Restart seems to help for one or two days, then it happens again.

Any way to prevent that?

mcanto73 commented 2 years ago

I have the same issue with raspbian on pi4. It works 2-3 days and after stop farming. Chia farm summary doesnt respond and I need to restart . Stop/restart application seems not work

github-actions[bot] commented 2 years ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

peter92427 commented 2 years ago

This happens regularly and is not fixed.

github-actions[bot] commented 2 years ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

mahaupt commented 2 years ago

This is still happening

mariano54 commented 2 years ago

This might be related to your node being overloaded with too many transactions. Try decreasing your number of peers from 80 to 30 or so.

scrutinously commented 2 years ago

This might be related to your node being overloaded with too many transactions. Try decreasing your number of peers from 80 to 30 or so.

If this were the case, wouldn't I see something similar happening on forks being farmed at the same time? There is no indication of high load when this happens, it just happens for about 2 minutes for no apparent reason. I also track the mempool stat with grafana, and where this happens in the logs does not occur when the the mempool is high.

I usually see several smaller blocks of these entries:

2022-01-19T01:08:25.883 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f81d3984a84ccd808a2ff6846acd0ff2aa3db7abe590c73107f4ff7acdf71caf. caching signage point 46.
2022-01-19T01:08:34.753 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f81d3984a84ccd808a2ff6846acd0ff2aa3db7abe590c73107f4ff7acdf71caf. caching signage point 47.
2022-01-19T01:08:43.694 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f81d3984a84ccd808a2ff6846acd0ff2aa3db7abe590c73107f4ff7acdf71caf. caching signage point 48.
2022-01-19T01:08:48.369 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '197.200.3.63', 'port': 8444}.Inbound limit reached.
2022-01-19T01:08:48.581 full_node full_node_server        : INFO     Connection closed: 197.200.3.63, node id: 82be531f40b48a458b261c37235e32e592af09f98a58fab259a6fe4c888b2320
2022-01-19T01:08:48.581 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '197.200.3.63', 'port': 8444}
2022-01-19T01:08:52.586 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f81d3984a84ccd808a2ff6846acd0ff2aa3db7abe590c73107f4ff7acdf71caf. caching signage point 49.
2022-01-19T01:09:02.833 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 6c396fba2cb9acba49fee0536a983c9226eefb58064f6dc3067e73d68b989ed4. caching signage point 50.
2022-01-19T01:09:11.581 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 6c396fba2cb9acba49fee0536a983c9226eefb58064f6dc3067e73d68b989ed4. caching signage point 51.

Followed by a 2 minute solid block of these entries:

2022-01-19T01:11:18.337 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 9a3757694801dd83ade87ba4276f8c3fe2a97ba82f014152d1bb15199c500593, caching EOS
2022-01-19T01:11:18.338 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 4d947b50fd44dda69d0b5af4960b3451267ab1dcd774681a04f79befe133588c
2022-01-19T01:11:18.440 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 9a3757694801dd83ade87ba4276f8c3fe2a97ba82f014152d1bb15199c500593, caching EOS
2022-01-19T01:11:18.441 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 4d947b50fd44dda69d0b5af4960b3451267ab1dcd774681a04f79befe133588c
2022-01-19T01:11:18.461 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 9a3757694801dd83ade87ba4276f8c3fe2a97ba82f014152d1bb15199c500593, caching EOS
2022-01-19T01:11:18.461 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 4d947b50fd44dda69d0b5af4960b3451267ab1dcd774681a04f79befe133588c
2022-01-19T01:11:18.560 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 9a3757694801dd83ade87ba4276f8c3fe2a97ba82f014152d1bb15199c500593, caching EOS
2022-01-19T01:11:18.561 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 4d947b50fd44dda69d0b5af4960b3451267ab1dcd774681a04f79befe133588c
2022-01-19T01:11:18.570 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 9a3757694801dd83ade87ba4276f8c3fe2a97ba82f014152d1bb15199c500593, caching EOS
2022-01-19T01:11:18.570 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 4d947b50fd44dda69d0b5af4960b3451267ab1dcd774681a04f79befe133588c

This occurs at least once every other day, and sometimes multiple times per day, with no indication of cause in the logs. There are even a few network logs interspersed in the middle of the large block of the End of slot messages, such as peer connects/disconnects and size of mempool: 1 spends. It's almost always 2 minutes of doing this.

github-actions[bot] commented 2 years ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

github-actions[bot] commented 2 years ago

This issue was automatically closed because it has been flagged as stale, and subsequently passed 7 days with no further activity from the submitter or watchers.