[BUG] My Raspberry Pi 4 4GB currently misses / doesn't finish plenty of signage points in a row

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)

Apache License 2.0

10.82k stars 2.03k forks source link

[BUG] My Raspberry Pi 4 4GB currently misses / doesn't finish plenty of signage points in a row #1796

Closed flotti455 closed 3 years ago

flotti455 commented 3 years ago

Describe the bug There seem to be phases when my Raspberry Pi 4 GB farmer is missing several signage points in a row. As a consequence, the farmer is not participating these challenges.

E.g. around height 113661:

$ chia farm challenges
Hash: 0x52aceefed2de9a7653456e3906dd7cebd8eac57edcbcea28140741bac4689860Index: 1
Hash: 0x52aceefed2de9a7653456e3906dd7cebd8eac57edcbcea28140741bac4689860Index: 0
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 63
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 62
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 61
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 60
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 37
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 36
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 35
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 34
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 33
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 32
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 31
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 30
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 29
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 28
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 27
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 26
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 25
Hash: 0xc847981b22e0ade25ad4e8bdeee45839876c82dda55d90d3a5472330c4ac5d32Index: 24

Missed signage points 38 until 59, inclusive. That's 22 uninished signage points in a row.

I have already limited the number of target_peer_count to 30, guessing my Raspi may be overloaded during some p2p network bursts, but this did not resolve the issue.

I have added logging statements to find out under which conditions signage points don't get added. A failing signage point always seems to happen when verify_n_wesolowski doesn't manage to return True eventually, i.e., all wesolowski tries for this signage point return False.

Expected behavior All signage points shall finish.

OS: Ubuntu Server 20 LTS
Machine: Raspberry Pi 4 4 GB

flotti455 commented 3 years ago

$ ls debug-full.log.2021-04-11* | sort -n | xargs -d '\n' grep -in -e 'finished signage point' -e 'finished sub slot' | wc -l
8628

As an example, on April 11 local timezone, my Raspberry Pi (always on, always Internet-connected) only finished 8628 signage points. That is 588 SPs short of the expected daily 9216 SPs.

mariano54 commented 3 years ago

This might be related to the ordering in which the objects arrive to the node. Sometimes there is a signage point right after a block, but your node receives the signage point first, but is not able to process it because the block has not been received.

We need to debug if it's a problem of ordering, or a problem of node overloaded. Have you checked htop to see if your node is extremely busy (100% CPU) in the times when signage points are busy?

flotti455 commented 3 years ago

We need to debug if it's a problem of ordering, or a problem of node overloaded. Have you checked htop to see if your node is extremely busy (100% CPU) in the times when signage points are busy?

The Raspi 4 has four cores. Whenever a new signage point comes in, it does get busy on one core for a split second. But nothing out of the ordinary I would say.

Would it make sense to reduce target_peer_count even more down, e.g. to 15 peers?

Also let me know if it helps to provide you with my logs of a problematic day, and I can send them to you via e.g. keybase.

flotti455 commented 3 years ago

So i reduced target_peer_count down to 15. Unfortunately, after five minutes or so my Raspoi already missed finishing two signage points in a row. So I have the feeling that a peer count too high is not the reason for this issue.

grocheireland commented 3 years ago

is this being looked by any developer ? What do I need to check in the logs to see this on my pi4

flotti455 commented 3 years ago

What do I need to check in the logs to see this on my pi4

@grocheireland:

Set logging level on INFO

Print out and filter live logging with the following shell command:

tail -F ~/.chia/mainnet/log/debug.log | grep -i -e "eligible" -e "updated peak" -e "signage point" -e "end of slot" -e "updated wallet peak" -e "sub slot"

If logging looks similar to this ...:

14:04:08.525 wallet chia.wallet.wallet_blockchain: INFO     🌱 Updated wallet peak to height 133314, weight 1712291,
14:04:13.991 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 24/64: f5442e9443dca38b43256cd447334f250dd491f3b04aabcba64c85cb5bc04803
14:04:16.034 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 1a59a1edcc... Found 0 proofs. Time: 0.00100 s. Total x plots
14:04:21.953 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 25/64: c6cbef744bb0b95445601cedff233bb150195b01429f3823d33ae355bfe3fa95
14:04:22.973 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 1a59a1edcc... Found 0 proofs. Time: 0.00100 s. Total x plots
14:04:30.930 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 26/64: a51864582b1390220bef19ac29d1862779635410aab727f86a57cb09444727c5
14:04:32.973 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 1a59a1edcc... Found 0 proofs. Time: 0.00100 s. Total x plots
14:04:41.499 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 27/64: 292aa90e9fc8eeae608f03c974cbf545c8c75f877fc047c38f475441edaced5e
14:04:43.536 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 1a59a1edcc... Found 0 proofs. Time: 0.00100 s. Total x plots
14:04:48.008 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 133315, weight 1712311, hh 689bdbcb933516a4519aec57278961bbb2fd1104e2db70c8f0928fd62d8b586d, forked at 133314, rh: 2c1016c523aad54d4cb0408797abcde5c9e51555ee7a49ab3171bcca63dda756, total iters: 444819451599, overflow: False, deficit: 8, difficulty: 20, sub slot iters: 111149056
14:04:51.455 wallet chia.wallet.wallet_blockchain: INFO     💰 Updated wallet peak to height 133315, weight 1712311,
14:04:53.468 full_node chia.full_node.full_node: INFO     Signage point 28 not added, CC challenge: 1a59a1edcc813071a715e572c8420f16b1628ad717599b64adaab68dc4cec830, RC challenge: b99c7933449a321959993de6e8a8da19a32c6cdb7a118607474bce373bc3c60d
14:04:53.534 full_node chia.full_node.full_node: INFO     Signage point 28 not added, CC challenge: 1a59a1edcc813071a715e572c8420f16b1628ad717599b64adaab68dc4cec830, RC challenge: b99c7933449a321959993de6e8a8da19a32c6cdb7a118607474bce373bc3c60d
14:04:53.597 full_node chia.full_node.full_node: INFO     Signage point 28 not added, CC challenge: 1a59a1edcc813071a715e572c8420f16b1628ad717599b64adaab68dc4cec830, RC challenge: b99c7933449a321959993de6e8a8da19a32c6cdb7a118607474bce373bc3c60d
14:04:53.615 full_node chia.full_node.full_node: INFO     Signage point 28 not added, CC challenge: 1a59a1edcc813071a715e572c8420f16b1628ad717599b64adaab68dc4cec830, RC challenge: b99c7933449a321959993de6e8a8da19a32c6cdb7a118607474bce373bc3c60d
14:04:53.743 full_node chia.full_node.full_node: INFO     Signage point 28 not added, CC challenge: 1a59a1edcc813071a715e572c8420f16b1628ad717599b64adaab68dc4cec830, RC challenge: b99c7933449a321959993de6e8a8da19a32c6cdb7a118607474bce373bc3c60d
14:04:53.784 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 133316, weight 1712331, hh a82795f5fbc87e24ad3a949d5ba0797e0d0e1462cec51da80d47e7dd3b0008fd, forked at 133315, rh: b99c7933449a321959993de6e8a8da19a32c6cdb7a118607474bce373bc3c60d, total iters: 444820423557, overflow: False, deficit: 7, difficulty: 20, sub slot iters: 111149056
14:04:54.149 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 28/64: 7eec6859a4d54fbebe3f6b9f3fe00c08636f07da47b600390df3aca90ba1e38d
14:04:55.176 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 1a59a1edcc... Found 0 proofs. Time: 0.00100 s. Total x plots
14:04:57.273 wallet chia.wallet.wallet_blockchain: INFO     💰 Updated wallet peak to height 133316, weight 1712331,
14:05:02.525 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 133317, weight 1712351, hh a23e5460b9db9f7759d9b64ad06587a426947bb2340cd9ded363a6676e4c84c8, forked at 133316, rh: 0bda997d3d94a828e8bad32a344952ba2bbd40ed04df2c66ad275ff9abd46e49, total iters: 444822015346, overflow: False, deficit: 6, difficulty: 20, sub slot iters: 111149056
14:05:02.856 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 29/64: 8f6e29969f7e9d15128377e5b9b797dd63ff856982979d18ce08aa9cef3fd6d4
14:05:03.880 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 1a59a1edcc... Found 0 proofs. Time: 0.00199 s. Total x plots
14:05:05.370 wallet chia.wallet.wallet_blockchain: INFO     💰 Updated wallet peak to height 133317, weight 1712351,

... then everything looks okay: "Finished signage point" increases one by one, and even though there were Signage point 28 not added in between, they were recovered from as you can tell by the line "Finished signage point 28/64"

But if logging looks similar to this ...:

10:57:54.021 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 23/64: fac21ad39ef5be009d0cf9f1d1e6149b3147e80388373931d44e3fbcf1adaffc
10:57:54.090 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming fa40723098... Found 0 proofs. Time: 0.04732 s. Total x plots
10:58:03.205 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 24/64: 709f77fc661822613166bd9040a73dcc22ca97d6d81724b9b85965c0cfee0bcc
10:58:03.275 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming fa40723098... Found 0 proofs. Time: 0.04630 s. Total x plots
10:58:13.670 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:13.967 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.007 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.045 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.081 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.117 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.153 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.201 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.268 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.300 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.333 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:14.365 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:16.229 full_node chia.full_node.full_node: INFO     Signage point 25 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d
10:58:27.164 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 127869, weight 1604557, hh 227e26ef0850ccbf7f23c6b2f35c64070edaa123cb9261d9b8f737c40b19e56c, forked at 127868, rh: df617e11559abc7cc405c257438f7a3bc780692fff5c6196521241b3d75eb87d, total iters: 426678738779, overflow: False, deficit: 7, difficulty: 19, sub slot iters: 111673344
10:58:27.640 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.664 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.688 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.711 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.735 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.758 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.782 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.805 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.829 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.852 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.876 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.900 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:27.926 full_node chia.full_node.full_node: INFO     Signage point 26 not added, CC challenge: fa407230985baf23a137fbda71552574e3e679e98370c972be12f91dd354ba6a, RC challenge: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a
10:58:31.272 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 127870, weight 1604576, hh b395c25df1b0a2f4b227cdf4baf5b247f7027975ddd08925f0894950ad26581b, forked at 127869, rh: dd9f923f89942f3c376c4c16bd2517804dbd31c7333534695a761e0b8e45337a, total iters: 426679376785, overflow: False, deficit: 6, difficulty: 19, sub slot iters: 111673344
10:58:31.273 wallet chia.wallet.wallet_blockchain: INFO     💰 Updated wallet peak to height 127869, weight 1604557,
10:58:32.131 wallet chia.wallet.wallet_blockchain: INFO     💰 Updated wallet peak to height 127870, weight 1604576,
10:58:33.102 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 27/64: 96804f20df4de062495a3ac7514460f8050d1a0da0ed5fe1dd697b8a5a0c7d6e
10:58:33.175 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming fa40723098... Found 0 proofs. Time: 0.05127 s. Total x plots
10:58:41.924 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 28/64: c7f9e39f8edc4c0655a058ecf8827a9d862e04a66350558d23abd84628bc2191

... then things go wrong as reported with this issue here. It missed finishing signage points 25 and 26. Meaning your farmer didn't check if it is eligible for winning Chia for those signage points missed.

Handri-Kosada commented 3 years ago

I have this issue also on Pi 4 4 core with 8gb RAM.

From htop, nothing much going on the processor or other RAM / disk IO. Internet also stable with 100mbps line.

flotti455 commented 3 years ago

It seems I have found a solution to this issue for my setup.

My current theory for this issue is that there is too much filesystem IO going on on the Raspberry Pi's SD card when Chia's config directory ~/.chia is located on the SD card. After all, not only OS-related IO is going on on the SD card, but any Chia-related filesystem IO such as SQLite database IO and Chia logging. SD Card-based ~/.chia is set up when following the current Raspberry Pi Chia installation documentation, therefore I expect many if not all Raspberry Pi Chia users facing this problem with "normal" Raspberry Pi-recommended SD cards.

To bring this issue theory to test by resolving the issue, I changed Chia's config directory ~/.chia from SD card to a separate (and fast) USB 3.1 flash drive (thumb drive):

cd chia-blockchain

# stop Chia processes and daemon, e.g. by running
chia stop -d all

deactivate

# assuming a ext4-formatted, fast thumb drive mounted on /media/thumb
cp -r ~/.chia /media/thumb
# this will take a while, copying gigabytes of log and sqlite databases. You can monitor progress in a separate shell with `watch progress`

# we now have a copy of our Chia configuration on the external thumb drive located at /media/thumb/.chia
# now let's move the SDCard-hosted Chia config directory to another location on the SDCard
mv ~/.chia ~/.chia-backup

# ~/.chia is now available for setting up a symlink to the USB Drive-hosted Chia config directory.
# Let's set up this symlink:
ln -s /media/thumb/.chia ~/.chia

# start up Chia again

. ./activate

# use your usual way of starting up, e.g.
chia start -r node wallet harvester farmer

Monitor your logs. If everything goes well like it did for me, all signage points should now finish correctly:

$ tail -F ~/.chia/mainnet/log/debug.log  | grep -i -e 'finished signage point' -e 'finished sub slot'

# ...
14:40:21.853 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 2cd0c7e5b505108550bb7b4753705e8c88b1eeafc544ab506cc8148f6b8b2f47
14:40:28.769 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 2/64: 88cd84df1e9bb6cd86f620ea87945fe0c766155cfaa9d0fa19923eedeb6eb7a2
14:40:36.964 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 3/64: 4bb197f81feac9c3279502a23d40ef2e6de94381298704464ff29604d4ffe49f
14:40:45.423 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 4/64: 987646852027fa39991739500f0a8d181ce081ed7473e770896d40090b37015e
14:40:53.951 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 5/64: 11c2242f961b2737547295e913d7f15bb19e14eadde875ef2ca75defe5107e3a
14:41:02.546 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 6/64: 42d24d8e56bda7ce9e791173bad0bc64dbc25816f3c006812c9df180b09340d5
14:41:11.190 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 7/64: e6dde889876f61e3c9699e90540e20ec7362eb3152e5940d08374bb41c77ff93
14:41:20.046 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 8/64: 47681dea271782ce8aa0c7f9ec2ba5b58fd28565cb956f0c9b1da7c30c64604b
14:41:29.609 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 9/64: 95a0929d553af1917628f76eacecd93268d8d384cb80eac5dbdfb971cac00f30
14:41:44.377 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 10/64: 50362f29a23c56cb4d109325bb9944fb187758e24aab101d1efa4d55bd403334
14:41:49.594 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 11/64: 604c6f9c2686bd612c35b2889e5e17cc126fc54ec4009d27be282335abe4ea3f
14:42:00.718 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 12/64: 4648d3fe29f22bc3c99c3e2bc6fa557931ac9a35ec05960f43fc8de32ad1dc20
14:42:10.534 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 13/64: 553af63a7de12a0a26f65236571ff0c28d92b321aac22635f59ea968e8cad5a2
14:42:18.691 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 14/64: ff3bab6360b21362d06aa9136397a652bebe32cfd6caf5dbeb828e7f162169f0
14:42:29.686 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 15/64: a93fc2ee6ca67c73fd96a0afb64d7b95c180f98111e64ee8313380bebcbc0afa
14:42:40.338 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 16/64: b94f40f29cf0efc37149614111f6b6f4b75ec3a26e74106503d40a14a34a9198
14:42:48.413 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 17/64: 2b8f976206cf955cc1d98f68e9c82f99ccc40b01be7324e6a367c5ffc66e5a09
14:42:56.613 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 18/64: e8f81b41b6ed5a18bf6307e97450f528490adc72ddbf08b0e6171dc89ff31a6e
14:43:05.596 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 19/64: 3aa204ec62f11f73190d24be0ef5fa018e6eb89d6087f0874df6e1cd773c55ce
14:43:14.429 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 20/64: 05f986f27221f2aaaf5ba5fd520dc21f0df9f252049c15044831f3441516e0b8
14:43:25.312 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 21/64: 205082a8489c143e1625b444455ff672eb403b92b65bb0ba1eb6ba7c46a07c77
14:43:33.129 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 22/64: f29737fa561e0b49e45c81139fec81f02792b5d3afbd056888445e047843ac89
14:43:42.656 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 23/64: d0e4bc2e061ba302a9ccfee48a556421eeaa27bcc7491f7d7cd0b7c5b4d8c9b8
14:43:53.444 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 24/64: b60d8dfad0c2801efcfc43d0df97bf76a5eaf1cddf7357c8b83eef9c0b170585
14:44:02.620 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 25/64: e7d35d0f6fce31d198635dba7329c0415aee2292f1db8bbb5eefa4b20de2ff9b
14:44:10.945 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 26/64: d8d7076944aa21210da046befc384977423198c5acacad6ea144e6d1c397f69c
14:44:19.960 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 27/64: 175616760ab0b6b68f7ff84bfd14e5d276e8cd8bf5e63b6f58f164bf789463ab
14:44:29.837 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 28/64: 330643eec4ea8b504e2a926bb0f4b6f43c37527fbe25bf5608ab7ab7ecfad6e8
14:44:39.324 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 29/64: 6338882650b27255b659348d12035bd23ad5fb2b772ab384c945baf7e50ec74a
14:44:51.346 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 30/64: 83d6550c046a21eeb3be6b4a61e525db94cee4d77f0067b87d02a3c1e5a2efae
14:44:59.673 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 31/64: d827ddb7b063d1ae6d2945d5a362c68ad630dba3a9143adf720af9c8f31069a2
14:45:09.719 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 32/64: 09e7980ffe6934e01d3fe29028e222244f7b51169ae7ecc029d82091c7c2b062
14:45:18.944 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 33/64: 4a769a8cb07494bec6483a595e712d310feb0accf4f6cfc413eac204fac53fe4
14:45:29.082 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 34/64: 57a4fbd6ce9d1887456dbdf558bb3a457bfbdb21cab4d79fcab66cf962b8dffb
14:45:39.912 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 35/64: 9e6f274415f9ba5d3f188529c1fbd1ffb655069c6bb08cee4bc10dc64d8de08a
14:45:47.710 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 36/64: 753af2a0fb607fa6eab58200096170e30098033bfd5d2a639d235cb613d2b204
14:45:57.915 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 37/64: 301e75efba9a26323bb186be00edfba257a1515d70bef99a92d0bb8a9b85b170
14:46:06.886 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 38/64: e3d76504ebc907cfd628a3d8b67833a114cfce7a1932568f4d23e320776494a4
14:46:16.643 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 39/64: f15721315550bda6ac1d76ce6c6201db8b61f6de1ab308e4ff905bc6291406f5
14:46:24.306 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 40/64: 4679e82bb1a1019e9a06eef60c96f673055987b16e72f849949de2c8c700cf98
14:46:32.652 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 41/64: 9a3b15accbd099941ee27fc8f1b87f80426002a00c25e16294dc17b543d7cb84
14:46:41.346 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 42/64: 82fa34e51c580abd76e0a5c2ec79c4f7e5ccc9689b28125ba5a088c1e231b0d3
14:46:51.384 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 43/64: 4ca57cc2b71a14c9ff71d9e221b64c9cbee6793b46c6800455e76cff33fb4289
14:47:01.698 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 44/64: ef61022f6eb670d4c3b4d0d79c953d497a4d55d61fc731129fc4d808148afed5
14:47:13.811 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 45/64: e0e0b36310d8fff02d051f915d2d24f892a998badabe813d0fbe40836aa28577
14:47:21.997 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 46/64: 715a2550fcec3264785479c6da1550ea6916812384919fee1bd3face843a9712
14:47:31.314 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 47/64: a07de247c58bd91aed6af183d79a4bdf53d68ceec0d7e769d44120cf0950e75a
14:47:40.013 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 48/64: 76a674197f55a7e4183b3d38a78332f7fb8ec3ca2f542ed425bd2d76149e40ba
14:47:52.103 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 49/64: 8dbbcda39a74eced6a3c995c2683805268837af43e981cb5116d4bc0bb0259d7
14:48:03.544 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 50/64: 4b811d10982726b26d84847a90312d18840f689620428fb5967b35916bc1edfe
14:48:11.410 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 51/64: 9703ef32b8ee9f20a16ed3abe30da745b6384cb0f92160e26b6338c3bb55552f
14:48:21.391 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 52/64: 254f7754adc93cc7b3fc308717f8430ed0f93cbc1fa88a9bb2d312dfa6872558
14:48:29.334 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 53/64: d4fdec0ffe55903f7cf2f9f810c4467b304cbb6a527012406f2ecf2c2ed2a8de
14:48:38.155 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 54/64: 859e5a4a2afeab845bbeebaf0c340d7995304115a727be635c518ce5d94413a5
14:48:48.253 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 55/64: a987b247e1d1bb16766cc38e5712c9776c27088ed3898b152080639094814738
14:48:56.283 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 56/64: 5a6ca4278e5ff095f9d223cdebb56aefc9e386f48a9a08138f0f2d0d07dafc01
14:49:05.101 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 57/64: 007ca332d998a58d0b3d5554d38471d63b99f9d9083fb4ecd3c38ea83672a075
14:49:15.318 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 58/64: a01e7a7810ef40d530320d013aadaa109030b0c45c5239da8fdfea508c125cd0
14:49:23.207 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 59/64: bbf1d59f1cba54587a4ba716c7b1ac31bd22474211fe8246e1986b239b077252
14:49:31.523 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 60/64: 8496409d140f14e30a44b497fb09c8ba2a4e53d8a6bd4ae2380a743d86d723e5
14:49:40.328 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 61/64: 2d31ac8065c16fb8981ca35fac39296eb792cb10ad14368e41f43079b7c4ba10
14:49:48.988 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 62/64: aa24fe8c5beb815c18bbea778151d81db5bb84301fb89b2336543a52dbabc499
14:49:57.891 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 63/64: b036085b13edd400334e4f16d1f1420172bc594cc77e5dee393672c6c9d0c200
14:50:06.695 full_node chia.full_node.full_node: INFO     ⏲️  Finished sub slot, SP 64/64, a3904419619f1bd96e1fc924bda4010916b5a72e9d84000f74944c264bc6dbd2, number of sub-slots: 2, RC hash: cab82e36be665098de0b47110e5672c51749a7dc2ae7b7ab45c8588f4e4e98f1, Deficit 16
14:50:17.094 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: c5ea35d58ae5f76bf42c4168faac5f31873b00e7f45866a7ced7cdda2dc15235
# ...

From 1 to 64, all signage points within a sub slot now finish.

I have been using this solution for four hours now and so far haven't missed finishing a single signage point. In two days, I will be able to report back full-day statistics.

In case this solution (i.e., moving ~/.chia onto a separate device) turns out to be a viable solution, and other Raspberry Pi users facing this issue can resolve it by following this solution as well, I recommend updating the Raspberry Pi Chia installation documentation accordingly.

P.S.: In case you added an additional external drive because of this and this solution works well for you, in all euphoria don't forget to mount this new drive on boot by adding a new entry to /etc/fstab.

P.P.S.: When things have worked well for some time, it will be save to free space on the SD card by deleting ~/.chia-backup, though it's always nice to have some backup of your SQLite databases around in case the live versions get corrupted for some reason.

dorkmo commented 3 years ago

wonder if a higher performance sd card will still have this problem? is it simply the bus being the limiting factor?

Handri-Kosada commented 3 years ago

If I lost 1 SP, will I lose to WIN all the 64 SP in that block? or just that 1 SP only?

I just updated to 1.0.5 for 3 hours, so far no lost in SP. Hope it is somehow fixed in 1.0.5

Thanks.

grocheireland commented 3 years ago

Thanks flotti455 Yes I am seeing the same prob on a pi4, but not all the time. My pi does NOT rely on an SD card either. Also I tried keeping the pi busy/overloaded but couldn't as of yet see any correlation between that and missing signage points.i ll look again on Saturday

heenning commented 3 years ago

First of all thank you @flotti455 for your effort. Our setup is Pi 4 with 8GB RAM and 128 SD Card on Ubuntu 20.04 LTS Server. I switched the location of .chia as you described in your comment but last night I received once again error message from the CHIADog:

[2021-04-15 22:14:38] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 27/64, current 29/64. (finished_signage_point_handler.py:89) [2021-04-16 00:20:43] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. (finished_signage_point_handler.py:89) [2021-04-16 01:11:26] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 1/64, current 3/64. (finished_signage_point_handler.py:89) [2021-04-16 01:13:28] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 14/64, current 16/64. (finished_signage_point_handler.py:89) [2021-04-16 01:17:17] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 38/64, current 40/64. (finished_signage_point_handler.py:89) [2021-04-16 02:23:38] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 17/64, current 19/64. (finished_signage_point_handler.py:89) [2021-04-16 03:19:38] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 58/64, current 60/64. (finished_signage_point_handler.py:89) [2021-04-16 04:39:24] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 53/64, current 55/64. (finished_signage_point_handler.py:89) [2021-04-16 05:29:58] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 57/64, current 59/64. (finished_signage_point_handler.py:89) [2021-04-16 06:02:28] [ WARNING] --- Experiencing networking issues? Skipped some signage points! Last 11/64, current 13/64. (finished_signage_point_handler.py:89) So it seems that we miss out some signage point randomly.

Handri-Kosada commented 3 years ago

is this problem missing signage only happen on Pi 4 ? Normal desktop no issues?

mariano54 commented 3 years ago

There is an additional issue which affects everyone (not just RPI users). That is that if you receive some blocks late, due to networking issues, the signage points you receive might not be verified correctly, since they depend on the last block. I will make a fix for this that will hopefully make it into 1.1

grocheireland commented 3 years ago

Thanks flotti455 Yes I am seeing the same prob on a pi4, but not all the time. My pi does NOT rely on an SD card either. Also I tried keeping the pi busy/overloaded but couldn't as of yet see any correlation between that and missing signage points.i ll look again on Saturday

@grocheireland , what do you mean with "My pi does not rely on an SD card"? Does it not use any SD card at all? If so, does the operating system plus Chia run on an external drive such as a thumb drive? If so, maybe the bottleneck then is just shifted from SD card to slow thumb drive.

@Abdull I have an argon m2 case and then internal SSD and it boots from that https://www.argon40.com/argon-one-m-2-case-for-raspberry-pi-4.html

flotti455 commented 3 years ago

@henningrettenmaier (and @grocheireland), how many SP did your Raspberry Pi miss before you moved ~/.chia to another device?

From your pasted log, your Raspberry Pi seems to be missing around one signage point per hour. That is already a lot better than missing around 25 signage points per hour, as my Raspberry Pi did before moving the directory.

From your pasted log (and assuming you are in German time zone), only one missed/unfinished signage point happened very close on my side as well:

debug-full.log.2021-04-16_06-00-03:943:06:01:43.254 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 8/64: cd5585d87db3788b8f2d9789cb97d1d5dfa9270e0c9c07f3be177c8e15441e36
debug-full.log.2021-04-16_06-00-03:1017:06:01:51.562 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 9/64: 6f48573420969cd6bb4a91db052f109e01a5964fd3e9fce6c691cceb1c581d18
debug-full.log.2021-04-16_06-00-03:1274:06:02:20.726 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 12/64: 8813c79c852eef450a33e3b25fccf0a62d196bdc348bc6f46a35b857dfb346e8
debug-full.log.2021-04-16_06-00-03:1362:06:02:28.703 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 13/64: cc0e07b5e938e1e04bc00e095e8253edb203ab0dbc6b01c485228117352f3001

I missed signage points 10 and 11, you missed signage point 12.

Also, in Keybase, a couple of days ago people in Europe reported missing identical signage points, e.g. I can confirm my farmer missed the same signage points that @martomi did (keybase://chat/chia_network.public#random/22366).

So far, we can already tell that there are at least two independent causes for missing signage points: problematic hardware (probably because of slow SD card IO), and P2P network "hiccups" (?) several people experience.

Handri-Kosada commented 3 years ago

1.0.5 still have the issue.

Tried with dell server also have the same issue.

Hope this can get fixed asap in the next release. Else the win chance is droping.

dorkmo commented 3 years ago

looks like improvements in #1978

mariano54 commented 3 years ago

Yes the fix will be in the 1.1 release

flotti455 commented 3 years ago

At least with 1.0.5, I noticed that there is statistically less signage points 64/64 (== end of sub slots) finishing than signage points 1 to 63:

When a signage point 64/64 is not finishing, logs look similar like this:

21:05:29.794 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 96dd52dd84638b63dd23f4f5dd4b10affb4ced2eff05c64da1c9daf9d112219a
21:05:29.795 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 682b3216b6458559965b0c69f29b7f2255f5ded32180b1cda7c3960b2133cc89
21:05:29.797 full_node full_node_server        : INFO     <- new_signage_point_or_end_of_sub_slot from peer 3d269f5ffb05dc2dd912ef94b8c2198e5efc8c2915cd63784fed529c83c200d7 51.15.193.248
21:05:29.800 full_node full_node_server        : INFO     -> request_signage_point_or_end_of_sub_slot to peer 51.15.193.248 3d269f5ffb05dc2dd912ef94b8c2198e5efc8c2915cd63784fed529c83c200d7
21:05:29.816 full_node full_node_server        : INFO     <- respond_end_of_sub_slot from peer e46641a8f6c6880413ec04303e02eed061839d7ec53108ae7e90299888fb2b5b 67.160.227.80
21:05:29.821 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 96dd52dd84638b63dd23f4f5dd4b10affb4ced2eff05c64da1c9daf9d112219a
21:05:29.822 full_node chia.full_node.full_node: INFO     End of slot not added CC challenge 682b3216b6458559965b0c69f29b7f2255f5ded32180b1cda7c3960b2133cc89
21:05:29.839 full_node full_node_server        : INFO     <- respond_end_of_sub_slot from peer 3d269f5ffb05dc2dd912ef94b8c2198e5efc8c2915cd63784fed529c83c200d7 51.15.193.248
21:05:29.844 full_node chia.full_node.full_node_store: INFO     Don't have challenge hash 96dd52dd84638b63dd23f4f5dd4b10affb4ced2eff05c64da1c9daf9d112219a

coderasm commented 3 years ago

@flotti455 do you still recommend moving .chia to a separate usb 3.0 drive? This problem is completely destroying my farming ability.

flotti455 commented 3 years ago

do you still recommend moving .chia to a separate usb 3.0 drive? This problem is completely destroying my farming ability.

@coderasm, in case your farmer misses hundreds of signage points per day, I recommend giving it a try to move ~/.chia/ on a separate drive. Since I have done it, I'm experiencing only a few missed signage points per day with version 1.0.5.

jon-sahi commented 3 years ago

Post 1.1.1 release, I think there are still problems with signage points. I am running full node, farmer and harvester together so all are on 1.1.1 On a Pi, running Ubuntu, I see a few end of slot points missed, some other signage points completed out of order, and duplicate "Finished signage point" messages.

eloneill commented 3 years ago

I'm getting this running 1.1.1 on Windows 10. Looks like I'm catching about maybe half the signage points? Maybe not even that....

2021-04-22T13:05:13.465 full_node chia.full_node.full_node_store: INFO Don't have rc hash c36104f57ce38d208f8217a2c583ac203670834da772960b8deb03ff01610c64. caching signage point 40. 2021-04-22T13:05:13.468 full_node chia.full_node.full_node: INFO Signage point 40 not added, CC challenge: 0e00979ecb39cd707f308902352e0d7521beaf9c10b4e5883c49db7d16edf165, RC challenge: c36104f57ce38d208f8217a2c583ac203670834da772960b8deb03ff01610c64 2021-04-22T13:05:13.491 full_node chia.full_node.full_node_store: INFO Don't have rc hash c36104f57ce38d208f8217a2c583ac203670834da772960b8deb03ff01610c64. caching signage point 40. 2021-04-22T13:05:13.493 full_node chia.full_node.full_node: INFO Signage point 40 not added, CC challenge: 0e00979ecb39cd707f308902352e0d7521beaf9c10b4e5883c49db7d16edf165, RC challenge: c36104f57ce38d208f8217a2c583ac203670834da772960b8deb03ff01610c64 2021-04-22T13:05:13.515 full_node chia.full_node.full_node_store: INFO Don't have rc hash c36104f57ce38d208f8217a2c583ac203670834da772960b8deb03ff01610c64. caching signage point 40. 2021-04-22T13:05:13.518 full_node chia.full_node.full_node: INFO Signage point 40 not added, CC challenge: 0e00979ecb39cd707f308902352e0d7521beaf9c10b4e5883c49db7d16edf165, RC challenge: c36104f57ce38d208f8217a2c583ac203670834da772960b8deb03ff01610c64 2021-04-22T13:05:13.526 full_node full_node_server : INFO <- respond_signage_point from peer 18a481e917ed0e79b3238f7335523585bd51468829a08517b4d726c7b1f0a241 101.78.169.19 2021-04-22T13:05:13.530 full_node full_node_server : INFO <- respond_signage_point from peer c60244af7e906687816240fcec4fe1dfa9e45a81cfcb25fc0f45478b3c36a265 218.66.226.152 2021-04-22T13:05:13.532 full_node full_node_server : INFO <- respond_signage_point from peer 1588247bb126dde16903febe358ba689885f30a63f4d58ac27907bc0c44c255f 115.221.241.38 2021-04-22T13:05:13.535 full_node full_node_server : INFO <- respond_signage_point from peer 0963f40ab609511c295b6849df6c1a1c9f3c41d74be004477d6cb34d78c59869 211.46.25.96 2021-04-22T13:05:13.538 full_node full_node_server : INFO <- respond_signage_point from peer b7e4b8a22961e503b09c87ed23a9d19cace5c9f03dccd2d7b743c2f01c86842e 118.168.140.16 2021-04-22T13:05:13.541 full_node full_node_server : INFO <- respond_signage_point from peer 6a2ac58e011e2f3794e66ed1954fb8bb0ad79b96514c91c95648f5a3cb2fc6df 112.97.214.214 2021-04-22T13:05:13.544 full_node full_node_server : INFO <- respond_signage_point from peer cbf19f92d3ae64cc2d35e480b38c5d1ea2169768f38b941ca59faf017db9a230 110.247.251.160 2021-04-22T13:05:13.547 full_node full_node_server : INFO <- respond_signage_point from peer e189bf30c2a277c9d369ed85228363268520901709263133cd83d097c514cec9 75.142.216.187 2021-04-22T13:05:13.550 full_node full_node_server : INFO <- respond_signage_point from peer 07f145116097358700f0f29cc2dc685a0286c3d21fd92ee7fdbf05e49418b2e2 95.174.105.199 2021-04-22T13:05:13.554 full_node full_node_server : INFO <- respond_signage_point from peer c4c0502c79612d960872a3ced662195075c42e98028f3a6aad3eae5086b4d77f 109.191.38.24 2021-04-22T13:05:13.556 full_node full_node_server : INFO <- respond_signage_point from peer 9b81d78862641866432c832b2cd9b6b69e6cb4c12bea39340d7c14750c6a260f 117.63.223.225 2021-04-22T13:05:13.560 full_node full_node_server : INFO <- respond_signage_point from peer 5367eae22d2b7836bb5562e8577af6395cc9f019baf7229a1dd2871769ea2fc2 180.108.197.24 2021-04-22T13:05:13.562 full_node full_node_server : INFO <- respond_signage_point from peer e783d96545645e47d3818cadf90866788e3e63da09045c25c528d72f6b8748f0 94.158.7.110 2021-04-22T13:05:13.565 full_node full_node_server : INFO <- respond_signage_point from peer 9bc64835b67ae2b8482160df9bfe731dfa1b0d236d88da3c45578b278ef8a81b 219.152.135.13 2021-04-22T13:05:13.568 full_node full_node_server : INFO <- respond_signage_point from peer ef9c4e6a65d542996a36a8ec6947b2191a9db695bd6e8bb58f66f4b02dd0bead 27.46.237.26 2021-04-22T13:05:13.593 full_node chia.full_node.full_node_store: INFO Don't have rc hash 413534d0ff5a17143c814b29148d4d3de623eb753d7935972a9cfaa1d625afd9. caching signage point 41. 2021-04-22T13:05:13.596 full_node chia.full_node.full_node: INFO Signage point 41 not added, CC challenge: 0e00979ecb39cd707f308902352e0d7521beaf9c10b4e5883c49db7d16edf165, RC challenge: 413534d0ff5a17143c814b29148d4d3de623eb753d7935972a9cfaa1d625afd9 2021-04-22T13:05:13.617 full_node chia.full_node.full_node_store: INFO Don't have rc hash 413534d0ff5a17143c814b29148d4d3de623eb753d7935972a9cfaa1d625afd9. caching signage point 41. 2021-04-22T13:05:13.620 full_node chia.full_node.full_node: INFO Signage point 41 not added, CC challenge: 0e00979ecb39cd707f308902352e0d7521beaf9c10b4e5883c49db7d16edf165, RC challenge: 413534d0ff5a17143c814b29148d4d3de623eb753d7935972a9cfaa1d625afd9 2021-04-22T13:05:13.646 full_node chia.full_node.full_node_store: INFO Don't have rc hash 413534d0ff5a17143c814b29148d4d3de623eb753d7935972a9cfaa1d625afd9. caching signage point 41.

System resources look like this (system drive is C:, plotting on E & I)

flotti455 commented 3 years ago

@eloneill, stop plotting and see if this resolves missing signage points. Plotting being a CPU- and IO-intensive task can starve your farmer, full node, and harvester from precious IO resources needed for handling signage points.

eloneill commented 3 years ago

@flotti455 That was painful. Yes, it's a better, but it still missed 2 of the last 30.

Quick update: I rebooted (hadn't rebooted after the upgrade to 1.1.1, and now I'm still getting the signage point not added errors, but typically those signage points are finishing now after a few misses. So I think restarting helped. But I'm slowly ramping back up my plotting, so not sure if it will come back or not.

grocheireland commented 3 years ago

Hi, with the latest version, on my pi4, I have not missed any signage points today, even when the pi was busy copying a new plot file,across the network, from my harvester. 👍👏

flotti455 commented 3 years ago

With version 1.1.1 I noticed the previously often-missed signage points 64/64 ("Finished sub slot, SP 64/64") are not missing so often anymore. But within a period of 24 hours, my Raspberry Pi still missed a SP 64 one time, just as it missed other signage points. Still, it's a significant improvement compared to 1.0.x.

I also noticed another phenomenon (all times Europe/Berlin):

$ grep -i -e 'finished sub slot' -e 'finished signage point' -e 'orphan' debug-full.log.2021-04-24_06-00-07
...
2021-04-24T06:04:52.708 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 62/64: 6b97c0e5efe94428454aa21cad715375d680962faddc67dd8a73256b717680ef
2021-04-24T06:05:00.944 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 63/64: b4cf9771a7b5d874bf67e60423ae1b06e767a952d3dd69c0e9b6ea0bb3683722
2021-04-24T06:05:09.824 full_node chia.full_node.full_node: INFO     ⏲️  Finished sub slot, SP 64/64, 5f0de559a89e5a56ecee339e7b4638560172ae810dc0522d02e31dc1faa51d2a, number of sub-slots: 2, RC hash: 1c0c5975dcf191052a71a67efc6da294a9cd2f34982bbcf374fcae68574e2ae0, Deficit 16
2021-04-24T06:05:18.733 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 95b61dcb4cce5d8241ea5d3a05e6dd53384cc049cccf05a90afb309029af6bea
2021-04-24T06:05:23.916 full_node chia.full_node.full_node: INFO     ⏲️  Finished sub slot, SP 64/64, 5f0de559a89e5a56ecee339e7b4638560172ae810dc0522d02e31dc1faa51d2a, number of sub-slots: 2, RC hash: bef8bfe1c03e72888461a8d430a0ece6d861cf177cb2742f23c4ec9246df97b8, Deficit 16
2021-04-24T06:05:36.589 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 95b61dcb4cce5d8241ea5d3a05e6dd53384cc049cccf05a90afb309029af6bea
2021-04-24T06:05:43.327 full_node chia.full_node.full_node: INFO     Received orphan block of height 176725 rh 8c608937139e2069e2973ff811fbc6b7ef5e566aa8faff4aa3125edbb774a5c7
2021-04-24T06:05:45.831 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 4/64: 125ddeac7fe1f3088e4773bb393d38db782f7d6f1929246275716f463e8eeb47
2021-04-24T06:05:47.071 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 95b61dcb4cce5d8241ea5d3a05e6dd53384cc049cccf05a90afb309029af6bea
2021-04-24T06:05:48.066 full_node chia.full_node.full_node: INFO     Received orphan block of height 176726 rh 85e38db72c357b92d33565af9fee64497852ccb9a5aa6f17291aa882d77a9f4d
2021-04-24T06:05:55.027 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 5/64: 235dff2a4d3c395bd184ae3be7fad2a7940ddb4f8f8442767562d21750aaa29b
2021-04-24T06:06:03.375 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 6/64: 5ba2653d5ce37a64b79de91561f80991d0ca34c11f50da437706b7d1436d2f6b
2021-04-24T06:06:12.742 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 7/64: 2d83f29c7fddc1c846c82e401dc1cdcbf3c2a96a6509eb0e3cdfc2a510d99b3f
...

Notice the repeated signage points 64 and 1. 64 is processed twice, 1 is processed three times. On the other hand, signage points 2 and 3 are missing. Also notice the "orphan blocks".

hoffmang9 commented 3 years ago

Which version of Chia?

jon-sahi commented 3 years ago

I have been using chia latest (currently commit 82c39c40), for about 10 hours, I am still seeing a few missed signage points, and duplicate / out of order "Finished signage point" messages.

heenning commented 3 years ago

Seems to me that currently I'm loosing just one point but always the last one:

ChiaDog, [23.04.21 10:18] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [23.04.21 12:39] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [23.04.21 13:19] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [23.04.21 15:09] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [23.04.21 21:39] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [24.04.21 00:09] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [24.04.21 06:15] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [24.04.21 12:23] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [24.04.21 13:22] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [25.04.21 04:03] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 3/64, current 2/64. ChiaDog, [25.04.21 10:43] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64. ChiaDog, [25.04.21 13:55] ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 63/64, current 1/64.

dorkmo commented 3 years ago

might update to 1.1.4 today

SKYBILLS commented 3 years ago

me on 1.1.4 version still seeing missing signage points and RC and i am using pc windows 10 , GUI

salsaly4 commented 3 years ago

The same thing on 1.1.5, Linux full node w/o any plotting on it, just farming. .chia directory resides on HDD. No networking problems seem to happen.

Handri-Kosada commented 3 years ago

CHIA 1.1.5 pi4 still missing the 64th SP below.

My AMD rig running on the same network is fine and got the missing 64th SP on pi4.

2021-05-11T14:50:17.690 full_node chia.full_node.full_node: INFO ⏲️ Finished sub slot, SP 64/64, 0220e27eac138468450634a8720484c2399b660c873fee8c5a4c1c743b7d7eb9, number of sub-slots: 2, RC hash: 4d57bacc8a8a1737378ce1e7c797faa0e96d1a3fdb15e1f115be02ce65796956, Deficit 16

erdincsk commented 3 years ago

@mariano54 Does this issue still exist on 1.1.5? I have limited 30 connections, nothing wrong with my internet connection and still getting missing signage points. I have 2 harvesters (1 on mac + 1 on Ubuntu) + 1 node / farmer (on Ubuntu)

It would great to give an actual feedback.

Thanks a lot.

lwndow commented 3 years ago

Adding my struggles with signage points here as well. My setup:

UnRAID 6.9.2 (kernel 5.10.28) (Intel i9 7900x w/32G ram)
Chia 1.1.5 in container (full node)
Plotting also done from inside same container
.chia is on NVMe raid1 pool (BTRFS, logs (INFO), mainnet config)
peer_connect_interval: 30 and target_peer_count: 60
Plots are written direct to disk pool (XFS, no NVMe)
1G link to server, shows no errors
Uplink is ~25up/450down, generally no issues outside of occasional congestion in neighborhood
CPU usage is generally never over 15%, not parallel plotting

With plotting stopped to avoid any possible contention (shouldn't exist in the above beyond CPU), still appears to see dupes/out-of-order in the logs:

2021-05-11T12:55:16.766 full_node chia.full_node.full_node: INFO     ⏲️  Finished sub slot, SP 64/64, f1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64, number of sub-slots: 2, RC hash: 7e132bedd94ad7c600776e3d61d0f8155406edf07739ab021ca914eca1ece87c, Deficit 16
2021-05-11T12:55:25.708 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 2d22e00f787fbc50da9f2a672907468fb807442c9d4596b8edcb4e8f191b84c4
2021-05-11T12:55:33.239 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 2/64: 6a34db5a21f39c93ce6a7aa59c350900461a09c9eadc9a093c715a4af88df9f8
2021-05-11T12:55:42.023 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 3/64: a71a2cfcb1431f6f714c04075bc97d41be281a40f3df0de36de73234305b73e3
2021-05-11T12:55:52.816 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 4/64: 26303286bc814863f1a22d7155134acaeafafcd3c7b63e2b9d3c4faf13b90044
2021-05-11T12:56:04.146 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 5/64: e2bc14963050f42f43d323124c94aa17992d17692e32b27205e9dc5b2addaf04
2021-05-11T12:56:11.390 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 6/64: 2a02db7b88cd1ea43928a6023ed3080ae550e839f3bd334f028e2a1d510233b4
2021-05-11T12:56:19.739 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 7/64: 6fcba1acb8edf5c026bbadf34fd8716537ddcdd94b098ff1f80df3e715eb9c4c
2021-05-11T12:56:28.294 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 8/64: a9c89d7e8a55b9a67103416e86ab7b77818c89965fe479fd4cbd57841f56f324
2021-05-11T12:56:37.389 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 9/64: 03c7984e18bf858309ee52e6dbc30a5c0c7b343a9ef9c47f42f902d4ca904139
2021-05-11T12:56:47.884 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 10/64: 79b71cd7ea3041fb07796da005b5cdb39c8616bd34a8e2e80942a88d82f7dda8
2021-05-11T12:56:51.258 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 10/64: 79b71cd7ea3041fb07796da005b5cdb39c8616bd34a8e2e80942a88d82f7dda8
2021-05-11T12:56:51.309 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 9/64: 03c7984e18bf858309ee52e6dbc30a5c0c7b343a9ef9c47f42f902d4ca904139
2021-05-11T12:56:51.563 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 5/64: e2bc14963050f42f43d323124c94aa17992d17692e32b27205e9dc5b2addaf04
2021-05-11T12:56:51.620 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 7/64: 6fcba1acb8edf5c026bbadf34fd8716537ddcdd94b098ff1f80df3e715eb9c4c
2021-05-11T12:56:51.621 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 8/64: a9c89d7e8a55b9a67103416e86ab7b77818c89965fe479fd4cbd57841f56f324
2021-05-11T12:56:51.808 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 4/64: 26303286bc814863f1a22d7155134acaeafafcd3c7b63e2b9d3c4faf13b90044
2021-05-11T12:56:52.832 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 3/64: a71a2cfcb1431f6f714c04075bc97d41be281a40f3df0de36de73234305b73e3
2021-05-11T12:56:53.197 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 2d22e00f787fbc50da9f2a672907468fb807442c9d4596b8edcb4e8f191b84c4
2021-05-11T12:56:53.241 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 2/64: 6a34db5a21f39c93ce6a7aa59c350900461a09c9eadc9a093c715a4af88df9f8
2021-05-11T12:56:59.480 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 11/64: ee46ab94f82ebe1e7e620fcb65b53f61e14badff5698c5c6d1dedf1c0c1606ea
2021-05-11T12:57:08.291 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 12/64: 8d3962166365d42b04e0ebec1ec275fba3437afc78a97a2c8d7ff264d39a1c6a
2021-05-11T12:57:17.648 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 13/64: ae01f1dd632ec125154a2e0fbaaefae5136751d1cd8a66dcc054feaf2a6afc71
2021-05-11T12:57:25.884 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 14/64: d38d0e366769e8e8a4da1ef07ca986398617f0b01397035c1efcb795e8353b1c
2021-05-11T12:57:34.770 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 15/64: e73e17547a879a3102afbc5d2d3276759736ca2487e53d420bfa9dc2a288f89a
2021-05-11T12:57:43.391 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 16/64: d4b80abaaf65486f7090e398345a84f9c48f09fdab92fc9aa40f361fdd7055fb
2021-05-11T12:58:02.060 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 17/64: eb4a105411e50b97cc953725e902440d8e41db650731dee06db9e805ea72f02e
2021-05-11T12:58:02.061 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 18/64: e6da94b45675a62d0c9b231ac5ad86b498af37683e7adf3c0ffa9ddc84aa3d48
2021-05-11T12:58:08.460 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 19/64: d49bf49d63d46c3b62ac29947d92af4f3de8937c5c66e9da0f0008efa3a79ce7
2021-05-11T12:58:17.828 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 20/64: 8c866cfe678cd60642bac3c9114777b6b1e7e87e853f1d6196f1dea91615a401
2021-05-11T12:58:29.192 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 21/64: b7d09601c38e78017b92062a08bc2919db6e085936d37d6bd11fc72803f27324
2021-05-11T12:58:37.724 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 22/64: 8631c89caf5baaa30d53836af740bb405bf9065d349362334b696b57454f79db
2021-05-11T12:58:46.404 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 23/64: 289dd9e17194a61c7494c37c116a47152269365426f602bff943a9fbca4107e4

Can see the dupes persist in challenges:

$ venv/bin/chia farm challenges
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 48
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 47
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 46
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 45
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 44
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 43
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 42
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 41
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 41
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 40
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 40
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 39
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 39
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 39
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 38
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 38
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 38
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 37
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 36
Hash: 0xf1b55ada82059b8df4d2032ab2e0d616a5294d170bf823b0a0f8641dff123b64Index: 36

I can move the plotting to another machine (Ryzen 3600XT), but I don't believe this is a solve since it also happens when plotting is stopped. Network interface is never more than 25% utilized. What else could those of us experiencing this provide to help narrow it down?

Haskell-mouse commented 3 years ago

Hmmm.... I don't think that it is a performance problem only. I have RPi4 8gb ram with microSD only (but it is a fast one). And I have no missed signage points at all chia 1.1.5

2021-05-12T15:55:42.224 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 20/64: f7d863bf7f7d42d0672283190c44417ea1d0a720e2267e8ce76c50eea45517cf 
2021-05-12T15:55:50.390 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 21/64: 45bb0362e135e28a9647128d179d11473c7fba642913b121b8195fc2f7a3e9c7 
2021-05-12T15:55:58.941 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 22/64: e836827e9a4e692e0eca93c41e8e867db1017bccf239fd8a1323778f0fa00837 
2021-05-12T15:56:09.097 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 23/64: 353ceeb2942b844238462765cf6f543b85c49dae8c5b0bbf8bb8e51b9260dbc8 
2021-05-12T15:56:17.541 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 24/64: 60ce926be551e9531698a441b7e47b1583ed669380b57fdedc9a87b465ec2d57 
2021-05-12T15:56:26.040 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 25/64: 6df068c9b8c8b6cbd86772c60cddcf6d5f1b019fc2a75cc4269dfac628bc0aab 
2021-05-12T15:56:33.768 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 26/64: 3a48e2ae3cea10d3250d57857b7603d80d82179953d68db7060d2156865cd256 
2021-05-12T15:56:43.685 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 27/64: 1d399fb0712525ae0080c7df5cbd448f7505777132d1b44b36fe2cf24f186a18 
2021-05-12T15:56:50.320 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 28/64: 4f5fd1b9082c723edce2dd84070dec28d6a17e51c219c4aeed316d94ed4d40c7 
2021-05-12T15:56:58.793 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 29/64: 8197d4f51b443729d2337d1e81257be9854eb1317fc2edaf626e33114de05b5b 
2021-05-12T15:57:08.303 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 30/64: b1139e461d93e870541835083081efe70ed021e240159131c04070b94590fdb0 
2021-05-12T15:57:17.067 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 31/64: 7dd4521d6648b05d2329fc36a2c4cefe7b98cc34afb7c3ab77023b97ca6d0d5f 
2021-05-12T15:57:24.176 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 32/64: ae7087ffefb0b44745bd3aed0fae6f38842fc83afae15257ba363dbf6f9b1065 
2021-05-12T15:57:32.265 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 33/64: a543251ba105d15dc2399e67e1d0ec362db0f7f79c41d71252fdceee5a0932ff 
2021-05-12T15:57:41.978 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 34/64: 6c8c5d5a921b796ca2582302711a093e203751d30d96fc91f80e15e3478793f4 
2021-05-12T15:57:50.271 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 35/64: 97d76c476b156e171a47a64a663e8d8e3b73b8cdf582d97194ac1f6d66a8508a 
2021-05-12T15:57:57.559 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 36/64: 435eb076cff6a538bc9be979a97fa5921462ce8a47a263a0d0ff82db81566c2d 
2021-05-12T15:58:05.620 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 37/64: 15c68ae6e00c154a706e353ece012ec4a48b88a72a6b8e6ec4f89a3c28f99f4b 
2021-05-12T15:58:14.659 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 38/64: 458af520d6a0080e7ffb83040dac141b2addf4bca1fd7b3103c21a8c6e8b9591 
2021-05-12T15:58:22.146 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 39/64: 0aef748674cddb4479fedaac16868b37c6beda2cf9db8521180dc78f9ac84ad0 
2021-05-12T15:58:29.875 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 40/64: 9c7b73628fa9fba82320c3f84501ffdb18bbb2e1fc4f9fda371cb328618d1f68 
2021-05-12T15:58:37.604 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 41/64: d6ee2b4458278bd0469d122f6592eee0ec11933099584f0510d3983debdaa8a3 
2021-05-12T15:58:45.826 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 42/64: bb8b318c831a3794e711aeaa92dfa8dd8c8aeb77e90587d388b34b718a69146b 
2021-05-12T15:58:54.580 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 43/64: 0f399ea5715d7006eb136192ead5b19b52418743c1a2e0302eb015f004cbc107 
2021-05-12T15:59:04.241 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 44/64: e8382503b75b4a6c58cf5253bf29be84fd8e11761a8bcdf94f12d4e0f120380f 
2021-05-12T15:59:12.818 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 45/64: f62c8d3a5a7be5247f04a0e896a21e449f72060d730ea60a021db33bdd1604be 
2021-05-12T15:59:20.243 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 46/64: 95b939b1c18c55b597ecc877bb69834015a9c2955803e0d49dcf25cd7ebd04a9 
2021-05-12T15:59:27.997 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 47/64: a497e267488fee768703064f75cb65b1ce96e2aef64361730b9295977827b63b 
2021-05-12T15:59:35.487 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 48/64: f980174a14dd2a9e0b90ca822713a269e213fb8a4b2d3c5f028700d845e6c1ed 
2021-05-12T15:59:44.010 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 49/64: a11009a16a80cf0b561c89d539ba000e8911b2ad5a1a375dc959ffe99ba63fb3 
2021-05-12T15:59:52.489 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 50/64: 9428965504d3d018bb2fd3c08f51b3ef039706fbe370bd9f2776a0dc50e6b06f 
2021-05-12T16:00:02.741 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 51/64: 46d86c2a0e6586e1e585649b55182ef7bfbddcec9a7868dcc1799e15afadfa00 
2021-05-12T16:00:11.081 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 52/64: 2e5c9ccecd008d70aeb6d8f7e084999fea9cdfda361bdb52c28410a8ec3fcd84 
2021-05-12T16:00:18.264 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 53/64: 07207be1716f668b5a0edf61fe51d75b5569fd3beaf6f3dd7e7243c17b6d6dfc 
2021-05-12T16:00:26.873 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 54/64: 42352f6d4993b51b52040abd5bd3f67b048a127177ed3199283b12203a2de7d8 
2021-05-12T16:00:36.787 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 55/64: c08edc3ebadf884a1f46bec4dcbfd1088a41dbc0bf5a63e1ae0ba746d74920b0 
2021-05-12T16:00:48.702 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 56/64: d1f23b9a31605e8d3b9f66b8d5728d0e7028fa241307a46dcf37bb62dc090962 
2021-05-12T16:00:58.510 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 57/64: d1d52813158851da8f8bfbcac235c96e70559f4bd7df6710e493a758062a2884 
2021-05-12T16:01:05.822 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 58/64: 08d74eafb98c1b36f8f8fdc676be30ffc9b929e5df5ba9f2b907b13c82fe3e8a 
2021-05-12T16:01:14.271 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 59/64: 107c8ae1aa69b1e20a085e80812fd88d469887f2ed54f6408c91895f4098d377 
2021-05-12T16:01:22.895 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 60/64: d734e843e46ad7ec3a3c7b1e13cbdee01347524378408709c5c757f6e8ccd3a2 
2021-05-12T16:01:32.379 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 61/64: e524a387ad18ba40781f84816759e3968d1c0f27cba012327ac59440f86b7f72 
2021-05-12T16:01:40.683 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 62/64: 4a3a15501e2448e947436dd7b955c6a71fbccdfeae2a2dda68dff1e9d37e9e54 
2021-05-12T16:01:48.961 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 63/64: acb95c8db1e8379d0a519987bf53d5cb640f9cf1bdfe218020d74296ef33e291 
2021-05-12T16:02:00.382 full_node chia.full_node.full_node: INFO     ⏲️  Finished sub slot, SP 64/64, 4ebad3dc1588791c09efbc012afa47aa09b713c655e492b27c3d4215357fea20, number of sub-slots: 2, RC hash: a7e7a11c934ddc3c1df858e2704f9ac67bcd775c10f9e8045d7a28fb80f4d59e, Deficit 16
2021-05-12T16:02:10.386 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 23a99833db37400346452f4022bc75a27acf00d14bdd90aaec2ab4f706718650 
2021-05-12T16:02:17.507 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 2/64: cbbf9e478edc168d5c781a2a2efd06b6573362d7236cdd3d26300d44255b59af 
2021-05-12T16:02:25.487 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 3/64: 9f811a35eccd4e54add8990221e829b3e4aa75f9e98cccd5676914fcaf89a36a 
2021-05-12T16:02:33.501 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 4/64: bcc1241fde475c623a7ff4d8425d272a986dc14d9e2a4f2b59255e7b53ed15eb 
2021-05-12T16:02:42.245 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 5/64: f551c8451fe0c773828aeea117aba7cb8b6f87190fb07a2705048b3b95b4081f 
2021-05-12T16:02:53.214 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 6/64: 6a4124d19fe92250844156e92df7266cea0636e58e517c7aa7c9be1bf8fd18f4

wriches commented 3 years ago

With plotting stopped to avoid any possible contention (shouldn't exist in the above beyond CPU), still appears to see dupes/out-of-order in the logs:

@lwndow I'm seeing the same issue on a machine and network that has no obvious performance issues. I think this might be a different issue since SPs are not missed but instead appear out of order and/or duplicated. I've raised this in #3507.

It may be that this is not an issue that is having any real effect since in most cases all the SPs are there but repeated several times. For example, in your logs provided, there are 11.596 seconds between the logging of SP 10 and 11, which seems to be not too far above what is expected, it's just that a bunch of prior SPs are repeated. Perhaps caused by remote harvesters lagging behind?

Either way it would be good if the devs could clarify here, as this may be a non-issue. Pinging @mariano54 and @arvidn for comment - but I'm sure they're lacking a little in sleep right now ;)

arvidn commented 3 years ago

I think we're a bit too slow to farm effectively on the Raspberry PI right now. I'm working on optimizations to fix this and our goal is definitely to work well on the RPi.

I don't have any hard numbers on farming specifically, but right after the transactions launch, the chain experienced a fair amount of traffic, which the RPi was just barely keeping up with.

lwndow commented 3 years ago

With plotting stopped to avoid any possible contention (shouldn't exist in the above beyond CPU), still appears to see dupes/out-of-order in the logs:

@lwndow I'm seeing the same issue on a machine and network that has no obvious performance issues. I think this might be a different issue since SPs are not missed but instead appear out of order and/or duplicated. I've raised this in #3507.

It may be that this is not an issue that is having any real effect since in most cases all the SPs are there but repeated several times. For example, in your logs provided, there are 11.596 seconds between the logging of SP 10 and 11, which seems to be not too far above what is expected, it's just that a bunch of prior SPs are repeated. Perhaps caused by remote harvesters lagging behind?

Either way it would be good if the devs could clarify here, as this may be a non-issue. Pinging @mariano54 and @arvidn for comment - but I'm sure they're lacking a little in sleep right now ;)

Agree, I'm seeing both issues, thanks for linking the other. I have complete skips of signage points as well, but should have posted the out-or-order over there.

salsaly4 commented 3 years ago

I think we're a bit too slow to farm effectively on the Raspberry PI right now.

I've got the same problem running farmer on HP Microserver gen 10 with Opteron X3418, which has way more performance than BCM2711...

Haskell-mouse commented 3 years ago

Ok, I've read my logs more closely. And I've found the following lines:

2021-05-12T18:52:37.362 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d. caching signage point 48.
2021-05-12T18:52:37.363 full_node chia.full_node.full_node: INFO     Signage point 48 not added, CC challenge: 212a3663765904f47166eeed9e957df57c89dd56f0c2b7f554c2217b7da936bc, RC challenge: f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d
2021-05-12T18:52:37.492 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d. caching signage point 48.
2021-05-12T18:52:37.493 full_node chia.full_node.full_node: INFO     Signage point 48 not added, CC challenge: 212a3663765904f47166eeed9e957df57c89dd56f0c2b7f554c2217b7da936bc, RC challenge: f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d
2021-05-12T18:52:37.588 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d. caching signage point 48.
2021-05-12T18:52:37.589 full_node chia.full_node.full_node: INFO     Signage point 48 not added, CC challenge: 212a3663765904f47166eeed9e957df57c89dd56f0c2b7f554c2217b7da936bc, RC challenge: f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d
2021-05-12T18:52:37.597 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d. caching signage point 48.
2021-05-12T18:52:37.598 full_node chia.full_node.full_node: INFO     Signage point 48 not added, CC challenge: 212a3663765904f47166eeed9e957df57c89dd56f0c2b7f554c2217b7da936bc, RC challenge: f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d
2021-05-12T18:52:37.611 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d. caching signage point 48.
2021-05-12T18:52:37.613 full_node chia.full_node.full_node: INFO     Signage point 48 not added, CC challenge: 212a3663765904f47166eeed9e957df57c89dd56f0c2b7f554c2217b7da936bc, RC challenge: f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d
2021-05-12T18:52:37.660 full_node chia.full_node.full_node_store: INFO     Don't have rc hash f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d. caching signage point 48.
2021-05-12T18:52:37.661 full_node chia.full_node.full_node: INFO     Signage point 48 not added, CC challenge: 212a3663765904f47166eeed9e957df57c89dd56f0c2b7f554c2217b7da936bc, RC challenge: f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d
2021-05-12T18:52:37.692 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 270146, weight 14033132, hh 6685a2963ba3a374636dafcdda53eb3a9561ed4a4fdc340ae3ad89e699c0cf62, forked at 270145, rh: f21a81f20dd86b81cb21c913104de06c002d28d7d8190ab10f2674a2fb77578d, total iters: 872724605402, overflow: False, deficit: 0, difficulty: 245, sub slot iters: 106430464, Generator size: No tx, Generator ref list size: No tx
2021-05-12T18:52:37.893 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 48/64: 5af41c0abc44bcdbe90cfa3a3e20b4de8976a68564778d3ec15ae40e22d75f32 
2021-05-12T18:52:38.290 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming 212a366376... Found 0 proofs. Time: 0.29585 s. Total 431 plots
2021-05-12T18:52:38.576 wallet chia.wallet.wallet_blockchain: INFO     💰 Updated wallet peak to height 270146, weight 14033132

A common pattern is:

Signage point x not added.....
Don't have rc hash yyyyy....
caching signage point x.... 
<repeat n times during 0.5 sec or so>

Finished signage point x

Is it ok ? I just don't understand..... The system finishes every signage point, but sometimes only after 5-10 messages Signage point x not added.....

salgo commented 3 years ago

I am seeing this happen regularly on an i7-7700T with 16GB and nothing other than Chia running. I believe in my case it was caused by the rsyncing of new plots onto the farming machine and have used the --bwlimit=80m option to reduce the network bandwidth used.

I will report back if I continue to see it but thought I'd mention it immediately in case someone else might be encountering the same issue.

n1ghtwish commented 3 years ago

I too am seeing odd signage behaviour, machine is doing other things as well but it's not 100% CPU maxed out.

Sometimes I get signage points out of order, sometimes missing totally.

Sometimes repeating ones.

Would be nice if it was working normally, internet speed is decent (1Gbps down/120Mbps up) and network is hard-wired.

omjanu commented 3 years ago

I am on windows and lately my challenge hash index has been going out of order a lot. can anyone tell me why it's doing that.

SKYBILLS commented 3 years ago

i am also using win 10 , and i am always missing last signage point number 64

Screenshot (10)

UltimateBrent commented 3 years ago

i am also using win 10 , and i am always missing last signage point number 64

I believe it's 0 indexed, so there's 64 of them and it stops at 63.

dukeofdrum commented 3 years ago

84B89C26-35CB-42B3-81D9-6F309B4C63AA Same issue here. Full Node is an older Phenom II 940 with 16GB RAM.

Never had issues (but high response times due to lot of smb Network shares) until I switched all plotter into harvesters.

6 Harvester/Plotter in the same network. Everything connected through 1Gbit/s. ISP connection is an old ADSL 6mbit/1mbit Connection with 24 hour reconnect.

Only Main Node ports 8444-8447 opened.

sometimes it’s running just fine from 1-64 in CLI and then it’s completely freaking out.

all Harvester and Full Node are on 1.1.6dev0 and 1 Windows Harvester on 1.1.5.

CPU usage is up to 80-90%. Only process running is GUI.

each Plotter got 4-5 HDDs installed
1100+ Plots Overall
Only reward 29th April with 180 plots yet

gvonbergen commented 3 years ago

Hi there, have the same issues with the RC has errors. Farming on a server with Intel SSDs and logs to to a RAM drive.

2021-05-14T09:40:18.267 full_node chia.full_node.full_node: INFO ⏲️ Finished signage point 9/64: 6e1d9ceacd4c730438d2d152c81c1cbb857ed86134a149df426a4e85bb8efe4b 2021-05-14T09:40:26.793 full_node chia.full_node.full_node: INFO ⏲️ Finished signage point 10/64: 44c633f1dd8ebc821342f498a1d120a500d9fbd2808e445366a4e47ec40814e0 2021-05-14T09:40:35.016 full_node chia.full_node.full_node: INFO ⏲️ Finished signage point 11/64: 6058aa41c101ef2c2e20649c8cae9bcc04f52b002a27298e3802353fcfb4071e 2021-05-14T09:40:45.058 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:45.722 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:45.842 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:45.855 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:46.271 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:46.550 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:46.799 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:47.276 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:47.485 full_node chia.full_node.full_node_store: INFO Don't have rc hash b6c0c598203302ce9f978605a334205efe04a6a4862382435ee19ca8f083e2b9. caching signage point 12. 2021-05-14T09:40:47.684 full_node chia.full_node.full_node: INFO ⏲️ Finished signage point 12/64: f567f9613626a1c56d688e83e52c10a0dfce70aec7d82ce97a08cfc9493b6071 2021-05-14T09:40:52.662 full_node chia.full_node.full_node: INFO ⏲️ Finished signage point 13/64: de118b3eb34072f5a1b555f615125645d658e5794901dd1d63bf9d31a1fe0b98 2021-05-14T09:41:00.798 full_node chia.full_node.full_node: INFO ⏲️ Finished signage point 14/64: 7aa91dd4b189aec11a0dce5a2b28b23e1f3b855327a9abb5befe3cb8e76cc66f