Inaccurate partials proof marked as OG proof

[verifyfx]

I suspected that there are issue where partials proof were parsed

2021-07-10T02:08:40.428 harvester chia.harvester.harvester: INFO     3 plots were eligible for farming 3ff73aafb4... Found 2 proofs. Time: 2.29660 s. Total 57 plots
2021-07-10T02:08:40.447 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://asia1.pool.space

from reddit

[stolk]

Thanks for reporting. Does the dot turn cyan if you resize the terminal (w tool running) to be shorter in height?

[kastermight]

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

[stolk]

Ok, I've been able to reproduce it now. And understand what is going on. This is triggered when it finds 2 proofs, it seems. And possibly only submits a partial for 1?

[stolk]

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log.

[verifyfx]

resizjng doesn't turn it cyan

[verifyfx]

pulling latest ( https://github.com/stolk/chiaharvestgraph/commit/1dbc2c17b013322151574563e93a1fbcb4c23f5e ) fixed this issue.

[kastermight]

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log.

OK I did. My idea is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries to differentiate between OG proofs and pool proof. I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging.

[stolk]

Huh? that makes absolutely no sense, to run 48 full nodes on one machine.

Are you sure you actually have that many running? How would you even start that many nodes?

On Sun, Jul 11, 2021 at 12:36 AM Kaster Might @.***> wrote:

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D [image: Screenshot from 2021-07-10 12-16-00] https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png

Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log.

OK I did. My ideas is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries, and I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/stolk/chiaharvestgraph/issues/42#issuecomment-877755786, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A .

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.***

[verifyfx]

Found another case: It's a case where farmer logged submitting partials before harvester logged proof found

2021-07-11T22:01:16.087 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://XXX
2021-07-11T22:01:16.406 harvester chia.harvester.harvester: INFO     2 plots were eligible for farming 315babe1db... Found 1 proofs. Time: 1.08288 s. Total 138 plots
2021-07-11T22:01:23.747 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 315babe1db... Found 0 proofs. Time: 0.00621 s. Total 138 plots
2021-07-11T22:01:34.160 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming 315babe1db... Found 1 proofs. Time: 2.18124 s. Total 138 plots
2021-07-11T22:01:34.203 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://asia1.pool.space
2021-07-11T22:01:35.091 farmer chia.farmer.farmer         : INFO     Pool response: {'new_difficulty': 4}
2021-07-11T22:01:41.452 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 315babe1db... Found 0 proofs. Time: 0.00669 s. Total 138 plots

[kastermight]

Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes?

afaik they start automatically when i do chia start farmer.

[stolk]

Those are threads of a single process. A thread per peer connection, probably.

On Sun, Jul 11, 2021 at 8:56 AM Kaster Might @.***> wrote:

Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes? … <#m-1223658502728578559> On Sun, Jul 11, 2021 at 12:36 AM Kaster Might @. > wrote: I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D [image: Screenshot from 2021-07-10 12-16-00] https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png Can you email your debug.log to b.stolk at gmail.com http://gmail.com please? I would like to test a fix on your log. OK I did. My ideas is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries, and I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment) https://github.com/stolk/chiaharvestgraph/issues/42#issuecomment-877755786>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A . -- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.

afaik they start automatically when i do chia start farmer. [image: Screenshot from 2021-07-11 08-56-13] https://user-images.githubusercontent.com/2565943/125201878-e2b34980-e225-11eb-8862-e5bb5eb8b01d.png

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub https://github.com/stolk/chiaharvestgraph/issues/42#issuecomment-877824507, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4AFQQMYTCBUAEXLTIDTXG5MHANCNFSM5AEPJL7A .

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.***

[kastermight]

Those are threads of a single process. A thread per peer connection, probably. On Sun, Jul 11, 2021 at 8:56 AM Kaster Might @.> wrote: Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes? … <#m-1223658502728578559> On Sun, Jul 11, 2021 at 12:36 AM Kaster Might @. > wrote: I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D [image: Screenshot from 2021-07-10 12-16-00] https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png Can you email your debug.log to b.stolk at gmail.com http://gmail.com please? I would like to test a fix on your log. OK I did. My ideas is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries, and I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment) <#42 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A . -- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @. afaik they start automatically when i do chia start farmer. [image: Screenshot from 2021-07-11 08-56-13] https://user-images.githubusercontent.com/2565943/125201878-e2b34980-e225-11eb-8862-e5bb5eb8b01d.png — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4AFQQMYTCBUAEXLTIDTXG5MHANCNFSM5AEPJL7A . -- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.

Hi. Those are not threads as they have separate PIDs and TGIDs. I dont' think it matters though.

I saw your email, but I still can see blue squares. Maybe you had different branch or something. I'm on latest master. Recompiled and everything. Attaching both screenshots and the log file here just in case.

PS: I noticed that if the terminal window is short enough, blue squares disappear. Maybe that's what you're seeing.

debug.log

[xordos]

I want share my log as I have simplest (single machine run full farmer) case and I do see the problem too.

Basically, in the log, sometime, the "Submitting partial" log line doesn't immediate after the eligible line. I have 2 such cases that show blue dot but are actually pool proof. I think this is a race condition that these 2 lines are from different process. So the order is not guaranteed, see the logging process name (farmer vs harvester) below:

2021-07-12T16:42:04.397 harvester chia.harvester.harvester: INFO 2 plots were eligible for farming 686599092a... Found 0 proofs. Time: 0.35231 s. Total 798 plots 2021-07-12T16:42:14.216 farmer chia.farmer.farmer : INFO Submitting partial for 10768b8bd5ab299b85a2c3d8fc80f1a70e03b898acd8292aeb7eb959a2765c04 to https://XXX 2021-07-12T16:42:14.299 harvester chia.harvester.harvester: INFO 4 plots were eligible for farming 686599092a... Found 1 proofs. Time: 1.75216 s. Total 798 plots 2021-07-12T16:42:14.398 farmer chia.farmer.farmer : INFO Pool response: {'new_difficulty': 1} 2021-07-12T16:42:22.282 harvester chia.harvester.harvester: INFO 2 plots were eligible for farming 686599092a... Found 0 proofs. Time: 0.06132 s. Total 798 plots

2021-07-13T06:56:25.522 harvester chia.harvester.harvester: INFO 1 plots were eligible for farming 4c1214cc87... Found 0 proofs. Time: 0.20828 s. Total 799 plots 2021-07-13T06:56:34.317 farmer chia.farmer.farmer : INFO Submitting partial for 10768b8bd5ab299b85a2c3d8fc80f1a70e03b898acd8292aeb7eb959a2765c04 to https://XXX 2021-07-13T06:56:34.357 harvester chia.harvester.harvester: INFO 3 plots were eligible for farming 4c1214cc87... Found 1 proofs. Time: 1.09751 s. Total 799 plots 2021-07-13T06:56:34.523 farmer chia.farmer.farmer : INFO Pool response: {'new_difficulty': 1} 2021-07-13T06:56:43.013 harvester chia.harvester.harvester: INFO 3 plots were eligible for farming 4c1214cc87... Found 0 proofs. Time: 0.12265 s. Total 799 plots

BTW, your tool is really useful and thanks for your hard work!!

[kastermight]

Hi @stolk Can you give a little update on this? I just want to understand if a) it's not a bug but a feature, b) it's a bug but not fixable at the moment. Thanks.

[stolk]

I was over thinking it, really...

I don't think there will ever be a farmer that farms both solo plots and pooled plots. So if there is a single pool proof seen, all proofs can be considered non-solo.

I will make a change to reflect this.

On Sat, Aug 7, 2021 at 3:47 PM Kaster Might @.***> wrote:

Hi @stolk https://github.com/stolk Can you give a little update on this? I just want to understand if a) it's not a bug but a feature, b) it's a bug but not fixable at the moment. Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stolk/chiaharvestgraph/issues/42#issuecomment-894714863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4EV5I6IJIIX2EWURZTT3WZYHANCNFSM5AEPJL7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.***

[stolk]

This should be fixed now. Sync to latest.

On Sat, Aug 7, 2021 at 4:46 PM Bram Stolk @.***> wrote:

I was over thinking it, really...

I don't think there will ever be a farmer that farms both solo plots and pooled plots. So if there is a single pool proof seen, all proofs can be considered non-solo.

I will make a change to reflect this.

On Sat, Aug 7, 2021 at 3:47 PM Kaster Might @.***> wrote:

Hi @stolk https://github.com/stolk Can you give a little update on this? I just want to understand if a) it's not a bug but a feature, b) it's a bug but not fixable at the moment. Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stolk/chiaharvestgraph/issues/42#issuecomment-894714863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4EV5I6IJIIX2EWURZTT3WZYHANCNFSM5AEPJL7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.***

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.***

[kastermight]

Seems to be working now :) I generally agree that soon pretty much no one will be doing both solo and pool. but at right this moment I know at least 5 people doing it. Not me though.

[stolk]

Are those people running multiple fullnodes and multiple farmers? Then it should be ok, they will run multiple graph tools.

On Sun, Aug 8, 2021 at 1:35 AM Kaster Might @.***> wrote:

Seems to be working now :) I generally agree that soon pretty much no one will be doing both solo and pool. but at right this moment I know at least 5 people doing it. Not me though.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/stolk/chiaharvestgraph/issues/42#issuecomment-894763350, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4CW5F7TC4YUAX4SE6LT3Y6WHANCNFSM5AEPJL7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.***

[kastermight]

Are those people running multiple fullnodes and multiple farmers? Then it should be ok, they will run multiple graph tools. On Sun, Aug 8, 2021 at 1:35 AM Kaster Might @.> wrote: Seems to be working now :) I generally agree that soon pretty much no one will be doing both solo and pool. but at right this moment I know at least 5 people doing it. Not me though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4CW5F7TC4YUAX4SE6LT3Y6WHANCNFSM5AEPJL7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email . -- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.

Not really. They still have their OG plots and nft plots on a single machine. They basically have a spare space to fill with nft plots and at the same time they keep solo farming OG plots. I was in the same boat but not anymore. I replaced all my OG plots with NFT plots already, and I'm sure soon everyone will be done with the same process. Depends on the farm size I guess.

[verifyfx]

just another idea, is it possible to include full node for full proof detection? basically we should look for the following line for full block?

🍀 ️Farmed unfinished_block {block_hash}, SP: {block.reward_chain_block.signage_point_index}, 

sauce: https://github.com/Chia-Network/chia-blockchain/blob/7927a459456c1f79c3604eb2aa5cf238545d15e8/chia/full_node/full_node.py#L1360