loading plots issue

[mnzer1987]

i cant see my plots , i delete the folder directory and add it again but not working at all

[Mi-Q]

For me when I started 1.2.4 or 1.2.5 only 30 Plots were loaded at first, then after quite a while 90 and finally 251 of my 263 plots while 12 were reported as erroneus. (for every restart of those two versions) After switching back to 1.2.3 all plots show instant and are farming correctly.

[emlowe]

We changed it to batch read in the plots so that while reading plots the harvester would still respond to signage points. However, the defaults were too low - it would load plots in 30 plot batches. You can change this in config.yaml - we currently recommend batch_size: 300 and batch_sleep_milliseconds: 1

  plots_refresh_parameter:
    batch_size: 30 # How many plot files the harvester processes before it waits batch_sleep_milliseconds
    batch_sleep_milliseconds: 10 # Milliseconds the harvester sleeps between batch processing

[dorifutoben]

We changed it to batch read in the plots so that while reading plots the harvester would still respond to signage points. However, the defaults were too low - it would load plots in 30 plot batches. You can change this in config.yaml - we currently recommend batch_size: 300 and batch_sleep_milliseconds: 1

  plots_refresh_parameter:
    batch_size: 30 # How many plot files the harvester processes before it waits batch_sleep_milliseconds
    batch_sleep_milliseconds: 10 # Milliseconds the harvester sleeps between batch processing

Where do we add this? I searched the config.yaml file for 'refresh' with no results found.

[sdbarrish]

I found that doing upgrades, especially from linux, the config.yaml file doesn't get updated.

Do a fresh install and the proper config.yaml file will be there with the aforementioned missing section in the harvester section.

[sdbarrish]

[UPDATE] - Still not refreshing plots after copying to NAS. I have the section in the config.yaml file for the harvester to allegedly refresh after failed plot check during copying and completed copying a plot from my plotter to my NAS.

The GUI shows the correct number of plots in the temp directory (plotter) while the file is being copied....then at some point, the exact number of plots get increased for the harvester that I'm copying to. Then that same number gets decreased by the number of files being copied. Finally, once the file(s) are copied to the NAS, I wait the 1200 seconds and the GUI nor the harvester (tail -f on the debug.log file) shows an increase in plots being harvested. I have to do the chia start harvester -r in the CLI to then show that the new plots that were copied are now counted and being farmed.

[dorifutoben]

Sdbarrish you just made my day. I got it working. I'm using windows but this below helped me. "I found that doing upgrades, especially from linux, the config.yaml file doesn't get updated."

I did an uninstall for chia using revo uninstaller which is usually pretty clean. Found the .chia folder still in my users folder.

I copied my db files from users\$user.chia\mainnet\wallet and users\$user.chia\mainnet\db and save it somewhere.

Deleted the .chia folder. Reinstalled. Moved my db files back to their original location.

Now I'm able to add the plot directories and the 'refresh' parameter is there.

Thank you sdbarrish

[emlowe]

As an FYI, if the setting is not in config.yaml, then the default will be used, for example, if you don't have plots_refresh_parameter: batch_size at all in config.yaml, in 1.2.6, the default of 300 would be used.

[sdbarrish]

I believe I have identified the bug….

So in the invalid refresh process, it doesn’t account for the fullnode’s harvester being a temp location of the newly created plot to be immediately farmed while at the same time copying the plot to the remote harvester (NAS). It does flag the file size incorrect, but I think it may also flag it as a duplicate plot and then never resets THAT plot on the remote harvester….then when the file is completely copied to the NAS, the temp directory file gets deleted (because I do a MOVE command, not a copy command). Then until I restart the harvester, the flag of a duplicate plot is reset (well, everything is reset) and the newly copied plot on the NAS is recognized and the correct plot count is reported.

So, what I’ve done to test this theory is I’m currently plotting a new batch of plots. I have deleted the temp plot directory references in my fullnode where the plots finish plotting, allowing for my copying procedure to remain as usual. I will then see once the copying is completed after 1200 seconds if the correct number of plots are reported on the remote harvester (NAS) in the GUI and via the debug file.

Sincerely, Scott

From: Earle Lowe @.> Sent: Tuesday, September 14, 2021 11:37 AM To: Chia-Network/chia-blockchain @.> Cc: Scott Barrish @.>; Comment @.> Subject: Re: [Chia-Network/chia-blockchain] loading plots issue (#8386)

As an FYI, if the setting is not in config.yaml, then the default will be used, for example, if you don't have plots_refresh_parameter: batch_size at all in config.yaml, in 1.2.6, the default of 300 would be used.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Chia-Network/chia-blockchain/issues/8386#issuecomment-919268755, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRM4GWNQ4WAUIX3UQMTJZTUB5TYZANCNFSM5DTR6NWA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[jack60612]

I believe I have identified the bug…. So in the invalid refresh process, it doesn’t account for the fullnode’s harvester being a temp location of the newly created plot to be immediately farmed while at the same time copying the plot to the remote harvester (NAS). It does flag the file size incorrect, but I think it may also flag it as a duplicate plot and then never resets THAT plot on the remote harvester….then when the file is completely copied to the NAS, the temp directory file gets deleted (because I do a MOVE command, not a copy command). Then until I restart the harvester, the flag of a duplicate plot is reset (well, everything is reset) and the newly copied plot on the NAS is recognized and the correct plot count is reported. So, what I’ve done to test this theory is I’m currently plotting a new batch of plots. I have deleted the temp plot directory references in my fullnode where the plots finish plotting, allowing for my copying procedure to remain as usual. I will then see once the copying is completed after 1200 seconds if the correct number of plots are reported on the remote harvester (NAS) in the GUI and via the debug file. Sincerely, Scott From: Earle Lowe @.> Sent: Tuesday, September 14, 2021 11:37 AM To: Chia-Network/chia-blockchain @.> Cc: Scott Barrish @.>; Comment @.> Subject: Re: [Chia-Network/chia-blockchain] loading plots issue (#8386) As an FYI, if the setting is not in config.yaml, then the default will be used, for example, if you don't have plots_refresh_parameter: batch_size at all in config.yaml, in 1.2.6, the default of 300 would be used. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#8386 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRM4GWNQ4WAUIX3UQMTJZTUB5TYZANCNFSM5DTR6NWA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

let me know how it goes

[sdbarrish]

Thank you Sean C. for troubleshooting with me on this issue.

ISSUE:

1. My farmer had the final destination folders for the plots as directories to farm from.
2. While copying the plot from the final destination folder from plotting to the NAS, the NAS flagged it as a duplicate.
3. This flag does not clear in the refresh_invalid_plots parameter.
   • Only gets cleared when you restart the harvester node.

SOLUTION:

1. Removed the plot directories from the fullnode that were the final destination folders for my plotters.
2. Restarted all harvesters; full node, NAS_1, NAS_2

CONFIRMATION:

1. When plotting finished, and the copying began, the plot was flagged as invalid plot due to file size.
   • Confirmed in the farmer’s GUI, plots tab showing the exact 2 plots being copied listed in the invalid box.
2. When copying completed, ~20 minutes later, plots were counted on the farmer GUI both by line item per harvester node and total plot count.
   • Confirmed in the farmer’s GUI, plots tab showing no plots in the invalid box.

So prior to pooling, I was able to farm the newly created plot from the plotter until the move to the NAS finished and then the plot counts would be automatically updated by decreasing the farmer’s harvester node line item count and increasing by the same number for the NAS’s harvester’s plot count and the total plot count would remain the same.

Post pooling, this doesn’t work anymore. What stinks is, in those 20 minutes, and if we consider 8 seconds interval, that’s 150 potential challenges per plot that could have past filter. Pre-pooling, I was taking advantage of that time gap….Post pooling, I cannot…at least I don’t know how to decrease that time period as of yet.

Sincerely, Scott

From: Jack Nelson @.> Sent: Tuesday, September 14, 2021 5:34 PM To: Chia-Network/chia-blockchain @.> Cc: Scott Barrish @.>; Comment @.> Subject: Re: [Chia-Network/chia-blockchain] loading plots issue (#8386)

I believe I have identified the bug…. So in the invalid refresh process, it doesn’t account for the fullnode’s harvester being a temp location of the newly created plot to be immediately farmed while at the same time copying the plot to the remote harvester (NAS). It does flag the file size incorrect, but I think it may also flag it as a duplicate plot and then never resets THAT plot on the remote harvester….then when the file is completely copied to the NAS, the temp directory file gets deleted (because I do a MOVE command, not a copy command). Then until I restart the harvester, the flag of a duplicate plot is reset (well, everything is reset) and the newly copied plot on the NAS is recognized and the correct plot count is reported. So, what I’ve done to test this theory is I’m currently plotting a new batch of plots. I have deleted the temp plot directory references in my fullnode where the plots finish plotting, allowing for my copying procedure to remain as usual. I will then see once the copying is completed after 1200 seconds if the correct number of plots are reported on the remote harvester (NAS) in the GUI and via the debug file. Sincerely, Scott From: Earle Lowe @.> Sent: Tuesday, September 14, 2021 11:37 AM To: Chia-Network/chia-blockchain @.> Cc: Scott Barrish @.>; Comment @.> Subject: Re: [Chia-Network/chia-blockchain] loading plots issue (#8386https://github.com/Chia-Network/chia-blockchain/issues/8386) As an FYI, if the setting is not in config.yaml, then the default will be used, for example, if you don't have plots_refresh_parameter: batch_size at all in config.yaml, in 1.2.6, the default of 300 would be used. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#8386 (comment)https://github.com/Chia-Network/chia-blockchain/issues/8386#issuecomment-919268755>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRM4GWNQ4WAUIX3UQMTJZTUB5TYZANCNFSM5DTR6NWA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

let me know how it goes

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Chia-Network/chia-blockchain/issues/8386#issuecomment-919529384, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRM4GUFJCVA6FXDAD7CMKDUB65WHANCNFSM5DTR6NWA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[loppefaaret]

@sdbarrish could you confirm which Chia version you are on ?

[sdbarrish]

Here’s what I’ve done:

1. Stopped the full node’s harvester Full node is on my desktop and I also plot with the desktop too. I believe this was creating a duplicate plot file flag in my CHIA local network which the refresh_invalid_plots parameter does not clear.
2. I SSH’ed into each harvester node that I’m actively copying files to and set the following CLI command: watch -n 3600 ‘chia start harvester -r’
3. I now plot and copy as usual…Every hour the harvesters that I’m copying newly created plots to from my plotters get restarted. The plot counts after restart are correct. Total plot counts are also correct after harvester restarts.

To troubleshoot when your harvester raises a duplicate plot message:

1. Turn off all docker containers that are running your remote harvesters.
2. In the GUI, set all directories where your plots reside
3. Run the following CLI command: chia plots check -n 0 -l

See if the output lists any locations of the same plot file. Figure out where it SHOULD be and delete the duplicate file.

I hope that gives folks some insight!

From: Jack Nelson @.> Sent: Tuesday, September 14, 2021 5:34 PM To: Chia-Network/chia-blockchain @.> Cc: Scott Barrish @.>; Comment @.> Subject: Re: [Chia-Network/chia-blockchain] loading plots issue (#8386)

I believe I have identified the bug…. So in the invalid refresh process, it doesn’t account for the fullnode’s harvester being a temp location of the newly created plot to be immediately farmed while at the same time copying the plot to the remote harvester (NAS). It does flag the file size incorrect, but I think it may also flag it as a duplicate plot and then never resets THAT plot on the remote harvester….then when the file is completely copied to the NAS, the temp directory file gets deleted (because I do a MOVE command, not a copy command). Then until I restart the harvester, the flag of a duplicate plot is reset (well, everything is reset) and the newly copied plot on the NAS is recognized and the correct plot count is reported. So, what I’ve done to test this theory is I’m currently plotting a new batch of plots. I have deleted the temp plot directory references in my fullnode where the plots finish plotting, allowing for my copying procedure to remain as usual. I will then see once the copying is completed after 1200 seconds if the correct number of plots are reported on the remote harvester (NAS) in the GUI and via the debug file. Sincerely, Scott From: Earle Lowe @.> Sent: Tuesday, September 14, 2021 11:37 AM To: Chia-Network/chia-blockchain @.> Cc: Scott Barrish @.>; Comment @.> Subject: Re: [Chia-Network/chia-blockchain] loading plots issue (#8386https://github.com/Chia-Network/chia-blockchain/issues/8386) As an FYI, if the setting is not in config.yaml, then the default will be used, for example, if you don't have plots_refresh_parameter: batch_size at all in config.yaml, in 1.2.6, the default of 300 would be used. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#8386 (comment)https://github.com/Chia-Network/chia-blockchain/issues/8386#issuecomment-919268755>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRM4GWNQ4WAUIX3UQMTJZTUB5TYZANCNFSM5DTR6NWA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

let me know how it goes

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Chia-Network/chia-blockchain/issues/8386#issuecomment-919529384, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRM4GUFJCVA6FXDAD7CMKDUB65WHANCNFSM5DTR6NWA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[github-actions[bot]]

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

[github-actions[bot]]

This issue was automatically closed because it has been flagged as stale and subsequently passed 7 days with no further activity.