[BUG] Harvester won't work with 6k plots

[rocketeeer]

Describe the bug I have two harvesters with identical configs, harv1 has 1700 plots and harv2 has 6000 plots. Harv1 connects to the farmer just fine, Harv2 connects to the farmer but shows up with 0 plots, even after 10+ hours.

Looking in the debug log for Harv2 I found this error:

2021-11-13T09:09:05.479 harvester chia.rpc.rpc_server     : WARNING  Sending data failed. Exception Traceback (most recent call last):
  File "chia/rpc/rpc_server.py", line 69, in _state_changed
  File "aiohttp/client_ws.py", line 150, in send_str
  File "aiohttp/http_websocket.py", line 687, in send
  File "aiohttp/http_websocket.py", line 643, in _send_frame
  File "aiohttp/http_websocket.py", line 660, in _write
ConnectionResetError: Cannot write to closing transport

The only difference between the two harvesters is the number of plots. Both are running v1.1.1, have been initialized using the farmer's certs, and are connecting to the farmer. I am not farming Cactus.

To Reproduce

Steps to reproduce the behavior:

1. Run a harvester with 6000+ plots
2. Connect harvester to farmer

Expected behavior Harvester shows up with correct number of plots.

Screenshots

Desktop

Farmer -

• OS: Ubuntu 20.04
• CPU: Intel i7

Harvester1 -

• OS: Ubuntu 20.04
• CPU: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz

Harvester2 -

• OS: Ubuntu 20.04
• CPU: 2 X Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz

Additional context

[albertcoady]

Please restart the harvester and try to load 3000 plots first, then after it show on the farmer, load the other 3000 plots.

[rocketeeer]

I was able to get it working with less than 3000 plots, but I'm not sure what you mean when you say "load the other 3000 plots". I stopped the harvester, updated the config to include the remaining plot directories, and started the harvester with more than 3000 plots - which resulted in 0 plots showing up on the farmer again.

I get inconsistent results when starting the harvester with less than 3000 plots, sometimes they appear as expected but sometimes I get a 0 in the farm summary.

I'm also seeing similar reports from others where harvesters with large numbers of plots are using a lot of resources and response times are blowing out into the 100's of seconds

[PrEzi3]

@kiote99 have 3000 plots in the pool / config, then run harvester and perform
sit plots add -d PATH to add more plot directories gradually one by one and see what happens and if the harvester recognizes then only small additions.

[rocketeeer]

This kind of worked, I wrote a script to add a new directory (each containing 145 plots) every 10 minutes and let it run over night. The farmer is reporting the harvester has ~4300 out of what should be 6000 plots. The debug log on the harvester shows response times are now up to 1400s.

[PrEzi3]

@kiote99 Thanks for reporting the behavior back.

@albertcoady It might be related to what I have reported right afterwards (really bad harvester performance observed on low powered devices incl. long loading times and high resource usage): https://github.com/silicoin-network/silicoin-blockchain/issues/137 Seems this issue is really hurting harvesting...