search

silicoin-network / silicoin-blockchain

Silicoin blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
79 stars 28 forks source link

Really high resource usage on low powered harvesters (e.g. Raspberry) and long time needed to recognize plots #175

Closed PrEzi3 closed 2 years ago

PrEzi3 commented 2 years ago

This is a clone of

137

since the old issue was closed without fixing the actual problem.

Since a few versions (I believe it all started more or less around when Chia version 1.2.10 codebase was introduced and staking was introduced into SIT) I have noticed extremely high resource usage (CPU time) when harvesting on low power devices (e.g. RaspberryPi). Please note --- this has not been the case a few versions back. Also extremely slow adding new plots and starting up (from starting harvester up to fully caching up all plots). In Chia it was nearly fully fixed with the codebase 1.2.11.

Steps to replicate : on Raspberry Pi set up SIT, add a lot of plots at once. Compared to old versions it takes forever for them to be cached. Also running SIT harvester on Raspberry uses extreme amount of resources (twice-triple the amount it used before in "old" versions).

Please note --- this is not a configuration issue of the Raspberry as I have three of them running and also testing other forks which introduced a similar behavior when incorporating chia 1.2.9-10 codebase commits. Flax has more or less fixed it updating from 0.1.4 to 0.1.5 (chia 1.2.11).

Can you please take a look at the commits introduced with chia 1.2.11 and incorporate them into the SIT codebase? Thank you!

//It looks like a lot of people have a performance problem with the harvester, plot recognitnion, processing of plots etc. Especially if the amount of plots is in higher numbers the performance drop is drastical. Starts around 1000+ plots, maybe earlier. Plot recognition after a new start takes forever, resource usage (CPU) compared to other forks is tripled or more etc.

Motophan commented 2 years ago

I am a micro farm on a amd 3200g (approx 4500 plots) and silicoin uses more resources than anything else on the pc

albertcoady commented 2 years ago

This is a clone of #137 since the old issue was closed without fixing the actual problem.

Since a few versions (I believe it all started more or less around when Chia version 1.2.10 codebase was introduced and staking was introduced into SIT) I have noticed extremely high resource usage (CPU time) when harvesting on low power devices (e.g. RaspberryPi). Please note --- this has not been the case a few versions back. Also extremely slow adding new plots and starting up (from starting harvester up to fully caching up all plots). In Chia it was nearly fully fixed with the codebase 1.2.11.

Steps to replicate : on Raspberry Pi set up SIT, add a lot of plots at once. Compared to old versions it takes forever for them to be cached. Also running SIT harvester on Raspberry uses extreme amount of resources (twice-triple the amount it used before in "old" versions).

Please note --- this is not a configuration issue of the Raspberry as I have three of them running and also testing other forks which introduced a similar behavior when incorporating chia 1.2.9-10 codebase commits. Flax has more or less fixed it updating from 0.1.4 to 0.1.5 (chia 1.2.11).

Can you please take a look at the commits introduced with chia 1.2.11 and incorporate them into the SIT codebase? Thank you!

//It looks like a lot of people have a performance problem with the harvester, plot recognitnion, processing of plots etc. Especially if the amount of plots is in higher numbers the performance drop is drastical. Starts around 1000+ plots, maybe earlier. Plot recognition after a new start takes forever, resource usage (CPU) compared to other forks is tripled or more etc.

The dev branch code has merged Chia 1.2.11 version. You can try to switch to the dev branch and install that version.

PrEzi3 commented 2 years ago

Here some first results -- ---installed from the dev branch just fine ---initial plot recognition still takes forever vs other forks plus the initial cpu usage is also extremely high -- 100% 1 core ---after it reads all the plots and settles (ca 10-15 minutes) it might behave a little better vs. 1.1.3 (better caching ? ) --- spiking to 100% 1 core usage seem to occur not so frequent as in 1.1.3.... but -- preliminary result -- further observations needed ---still one of the top processes in regard of cpu consumption, which is a pity...

PrEzi3 commented 2 years ago

Seems there was no change in the cpu usage or speed of harvester with the dev branch... Plot recognition still incredibly slow. 100% cpu 1 core usage when doing that. Plot harvesting still uses triple or more cpu resources than other forks.

Have still a feeling that it goes a little bit less "berserk" and uses 100% a little bit less but can be a placebo.

PrEzi3 commented 2 years ago

Nah... ok it was a placebo. CPU Usage and spikes are still roughly the same...
Since only SIT has this problem (and some forks based on chia 1.2.09-10 code) and as you state - the code was merged with chia 1.2.11 (which fixed most of the issues) I assume the problem might be in the introduced staking algorithm that would require optimization ? But hey.... at least I can "enjoy" now the newest branch on a python 3.9.x system... so that's a positive :)

centrd commented 2 years ago

The following is observed on a beefy system with a number of forks running. These issues are not observed on any of the other forks. OS is Windows Server 2019, fully updated. Dual CPU E5-26xx series. System is running a full SIT node, farmer, and harvester.

Motophan commented 2 years ago

looks like a python global lock caused by the staking code.

albertcoady commented 2 years ago

The following is observed on a beefy system with a number of forks running. These issues are not observed on any of the other forks. OS is Windows Server 2019, fully updated. Dual CPU E5-26xx series. System is running a full SIT node, farmer, and harvester.

  • Seeing the plot recognition perf problem on my farm too, with version 1.1.5. The more plots you have, the worse it is.
  • When switching to the staking tab in the GUI, the action seems to cause a blocking event or to otherwise affect the harvester or farmer, because I consistently see extremely long lookup times after switching to the tab (as reported by ForkFarmer). We're talking 60+ seconds lookup time. The log generation seems to be affected by the event, as there is a significant delay, followed by a dump of a few lines in the log. An error message of "harvester did not respond" also shows up, though I have not checked whether this happens every time.

It will look up your plot farmer key in your plot files, so it will take some time to get the result.

PrEzi3 commented 2 years ago

Still --- this behavior is observed ONLY in SIT. Only after staking was introduced. It eats up a whole CPU core. Behavior worsens the more plots you load. Impacts everything --- plot loading, harvesting, cpu resources...

Motophan commented 2 years ago

maybe this is a new issue but Linux CPU time silicoin is 5x any other coin including chia with their daily dust storms.

On Tue, Dec 21, 2021, 12:52 AM PrEzi3 @.***> wrote:

Still --- this behavior is observed ONLY in SIT. Only after staking was introduced. It eats up a whole CPU core. Behavior worsens the more plots you load. Impacts everything --- plot loading, harvesting, cpu resources...

— Reply to this email directly, view it on GitHub https://github.com/silicoin-network/silicoin-blockchain/issues/175#issuecomment-998632111, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKEIUFBVLAXUHME7WXO22J3USBE5NANCNFSM5KABNMCA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

PrEzi3 commented 2 years ago

Just to make sure I have made it clear enough..... Even though changelog states otherwise --- The issue is still not solved in SIT 1.2.0.

PrEzi3 commented 2 years ago

Bump

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

Bump. Still not solved.

Motophan commented 2 years ago

bump not solved

On Fri, Jan 14, 2022, 2:30 AM PrEzi3 @.***> wrote:

Bump. Still not solved.

— Reply to this email directly, view it on GitHub https://github.com/silicoin-network/silicoin-blockchain/issues/175#issuecomment-1013040106, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKEIUFE4BJLNPN5WTJ4AM6LUWACMPANCNFSM5KABNMCA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

PrEzi3 commented 2 years ago

Bump - not solved.

PrEzi3 commented 2 years ago

BUMP

Valdgeer commented 2 years ago

It's the same for me. Harvester eats all 1 CPU core (it has 2 CPUs and 32 threads at its disposal). At 50TB it still works, but more often it takes 100% of the core and does not work at all with the entire 800TB farm. I do not recommend anyone to seriously get involved in this project. Waste of time. Anyway, when I look at the price, it's not just me. The other forks and Chia work very well on the same hardware.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

BUMP -- NOT solved.

PrEzi3 commented 2 years ago

Bump...

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

bump - not solved

PrEzi3 commented 2 years ago

bump not solved

PrEzi3 commented 2 years ago

bump - not solved

PrEzi3 commented 2 years ago

aaaaand it is still not solved..... I think silicoin will just die as a crypto before anything meaningful happens here....

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

LeroyINC commented 2 years ago

is the development of this dead?

centrd commented 2 years ago

do yourself a favour and move on.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

PrEzi3 commented 2 years ago

.

github-actions[bot] commented 2 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

github-actions[bot] commented 2 years ago

This issue was automatically closed because it has been flagged as stale and subsequently passed 7 days with no further activity.