search

yacy / yacy_search_server

Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance
http://yacy.net
Other
3.4k stars 426 forks source link

Maximum of System-Load (for Local Crawl) is hard-limited to 8.0 #363

Closed onlyjob closed 1 year ago

onlyjob commented 4 years ago

I'm running YaCy on a busy server with normal load average greater than 10. Server have a lot of resources to spare and average load is mostly due to IOWAIT time on some local HDDs allocated to a different task.

To operate YaCy I had to adjust Maximum of System-Load values (in System Administration --> Performance Settings of Queues and Processes) by multiplying default values by 10.

However Maximum of System-Load value for Local Crawl is being automatically reset to 8.0 which effectively stops crawling almost entirely.

How can I adjust or disable the hard-coded limit for Local Crawl Maximum of System-Load?

Thanks.

onlyjob commented 4 years ago

Maximum of System-Load value for Local Crawl resets when I click Set button on Crawler_p.html page... This is unexpected, un-intuitive, unnecessary and very inconvenient.

lflare commented 2 years ago

Note, this issue is still present. The following line of code may be relevant to solving this. https://github.com/yacy/yacy_search_server/blob/9c38b1254ecd7693b2d49ff9c5b25533496b47f9/htroot/Crawler_p.java#L786

Orbiter commented 1 year ago

We still need to set the load in the process because a demand for higher crawl speed may require to increase the maximum load limit. However, following the criticism in the bug, we do never reduce the load limit again.