search

LOD-Laundromat / lodlaundry.github.io

http://lodlaundromat.org
2 stars 2 forks source link

Add default nice values #78

Closed LaurensRietveld closed 9 years ago

LaurensRietveld commented 9 years ago

Suggestion: set nice value of 10 for all daemon processes (i.e., only the washing machine process. All child processes (e.g. the callback) inherit this nice value) That way, we won't interfere with other users that much

LaurensRietveld commented 9 years ago

In the prolog initialization script, we'd need the equivalent to this bash command: renice -n 10 $$

hrbazoo commented 9 years ago

Hi,

Suggestion: set nice value of 10 for all daemon processes (i.e., only the washing machine process. All child processes (e.g. the callback) inherit this nice value) That way, we won't interfere with other users that much

I don’t really think its needed. From what I see there are many cores left almost idle. Moreover, there are so many documents in the pipeline waiting to be washed. My suggestion is the opposite, to let lodlaundromat consume more resources to speed up the washing machine ;-)

Cheers, Hamid

LaurensRietveld commented 9 years ago

discussed this a bit internally. We'll leave things be for now (i.e. nice 0), as the bottle-neck is disk i/o anyway (we barely fully use the CPU anyway, because of the busy disk). Lets just monitor the CPU use, and keep this ticket open for now

wouterbeek commented 9 years ago

We have now observed that disk I/O is the only hardware-related bottleneck of LOD Laundromat scrape 12 (thanks to many improvements w.r.t. earlier versions!). I'm closing this topic.