search

spencermountain / dumpster-dive

roll a wikipedia dump into mongo
Other
241 stars 46 forks source link

Number of CPU workers doesn't work #64

Open aymansalama opened 6 years ago

aymansalama commented 6 years ago

Hi again, I noticed that the number of CPU workers are much less than what is supposed to be even when specify --workers. The script seems to use on average of 5 cpu workers regardless of the size of the machine or the configuration available. 

**top - 05:23:18 up 13:27,  4 users,  load average: 6.98, 6.76, 5.55**
Tasks: 235 total,   6 running, 229 sleeping,   0 stopped,   0 zombie
%Cpu(s): 34.7 us,  3.7 sy,  0.0 ni, 61.1 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem : 61851660 total,   570464 free, 13277764 used, 48003432 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 47567560 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                             
17559 root      20   0 1531264 719272  25636 R 121.9  1.2  23:30.91 node                                
17565 root      20   0  981404 174652  25712 R 120.5  0.3  22:17.39 node                                
17567 root      20   0 1114476 313784  25476 R 119.5  0.5  23:01.75 node                                
17635 root      20   0  986044 179396  25524 R 119.5  0.3  22:20.90 node
spencermountain commented 6 years ago

may be an issue for the worker-nodes library we're using. we're spawning those workers, and it's up to your o/s who should run and when

aymansalama commented 6 years ago

@spencermountain thank you so much for keeping up with all of our questions and comments. Respect! for me it is totally okay, i can survive with 5 or 6 cores, I just wanted to highlight the behaviour.