search

mikelynn2 / blacklistmonitor

Web based application for monitoring Domains and IPs on RBLs
MIT License
85 stars 42 forks source link

Only 2 cores? #4

Closed johncv23 closed 9 years ago

johncv23 commented 9 years ago

It seems this app/program is only taking advantage of 2 cores when I have 8 cores in a VM I increased the worker processes but it still is confining them to 2 cores and both those cores are at 100% 24/7

I have approximately 90000 IP addresses in blacklistmonitor so if I'm able to get it to span across more cores i'm sure it would go a lot faster

mikelynn2 commented 9 years ago

A few things about performance. 90k is no problem. How much memory do you have? How many worker processes? But you don't want to hammer the black list servers. So ideally only check them once a day if you can. Each worker is a separate process. They don't even know about each other kernel level so it should use everything you have. I tested it doing 500k a day 8 gigs ramen with 8 cores. It also depends how many rBLs your checking for each Host. Disable the ones you don't care about to speed it up a lot

On Jun 13, 2015, at 6:03 PM, johncv23 notifications@github.com wrote:

It seems this app/program is only taking advantage of 2 cores when I have 8 cores in a VM I increased the worker processes but it still is confining them to 2 cores and both those cores are at 100% 24/7

I have approximately 90000 IP addresses in blacklistmonitor so if I'm able to get it to span across more cores i'm sure it would go a lot faster

— Reply to this email directly or view it on GitHub.

johncv23 commented 9 years ago

Ok so the issue was the ubuntu version seems to be a known issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1386473

I upgraded to 15.04 and installed upstart so that this script works and it now uses all 8 cores

How many worker processes should I be using?

I am only checking the blacklists daily

mikelynn2 commented 9 years ago

I'd start with 5 per core. It's seems to use a bit of cpu with the MySQL and process switching but I think you can. Go higher depending on ram. I think you can turn it to what cpu isn't pegged but it's not swapping

On Jun 13, 2015, at 6:40 PM, johncv23 notifications@github.com wrote:

Ok so the issue was the ubuntu version seems to be a known issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1386473

I upgraded to 15.04 and installed upstart so that this script works and it now uses all 8 cores

How many worker processes should I be using?

I am only checking the blacklists daily

— Reply to this email directly or view it on GitHub.