Closed johncv23 closed 9 years ago
A few things about performance. 90k is no problem. How much memory do you have? How many worker processes? But you don't want to hammer the black list servers. So ideally only check them once a day if you can. Each worker is a separate process. They don't even know about each other kernel level so it should use everything you have. I tested it doing 500k a day 8 gigs ramen with 8 cores. It also depends how many rBLs your checking for each Host. Disable the ones you don't care about to speed it up a lot
On Jun 13, 2015, at 6:03 PM, johncv23 notifications@github.com wrote:
It seems this app/program is only taking advantage of 2 cores when I have 8 cores in a VM I increased the worker processes but it still is confining them to 2 cores and both those cores are at 100% 24/7
I have approximately 90000 IP addresses in blacklistmonitor so if I'm able to get it to span across more cores i'm sure it would go a lot faster
— Reply to this email directly or view it on GitHub.
Ok so the issue was the ubuntu version seems to be a known issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1386473
I upgraded to 15.04 and installed upstart so that this script works and it now uses all 8 cores
How many worker processes should I be using?
I am only checking the blacklists daily
I'd start with 5 per core. It's seems to use a bit of cpu with the MySQL and process switching but I think you can. Go higher depending on ram. I think you can turn it to what cpu isn't pegged but it's not swapping
On Jun 13, 2015, at 6:40 PM, johncv23 notifications@github.com wrote:
Ok so the issue was the ubuntu version seems to be a known issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1386473
I upgraded to 15.04 and installed upstart so that this script works and it now uses all 8 cores
How many worker processes should I be using?
I am only checking the blacklists daily
— Reply to this email directly or view it on GitHub.
It seems this app/program is only taking advantage of 2 cores when I have 8 cores in a VM I increased the worker processes but it still is confining them to 2 cores and both those cores are at 100% 24/7
I have approximately 90000 IP addresses in blacklistmonitor so if I'm able to get it to span across more cores i'm sure it would go a lot faster