18 VMs max loaded at the same time?

[Tesk]

Hello,

I have been trying to set this up to run as many VMs as possible on a server to process a large backlog of malwares.

But I encounter an strange issue. I have configured KVM with static ips for all the vms. But when I try to start more than 18 vm machines cuckoo never loads.

If I start 18 machines I can see that cuckoo stops the VMs one by one, but if I try with 19, it never stops a single VM and just hang. I need to kill it using another terminal (crtl + c not working in this state). If I configure machines in conf/kvm.conf: machines=cuckoo3,cuckoo4,cuckoo5,cuckoo6,cuckoo7,cuckoo8,cuckoo9,cuckoo10,cuckoo11,cuckoo12,cuckoo13,cuckoo14,cuckoo15,cuckoo16,cuckoo17,cuckoo18,cuckoo19,cuckoo20 it works.

If I do: machines=cuckoo3,cuckoo4,cuckoo5,cuckoo6,cuckoo7,cuckoo8,cuckoo9,cuckoo10,cuckoo11,cuckoo12,cuckoo13,cuckoo14,cuckoo15,cuckoo16,cuckoo17,cuckoo18,cuckoo19,cuckoo20,cuckoo21 It hangs.

If I do: machines=cuckoo3,cuckoo4,cuckoo5,cuckoo6,cuckoo7,cuckoo8,cuckoo9,cuckoo10,cuckoo11,cuckoo12,cuckoo13,cuckoo14,cuckoo15,cuckoo16,cuckoo17,cuckoo18,cuckoo19,cuckoo21 It works.

This is output from the logfile when playing around with the number of machines included in my config: 2017-04-29 08:58:20,988 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 08:59:38,220 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:03:53,280 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:10:25,497 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:12:40,649 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:19:24,994 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:19:25,729 [lib.cuckoo.core.scheduler] INFO: Loaded 1 machine/s 2017-04-29 09:19:25,738 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks. 2017-04-29 09:21:26,360 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:22:47,365 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:23:21,922 [lib.cuckoo.core.scheduler] INFO: Loaded 18 machine/s 2017-04-29 09:23:21,935 [lib.cuckoo.core.scheduler] WARNING: When running many virtual machines it is recommended to process the results in a separate process.py to increase throughput and stability. Please read the documentation about the Processing Utility. 2017-04-29 09:23:21,935 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks. 2017-04-29 09:23:48,547 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:28:56,707 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:30:34,947 [lib.cuckoo.core.scheduler] INFO: Using "kvm" machine manager with max_analysis_count=0, max_machines_count=0, and max_vmstartup_count=55 2017-04-29 09:31:06,709 [lib.cuckoo.core.scheduler] INFO: Loaded 18 machine/s 2017-04-29 09:31:06,722 [lib.cuckoo.core.scheduler] WARNING: When running many virtual machines it is recommended to process the results in a separate process.py to increase throughput and stability. Please read the documentation about the Processing Utility.

Are you aware of any limitations to the amount of VMs I can run at the same time?

[Nwinternights]

did you try to: set to "off" process_results in cuckoo.conf and start a new screen with "./utils/process.py auto" ? taken from https://github.com/spender-sandbox/cuckoo-modified/blob/55bafa2a325379418da9c2cdc66530458d827d17/docs/book/src/usage/utilities.rst

[Tesk]

Hey

Sorry for the late comment.

It does not seem to work.

I have tried it on the main fork of cuckoo 2 as well and it seems its suffering under the same limitation somehow.

[Tesk]

Okay, I figured it out finally.

The "issue" (its really not an issue) is within this file (on a main cuckoo 2.0.2 branch installed using pip, but same goes for any adaptions since its related to KVM): /usr/local/lib/python2.7/dist-packages/cuckoo/common/abstracts.py

When the module loads, this code executes: class LibVirtMachinery(Machinery): """Libvirt based machine manager.

If you want to write a custom module for a virtualization software
supported by libvirt you have just to inherit this machine manager and
change the connection string.
"""

# VM states.
RUNNING = "running"
PAUSED = "paused"
POWEROFF = "poweroff"
ERROR = "machete"
ABORTED = "abort"

def __init__(self):
    if not HAVE_LIBVIRT:
        raise CuckooDependencyError(
            "The libvirt package has not been installed "
            "(`pip install libvirt-python`)"
        )

    super(LibVirtMachinery, self).__init__()

def initialize(self, module):
    """Initialize machine manager module. Override default to set proper
    connection string.
    @param module:  machine manager module
    """
    super(LibVirtMachinery, self).initialize(module)

The last line "super(LibVirtMachinery, self).initialize(module)" initializes a new connection for each machine loading to the libvirt component from KVM.

A short look over the /etc/libvirt/libvirtd.conf and I saw that there is a "max_clients" which is defaulting to 20. I raised that value and I can now load 52 VM's at the same time using KVM.

I hope this may help someone else out in the future. :)

[doomedraven]

thanks for info :)