Error: adding task to database

[allewwaly]

I install cuckoo on windows as instructed by http://cyberwarzone.com/installing-cuckoo-sandbox-on-a-windows-operating-system/.

Everything works find until I submit samples for analyzing via submit.py. The Error is:

WARNING:lib.cuckoo.common.objects:Unable to import pydeep (install with `pip ins
tall pydeep`)
Error: adding task to database

As pydeep is optional, I don't think it was the cause. Then I write a python script to test the connection to mysql database, it works fine too. The Cuckoo.py side works find just as follows:

2016-04-08 20:43:29,604 [lib.cuckoo.core.scheduler] INFO: Using "virtualbox" machine manager
2016-04-08 20:43:32,197 [lib.cuckoo.core.scheduler] INFO: Loaded 1 machine/s
2016-04-08 20:43:32,197 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks...

So the cuckoo.py is just waiting for analysis tasks but the submit.py never succeed in adding tasks.

The VM is a winxp whose network is configured as host-only, and it can ping through with the host. I hope that is not a problem.

[jbremer]

Are you doing this from the same user? Can you write to the cuckoo/storage/binaries directory?

[allewwaly]

A deeper insight shows that the "Error: adding task to database" is caused by an exception of following:

(OperationalError) (1048, **"Column 'file_type' cannot be null"**) 'INSERT INTO samples (file_size, file_type, md5, crc32, sha1, sha256, sha512, ssdeep) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)' (86528L, None, '92d905bdfe13c798a2cda2bbacdad932', 'FF729CFF', '66f1355f16ac1e328243e877880eb6e45e8b30e2', 'aab27bd5547d35dc159288f3b5b8760f21b0cfec86e8f0032b49dd0410f232bc', '5c96c7be222d5c836402df302f5a1866df72bcad3d13643e8703e536cea9c6e42fde344ca79d564051fd3cd93326e834b3c4b7f59e5591d61cba3d59b7c9a180', None) And the file_type being null is caused by the following exception:

Database.add()->file_type=obj.get_type()//database.py
get_type()->subprocess.Popen(stdout=PIPE)//objects.py
Popen()->**_execute_child()**//subprocess.py

The details of exception of _execute_child() is:

exc_type: <type 'exceptions.WindowsError'>
exc_value: [Error 2] 
exc_trace:<traceback object at 0x045412D8>

Using web.py to submit through web page outputs this error: Error: The server encountered an internal error while submitting example.exe

[allewwaly]

@jbremer yes, I run both cuckoo.py and submit.py as a administrator. What's different with linux is that there is no user group like vboxusers or libvirtd.

[allewwaly]

@jbremer Should the user running cuckoo be the same user of the database in cuckoo.conf? I've assign all the database operating privileges to another user and use it to connect the database in cuckoo.conf

[jbremer]

Heh, so Cuckoo is unable to determine the file magic on your machine, interesting. What OS are you running on the host?

[allewwaly]

I'm running Windows 7 Home Basie 64bit

[allewwaly]

After installing python-magic-0.4.11, it still doesn't work

[allewwaly]

It seems that "import magic" of object.py failed for unknown reason, which caused the file_type to be none.

[jbremer]

Can you open the Python interpreter and see if you get any interesting errors when trying to run import magic?

[allewwaly]

Can you open the Python interpreter and see if you get any interesting errors when trying to run import magic?

Yeah, seems like magic relies on libmagic...

>>> import magic
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\magic.py", line 171, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

[allewwaly]

Well, after install FILE for windows and configure as python-magic dependencies requires, I can now import magic successfully. But the magic function still failed to recognize the file type in https://github.com/cuckoosandbox/cuckoo/blob/master/lib/cuckoo/common/objects.py#L273. It just raise exceptions and return null.

[allewwaly]

Any ideas?

[jbremer]

Can you try manually and report said exceptions?

[allewwaly]

It is at https://github.com/cuckoosandbox/cuckoo/blob/master/lib/cuckoo/common/objects.py#L275, which raised an exception and failed to get file type. Then the exception handler raised another exception at https://github.com/cuckoosandbox/cuckoo/blob/master/lib/cuckoo/common/objects.py#L280. These two excpetions combined with that at https://github.com/cuckoosandbox/cuckoo/blob/master/lib/cuckoo/common/objects.py#L291 made the file_type being none . The reason caused these exceptions are not clear to me. Maybe some step intos before I find the real reason. It's awkward.

[jbremer]

Yeah, but can you log the exact exception? :-)

[allewwaly]

@jbremer The exceptions happened at line 278, 281, 286, 293 of objects.py are separately listed below:

(<type 'exceptions.AttributeError'>, AttributeError("'module' object has no attribute 'open'",), <traceback object at 0x032F54E0>)
(<class 'magic.MagicException'>, MagicException('could not find any magic files!',), <traceback object at 0x04512B48>)
(<type 'exceptions.UnboundLocalError'>, UnboundLocalError("local variable 'ms' referenced before assignment",), <traceback object at 0x044B5EE0>)
(<type 'exceptions.WindowsError'>, WindowsError(2, ''), <traceback object at 0x04512AA8>)

[allewwaly]

This magic library for windows seems different with that for linux. Seems that magic.open(), magic.load() and magic.close() are all different. Maybe some adjustments are required to accommodate it with windows.

For example, there is no magic.open() function defined in magic.py, instead magic.magic_open exists. So I modified the funtion names in object.py but found that the magic.load() function called in object.py has no args while magic.magic_load(cookie, filename) has, I'm not sure if they are correspond ones or just some mistake.

[allewwaly]

Problem solved by the following steps:

• I downloaded the 3 files from the GNUWin32 project (see "Dependencies on Windows")
• Instead of putting them in \Windows\system32, i found it better to add C:\Program Files (x86)\GnuWin32\bin to the PATH environment variable.
• Then i had a exception message "magic.MagicException: could not find any magic files" at process.py, so when instantiating the wrapper, instead of:

import magic magic.from_file(blah blah blah) I used

import magic m = magic.Magic(magic_file=r'C:\Program Files (x86)\GnuWin32\share\misc\magic', mime=True) m.from_file(blah blah blah)

[allewwaly]

What's confusing is that the report only contains results of process modules such as info, target, strings, statics and Virustototal. Other modules including behavior, debug, dropped files and network are all empty. Besides, during the running process, the debug always outputs "analysis not completed yet" and it ends with timeout. I've tried various samples and manually complete some samples to test, but they all seems to unable to complete and come to timeout in the end.

[allewwaly]

What's confusing is that the report only contains results of process modules such as info, target, strings, statics and Virustototal. Other modules including behavior, debug, dropped files and network are all empty. Besides, during the running process, the debug always outputs "analysis not completed yet" and it ends with timeout. I've tried various samples and manually complete some samples to test, but they all seems to unable to complete and come to timeout in the end.

Problem solved again by allowing python to access both private and public network in windows firewall :). Looks like agent cannot transfer the results back when public network access is denies by firewall.