syslog reporting module.

[mallorybobalice]

hello,

trying to configure modules/reporting/syslog.py and the upstream parser from it.

I'm seeing very frequent occurrences of 'Error creating syslog formatted log.' which looks to be a generic catch for createLog (I'm thinking one explanation is possibly tasks for corrupted (partial) files where the analysis handler in the guest didn't run properly for say a corruped exe but TBC)

What's the official module status/how do people normally use it?

Before I'll add a print trace to error catching bit around createLog and go from there...Want to see if we expect the module to normally work (and for the example could the maintainers have a new version that fixes the issue above due to a recent change in the field names, etc) , or if it's more an example module we'd base our specific ones on by removing the unrequired field? As far as i can tell the module is rather comprehensive but not sure.

thanks, mb

[KillerInstinct]

Hi,

I made this quite a while ago. I am not aware of any bugs with it other than UDP support doesn't work. You can see what the error message is by modifying lines https://github.com/spender-sandbox/cuckoo-modified/blob/master/modules/reporting/syslog.py#L202

to be:

except:
    import traceback
    print traceback.print_exc()
    raise CuckooReportError("Error creating syslog formatted log.")

This will drop you the traceback without halting the execution of other modules.

[mallorybobalice]

hello KillerInstinct,

thank you for the input and help

yup I've added the extra imports and tracebacks but didn't get the change to sort through the output yet.

There seem to be 3 places in it that are causing exceptions (missing ifs somewhere maybe?)

diff modules/reporting/syslog.py /modules/reporting/syslog.pyO

21,22d20
import traceback
import sys

205,209c202,203
         except Exception as e:
             exc_type, exc_value, exc_traceback = sys.exc_info()
             tbi = repr(traceback.format_exception(exc_type, exc_value, exc_traceback))
             tbi += ('\n' + repr(traceback.format_tb(exc_traceback)))
             raise CuckooReportError("Error creating syslog formatted log. %s" % tbi)

exceptions: (network and object most frequent, VT - rarely)

2016-04-13 16:40:20,377 [lib.cuckoo.core.plugins] WARNING: The reporting module "Syslog" returned the following error: Error creating syslog formatted log. ['Traceback (most recent call last):\n', '  File "modules/reporting/syslog.py", line 204, in run\n    result = self.createLog(results)\n', '  File "modules/reporting/syslog.py", line 76, in createLog\n    syslog += \'Object_Count="\' + str(len(results["static"]["Objects"])) + \'" \'\n', "KeyError: 'Objects'\n"]
['  File "modules/reporting/syslog.py", line 204, in run\n    result = self.createLog(results)\n', '  File "modules/reporting/syslog.py", line 76, in createLog\n    syslog += \'Object_Count="\' + str(len(results["static"]["Objects"])) + \'" \'\n']

2016-04-13 16:32:08,933 [lib.cuckoo.core.plugins] WARNING: The reporting module "Syslog" returned the following error: Error creating syslog formatted log. ['Traceback (most recent call last):\n', '  File "modules/reporting/syslog.py", line 204, in run\n    result = self.createLog(results)\n', '  File "modules/reporting/syslog.py", line 105, in createLog\n    for ip in results["network"]["hosts"]:\n', "KeyError: 'hosts'\n"]
['  File "modules/reporting/syslog.py", line 204, in run\n    result = self.createLog(results)\n', '  File "modules/reporting/syslog.py", line 105, in createLog\n    for ip in results["network"]["hosts"]:\n']

2016-04-13 11:40:50,258 [lib.cuckoo.core.plugins] WARNING: The reporting module "Syslog" returned the following error: Error creating syslog formatted log. ['Traceback (most recent call last):\n', '  File "modules/reporting/syslog.py", line 204, in run\n    result = self.createLog(results)\n', '  File "modules/reporting/syslog.py", line 133, in createLog\n    if \'positives\' in results["virustotal"]:\n', "KeyError: 'virustotal'\n"]
['  File "modules/reporting/syslog.py", line 204, in run\n    result = self.createLog(results)\n', '  File "modules/reporting/syslog.py", line 133, in createLog\n    if \'positives\' in results["virustotal"]:\n']

[KillerInstinct]

Yep, you're right -- I'm currently on travel, so I won't be able to fix it until early next week. But it's just adding a few if statements/conditions checking for those keys I believe.

EDIT: The "Objects" one actually is an error in handling the static analysis dict. I had forgotton that we changed up the static processing module to be more clear of results in the dict output. So instead of being results["static"]["Objects"] I believe it should be results["static"]["pdf"]["Objects"] or something similar -- I'd have to double check.

[mallorybobalice]

=). I I'll wait for you to have a look when you get the chance. Was going to have a look but then got caught up trying to figure out https://github.com/spender-sandbox/cuckoo-modified/issues/92. heh.

[KillerInstinct]

@mallorybobalice Give that PR a test run. I am currently unable to test the TCP/UDP portion as I don't really use this code anymore. I validated the syslog output however for various analysis tasks.

The reason you saw exceptions in different rarities, is likely due to the way I was handling the checks for dict fields. EG the VT one probably popped up after you ran several analysis back-to-back and rate limited your API key (or similar issue)

You'd get an exception for the PDF stuff every single PDF analysis as we changed up the static processing module (note the dict change in the git diff)

The network one likely happened when there was no pcap, or an error in the network processing module. There was also a bug in the host dict parsing which was now fixed -- I believe this was a result of fixing one of the ram_boost issues.

[mallorybobalice]

Much appreciated @KillerInstinctKillerInstinct . will try out when merging back from master in a few days.