add exim4 log parser

[juju4]

Trying to push a codereview for https://github.com/juju4/plaso/tree/exim followed https://github.com/log2timeline/plaso/wiki/Codereview

from my fork directory and feature branch

$ ./utils/review.py create
Traceback (most recent call last):
  File "./utils/review.py", line 2141, in <module>
    if not Main():
  File "./utils/review.py", line 2106, in Main
    if not review_helper.Lint():
  File "./utils/review.py", line 1675, in Lint
    if not pylint_helper.CheckUpToDateVersion():
  File "./utils/review.py", line 1139, in CheckUpToDateVersion
    exit_code, output, _ = self.RunCommand(u'pylint --version')
  File "./utils/review.py", line 49, in RunCommand
    arguments, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Only special things, I tried to use github personal token with minimal scope and not password in netrc You should documented if it's possible and with which (minimal) scope.

Thanks

[Onager]

It looks like pylint isn't installed - I'll add something to the codereview page about this - thanks for the report.

[juju4]

Documentation should advise to build a virtualenv with all requirements to execute codereview.

Even with that, I have dependencies failure.

$ cd plaso
$ virtualenv env-codereview
$ . env-codereview/bin/activate
$ pip install pylint
$ pip install -r requirements.txt
$ ./utils/review.py create
Running linter on changed files.
Checking: plaso/formatters/__init__.py
Checking: plaso/formatters/exim.py
Checking: plaso/parsers/__init__.py
Checking: plaso/parsers/exim.py
Checking availability and versions of dependencies. 
[OK]            Crypto version: 2.6.1
[OK]            IPython version: 5.1.0
[FAILURE]       missing: artifacts.
[FAILURE]       missing: bencode.
[FAILURE]       missing: binplist.
[FAILURE]       missing: construct.
[OK]            dateutil version: 2.6.0
[FAILURE]       missing: dfdatetime.
[FAILURE]       missing: dfvfs.
[FAILURE]       missing: dfwinreg.
[OK]            dpkt version: 1.8.8
[FAILURE]       missing: efilter.
[FAILURE]       missing: hachoir_core.
[FAILURE]       missing: hachoir_metadata.
[FAILURE]       missing: hachoir_parser.
[OPTIONAL]      missing: lzma.
[OK]            pefile version: 2016.3.28
[OK]            psutil version: 5.0.0
[FAILURE]       missing: pybde.
[FAILURE]       missing: pyesedb.
[...]

[joachimmetz]

Documentation should advise to build a virtualenv with all requirements to execute codereview.

documentation advises not to use virtualenv: https://github.com/log2timeline/plaso/wiki/Running-plaso-in-virtualenv

[joachimmetz]

Not sure why this is not working for you but the following works for me:

cd Projects/tmp/
virtualenv plasoenv
cd plasoenv/
source ./bin/activate
pip install --upgrade pip
curl -O https://raw.githubusercontent.com/log2timeline/plaso/master/requirements.txt
pip install -r requirements.txt
PYTHONPATH=~/Projects/plaso/ ~/Projects/plaso/utils/check_dependencies.py 

[juju4]

Ok, so tried with a pristine box (ubuntu 16.04.2), with and without virtualenv

root@dotest:~/plaso# apt -y install python python-pip python-virtualenv liblzma-dev
[...]
root@dotest:~/plaso# virtualenv ~/env-plaso
Running virtualenv with interpreter /usr/bin/python2
New python executable in /root/env-plaso/bin/python2
Also creating executable in /root/env-plaso/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
root@dotest:~/plaso# . ~/env-plaso/bin/activate
(env-plaso) root@dotest:~/plaso# pip install --upgrade pip
Requirement already up-to-date: pip in /root/env-plaso/lib/python2.7/site-packages
(env-plaso) root@dotest:~/plaso# curl -O https://raw.githubusercontent.com/log2timeline/plaso/master/requirements.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1128  100  1128    0     0   3188      0 --:--:-- --:--:-- --:--:--  3195
(env-plaso) root@dotest:~/plaso# pip install -r requirements.txt
[... OK]
(env-plaso) root@dotest:~/plaso# PYTHONPATH=~/plaso/ ~/plaso/utils/check_dependencies.py 
Checking availability and versions of dependencies.
[FAILURE]   missing: Crypto.
[FAILURE]   missing: IPython.
[FAILURE]   missing: artifacts.
[FAILURE]   missing: bencode.
[FAILURE]   missing: binplist.
[FAILURE]   missing: construct.
[FAILURE]   missing: dateutil.
[FAILURE]   missing: dfdatetime.
[FAILURE]   missing: dfvfs.
[FAILURE]   missing: dfwinreg.
[FAILURE]   missing: dpkt.
[FAILURE]   missing: efilter.
[FAILURE]   missing: hachoir_core.
[FAILURE]   missing: hachoir_metadata.
[FAILURE]   missing: hachoir_parser.
[OPTIONAL]  missing: lzma.
[FAILURE]   missing: pefile.
[FAILURE]   missing: psutil.
[FAILURE]   missing: pybde.
[FAILURE]   missing: pyesedb.
[FAILURE]   missing: pyevt.
[FAILURE]   missing: pyevtx.
[...]

outside of virtualenv

(env-plaso) root@dotest:~/plaso# deactivate
root@dotest:~/plaso# pip install -r requirements.txt
[...]
Installing collected packages: py, pytest, six, funcsigs, pbr, mock, scandir, pathlib2, pickleshare, simplegeneric, enum34, decorator, ipython-genutils, traitlets, backports.shutil-get-terminal-size, pygments, ptyprocess, pexpect, wcwidth, prompt-toolkit, IPython, PyYAML, XlsxWriter, artifacts, bencode, pytz, binplist, construct, dfdatetime, dfvfs, dfwinreg, dpkt, python-dateutil, efilter, hachoir-core, hachoir-metadata, hachoir-parser, libbde-python, libesedb-python, libevt-python, libevtx-python, libewf-python, libfsntfs-python, libfvde-python, libfwnt-python, libfwsi-python, liblnk-python, libmsiecf-python, libolecf-python, libqcow-python, libregf-python, libscca-python, libsigscan-python, libsmdev-python, libsmraw-python, libvhdi-python, libvmdk-python, libvshadow-python, libvslvm-python, future, pefile, psutil, pycrypto, pyliblzma, pyparsing, pytsk3, pyzmq, requests, yara-python
Successfully installed IPython-5.3.0 PyYAML-3.12 XlsxWriter-0.9.6 artifacts-20161022 backports.shutil-get-terminal-size-1.0.0 bencode-1.0 binplist-0.1.4 construct-2.5.3 decorator-4.0.11 dfdatetime-20170103 dfvfs-20160918 dfwinreg-20170327 dpkt-1.9.0 efilter-1453815385 enum34-1.1.6 funcsigs-1.0.2 future-0.16.0 hachoir-core-1.3.3 hachoir-metadata-1.3.3 hachoir-parser-1.3.4 ipython-genutils-0.2.0 libbde-python-20170204 libesedb-python-20170121 libevt-python-20170120 libevtx-python-20170122 libewf-python-20160802 libfsntfs-python-20170315 libfvde-python-20160801 libfwnt-python-20170115 libfwsi-python-20170117 liblnk-python-20160420 libmsiecf-python-20170116 libolecf-python-20170129 libqcow-python-20170222 libregf-python-20170130 libscca-python-20170205 libsigscan-python-20170124 libsmdev-python-20170225 libsmraw-python-20160424 libvhdi-python-20170223 libvmdk-python-20170226 libvshadow-python-20161111 libvslvm-python-20160110 mock-2.0.0 pathlib2-2.2.1 pbr-2.0.0 pefile-2016.3.28 pexpect-4.2.1 pickleshare-0.7.4 prompt-toolkit-1.0.14 psutil-5.2.1 ptyprocess-0.5.1 py-1.4.33 pycrypto-2.6.1 pygments-2.2.0 pyliblzma-0.5.3 pyparsing-2.2.0 pytest-3.0.7 python-dateutil-2.6.0 pytsk3-20170128 pytz-2017.2 pyzmq-16.0.2 requests-2.13.0 scandir-1.5 simplegeneric-0.8.1 six-1.10.0 traitlets-4.3.2 wcwidth-0.1.7 yara-python-3.5.0
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@dotest:~/plaso# PYTHONPATH=~/plaso/ ~/plaso/utils/check_dependencies.py 
Checking availability and versions of dependencies.
[OK]        Crypto version: 2.6.1
[OK]        IPython version: 5.3.0
[OK]        artifacts version: 20161022
[OK]        bencode
[OK]        binplist version: 0.1.4
[OK]        construct version: 2.5.3
[OK]        dateutil version: 2.6.0
[OK]        dfdatetime version: 20170103
[OK]        dfvfs version: 20160918
[OK]        dfwinreg version: 20170327
[OK]        dpkt version: 1.9.0
[OK]        efilter
[OK]        hachoir_core version: 1.3.3
[OK]        hachoir_metadata version: 1.3.3
[OK]        hachoir_parser version: 1.3.4
[OPTIONAL]  missing: lzma.
[OK]        pefile version: 2016.3.28
[OK]        psutil version: 5.2.1
[OK]        pybde version: 20170204
[OK]        pyesedb version: 20170121
[OK]        pyevt version: 20170120
[OK]        pyevtx version: 20170122
[OK]        pyewf version: 20160802
[OK]        pyfsntfs version: 20170315
[OK]        pyfvde version: 20160801
[OK]        pyfwnt version: 20170115
[OK]        pyfwsi version: 20170117
[OK]        pylnk version: 20160420
[OK]        pymsiecf version: 20170116
[OK]        pyolecf version: 20170129
[OK]        pyparsing version: 2.2.0
[OK]        pyqcow version: 20170222
[OK]        pyregf version: 20170130
[OK]        pyscca version: 20170205
[OK]        pysigscan version: 20170124
[OK]        pysmdev version: 20170225
[OK]        pysmraw version: 20160424
[OK]        pytsk3 version: 20161109
[OK]        pytz
[OK]        pyvhdi version: 20170223
[OK]        pyvmdk version: 20170226
[OK]        pyvshadow version: 20161111
[OK]        pyvslvm version: 20160110
[OK]        requests version: 2.13.0
[OK]        six version: 1.10.0
[OK]        xlsxwriter version: 0.9.6
[OK]        yaml version: 3.12
[OK]        yara version: 3.5.0
[OK]        zmq version: 16.0.2
[OK]        sqlite3 version: 3.11.0

If now I try the code review

root@dotest:~/plaso# pip install pylint
[...]
root@dotest:~/plaso# PYTHONPATH=~/plaso/ ~/plaso/utils/review.py create
Running linter on changed files.
Checking: plaso/formatters/__init__.py
Checking: plaso/formatters/exim.py
Checking: plaso/parsers/__init__.py
Checking: plaso/parsers/exim.py
Checking availability and versions of dependencies.
[OK]        Crypto version: 2.6.1
[...]
Tests that results are produced correctly. ... ok
testHasherInitialization (analyzers.hashing_analyzer.HashingAnalyzerTest)
Test the creation of the analyzer, and the enabling of hashers. ... ok
testFileRuleParse (analyzers.yara_analyzer.YaraAnalyzerTest)
Tests that the Yara analyzer can read rules. ... ok
testMatchFile (analyzers.yara_analyzer.YaraAnalyzerTest)
Tests that the Yara analyzer correctly matches a file. ... ok

======================================================================
ERROR: testExamineEventAndCompileReport (analysis.tagging.TaggingAnalysisPluginTest)
Tests the ExamineEvent and CompileReport functions.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/plaso/tests/analysis/tagging.py", line 100, in testExamineEventAndCompileReport
    plugin.SetAndLoadTagFile(test_file)
  File "./plaso/analysis/tagging.py", line 221, in SetAndLoadTagFile
    self._tag_rules = self._ParseTaggingFile(self._tagging_file_name)
  File "./plaso/analysis/tagging.py", line 146, in _ParseTaggingFile
    for label_name, rules in self._ParseDefinitions(tag_file_path):
  File "./plaso/analysis/tagging.py", line 97, in _ParseDefinitions
    query = self._ParseRule(rule)
  File "./plaso/analysis/tagging.py", line 127, in _ParseRule
    return efilter_query.Query(rule, syntax=syntax)
  File "/usr/local/lib/python2.7/dist-packages/efilter/query.py", line 93, in __init__
    (self.syntax, self.source))
ValueError: Cannot find parser for syntax u'objectfilter'. Source was "data_type is 'windows:prefetch'".

======================================================================
ERROR: testParseTaggingFile (analysis.tagging.TaggingAnalysisPluginTest)
Tests the _ParseTaggingFile function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/plaso/tests/analysis/tagging.py", line 132, in testParseTaggingFile
    tag_expression = plugin._ParseTaggingFile(test_path)
  File "./plaso/analysis/tagging.py", line 146, in _ParseTaggingFile
    for label_name, rules in self._ParseDefinitions(tag_file_path):
  File "./plaso/analysis/tagging.py", line 97, in _ParseDefinitions
    query = self._ParseRule(rule)
  File "./plaso/analysis/tagging.py", line 127, in _ParseRule
    return efilter_query.Query(rule, syntax=syntax)
  File "/usr/local/lib/python2.7/dist-packages/efilter/query.py", line 93, in __init__
    (self.syntax, self.source))
ValueError: Cannot find parser for syntax u'objectfilter'. Source was "data_type is 'windows:prefetch'".

----------------------------------------------------------------------
Ran 950 tests in 114.345s

FAILED (errors=2, skipped=3)
Create aborted - unable to pass tests.

=> seems fine on create review outside of objectfilter error (which I don't see related to my little exim parser).

So document rightfully says no virtualenv... Alternative reco, do it inside a lxd or docker container to avoid bad interactions.

[joachimmetz]

thx the issues you are encountering with objectfilter are related to the efilter package

Alternative reco, do it inside a lxd or docker container to avoid bad interactions.

For Ubuntu we recommend using the GIFT PPA https://github.com/log2timeline/plaso/wiki/Development-release-Ubuntu https://github.com/log2timeline/plaso/wiki/Dependencies---Ubuntu#prepackaged-dependencies

[juju4]

Concerning the efilter issue, is it a known bug? is there a workaround? exim4 log parsing is hardly linked to this error... Thanks

[joachimmetz]

Concerning the efilter issue, is it a known bug?

Not sure: https://github.com/google/dotty/issues

is there a workaround?

Not installing it via pip (from pypi)

[joachimmetz]

@juju4 were you able to resolve this or should I proceed and start a code review from our side?

[joachimmetz]

https://github.com/joachimmetz/plaso/tree/exim

I'll make some additional fixes for:

Traceback (most recent call last):
  File "tools/log2timeline_test.py", line 153, in testListParsersAndPlugins
    self.assertEqual(number_of_tables, 9)
AssertionError: 10 != 9

Could you also provide a test file and tests?

[joachimmetz]

relevant patch set: https://github.com/joachimmetz/plaso/commit/2249a040de64401f277706a1e3281cf8b91f01c6

[juju4]

Hello @joachimmetz I have not retested recently and I will not be able to do so before 10d or so. If you want to go forward, please do, else I will review when time permits. Thanks

[joachimmetz]

@juju4 I can, might take a bit, if you have test data that you can share that could be useful for me to add the missing tests.

[juju4]

My test case was honeynet challenge 7 https://www.honeynet.org/challenges/2011_7_compromised_server

[joachimmetz]

Thx I'll have a look if we can use this test file

[Onager]

@joachimmetz I see you have a branch for this parser - are you still working on this?

[joachimmetz]

I'll try to merge this as part of January features.

[joachimmetz]

License allows reuse of test file: https://creativecommons.org/licenses/by-nc-nd/3.0/

This work by Guillaume Arcas, Hugo Gonzales and Julia Cheng is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

[joachimmetz]

This will need some work since the current parser is directly based on syslog instead of a syslog plugin. @Onager this is something we likely need to document, what approach to a parser or plugin to use when.

[Onager]

Removing this from the January release

[joachimmetz]

So currently my test exim4 mainlog is not parsed by plaso, but the code provided by @juju4 does not address parsing the file. I'll write a syslog plugin from scratch.

[joachimmetz]

The log format is very similar to dpkg.log so the better approach is likely to refactor dpkg log to a syslog like plugin approach first then add the exim log parser.

[joachimmetz]

Blocked on https://github.com/log2timeline/plaso/issues/1656