fsf in fails with yara-python 3.4.0.00 in virtualenv

[mpurzynski]

Hey!

I installed fsf in a virtualenv and stumbled upon a different problem

Test.zip fails to be analyzed

(fsf)brobuntu% ./fsf_client.py ../docs/Test.zip { "Scan Time": "2016-07-15 06:03:07.894859", "Filename": "Test.zip", "Source": "Analyst", "Object": { "META_BASIC_INFO": { "MD5": "f77489b604a106a0e0cd715d84a975a4", "SHA1": "872f221d3a893afb3895f3eae82b7877ba38b655", "SHA256": "1e97ee119b350f12ffb366aebab625a46c24d44f22416f289412e5de5053794d", "SHA512": "cd10a807da10ed796f1b9a8fa10e378efec9b12e16acbb0a983c3e2579f622dc6e96f65e69e19e056934ab1999777492638bb75dd04ed2b35a1e9944418af3a3", "ssdeep": "6144:Cl5QCGRIcQ5lunKBVOEfvWRmG7t1u9VhcLjs0ZAIdB084E5fnKESn4zbhxJfmeH1:CvQ1uf5l4KBIEfvW3wT6jjafLWK1nibV", "Size": "351521 bytes" } }, "Summary": { "Yara": [], "Modules": [ "META_BASIC_INFO" ], "Observations": [ "Only one kind of module was run on for this report.", "There doesn't appear to be any Yara signature hits for this scan." ] }, "Alert": false }

(fsf)brobuntu% cat /tmp/scan.log {"Scan Time": "2016-07-15 06:03:07.894859", "Filename": "Test.zip", "Source": "Analyst", "Object": {"META_BASIC_INFO": {"MD5": "f77489b604a106a0e0cd715d84a975a4", "SHA1": "872f221d3a893afb3895f3eae82b7877ba38b655", "SHA256": "1e97ee119b350f12ffb366aebab625a46c24d44f22416f289412e5de5053794d", "SHA512": "cd10a807da10ed796f1b9a8fa10e378efec9b12e16acbb0a983c3e2579f622dc6e96f65e69e19e056934ab1999777492638bb75dd04ed2b35a1e9944418af3a3", "ssdeep": "6144:Cl5QCGRIcQ5lunKBVOEfvWRmG7t1u9VhcLjs0ZAIdB084E5fnKESn4zbhxJfmeH1:CvQ1uf5l4KBIEfvW3wT6jjafLWK1nibV", "Size": "351521 bytes"}}, "Summary": {"Yara": [], "Modules": ["META_BASIC_INFO"], "Observations": ["Only one kind of module was run on for this report.", "There doesn't appear to be any Yara signature hits for this scan."]}, "Alert": false}

(fsf)brobuntu% cat /tmp/dbg.log 2016-07-15 06:03:07.923590 Failed to run module SCAN_YARA on 351521 byte buffer supplied for file Test.zip. Error: <class 'yara.SyntaxError'>

(fsf)brobuntu% which yara

/usr/local/bin/yara

(fsf)brobuntu% yara -v

yara 3.4.0

(fsf)brobuntu% pip list

argparse (1.2.1) cffi (1.7.0) ConcurrentLogHandler (0.9.1) czipfile (1.0.0) future (0.15.2) hachoir-core (1.3.3) hachoir-metadata (1.3.3) hachoir-parser (1.3.4) hachoir-regex (1.0.5) hachoir-subfile (0.5.3) javatools (1.3) oletools (0.47) pefile (2016.3.28) pip (1.5.4) pyasn1 (0.1.9) pyasn1-modules (0.0.8) pycparser (2.14) pyelftools (0.23) pylzma (0.4.8) PyPDF2 (1.26.0) rarfile (2.8) requests (2.10.0) setuptools (2.2) six (1.10.0) ssdeep (3.1.1) wsgiref (0.1.2) xmltodict (0.10.2)

yara-python (3.4.0.00) <-- should match

On the other hand, when I run yara manually, using your set of rules, it can find two exe files inside.

That is Ubuntu 14.04 LTS, newest master, yara-python from pip in a virtualenv.

[jxb5151]

Howdy @mpurzynski. I initially tried to replicate your issue but was unable to. After a bit of poking however, I think I figured out what happened.

When I installed the yara-python module using the following command:

(virtualenv) cartman@ubuntu:/tmp/virtualenv$ pip install yara-python

Then checked:

pip list
...
yara-python (3.4.0.0)

Then ran FSF, I was able to repeat the behavior.

However, when removing that module and instead installing Yara python directly from the source build, I was able to get this to work as intended. You'll note that this is the suggested path for installing in the guide, however, one would think the module from pip would work just as well?

wget https://github.com/plusvic/yara/archive/v3.4.0.tar.gz
tar -xvzf v3.4.0.tar.gz
cd yara-3.4.0/yara-python
python setup.py install **DO NOT SUDO**

I would recommend doing this as a stop gap until we can figure out why this is happening with the module distributed by pip?

My environment:

OS: Ubuntu 14.04 amd64

(virtualenv) cartman@ubuntu:/tmp/virtualenv$ yara -v
yara 3.4.0

(virtualenv) cartman@ubuntu:/tmp/virtualenv$ bin/pip list
cffi (1.7.0)
ConcurrentLogHandler (0.9.1)
czipfile (1.0.0)
future (0.15.2)
hachoir-core (1.3.3)
hachoir-metadata (1.3.3)
hachoir-parser (1.3.4)
hachoir-regex (1.0.5)
hachoir-subfile (0.5.3)
javatools (1.3)
oletools (0.47)
pefile (2016.3.28)
pip (8.1.2)
pyasn1 (0.1.9)
pyasn1-modules (0.0.8)
pycparser (2.14)
pyelftools (0.23)
pylzma (0.4.8)
PyPDF2 (1.26.0)
rarfile (2.8)
requests (2.10.0)
setuptools (24.0.3)
six (1.10.0)
ssdeep (3.1.1)
wheel (0.29.0)
xmltodict (0.10.2)
yara-python (3.4.0)

Details:

(virtualenv) cartman@ubuntu:/tmp/virtualenv/fsf/fsf-server$ ./main.py start
(virtualenv) cartman@ubuntu:/tmp/virtualenv/fsf/fsf-client$ ./fsf_client.py ../docs/Test.zip
{
    "Scan Time": "2016-07-15 11:51:22.647340",
    "Filename": "Test.zip",
    "Source": "Analyst",
... more JSON ...
    "Summary": {
        "Yara": [
            "compressed_exe_in_rar",
            "compressed_exe_in_zip",
            "ft_exe",
            "ft_office_open_xml",
            "ft_pdf",
            "ft_rar",
            "ft_zip",
            "misc_compressed_exe",
            "misc_ooxml_core_properties"
        ],
        "Modules": [
            "EXTRACT_EMBEDDED",
            "EXTRACT_RAR",
            "EXTRACT_ZIP",
            "META_BASIC_INFO",
            "META_OOXML",
            "META_PDF",
            "META_PE",
            "SCAN_YARA"
        ],
        "Observations": [
            "An executable was found inside a ZIP file.",
            "An embedded file contained a self-extracting RAR that itself contained an executable payload.",
            "More than 10 unique objects were observed in this file."
        ]
    },
    "Alert": false
}

[mpurzynski]

I have just tested it and you are right! Seems like something funky is going on with the python-yara package on pip.

Installed yara's python module from the source, inside the virtualenv and my fsf is happy.

[jxb5151]

Cool! I will close this out for now and have brought this to the attention of a friend close to the Yara project. I will re-open if there is anything we end up needing to change on our end.