yarascan attribute error

[GoogleCodeExporter]

I haven't been able to duplicate this yet, but it seems that some people have 
had issues with yara 1.7 as seen in the links below:

http://pastie.org/pastes/8338566/text

http://list-archives.org/2013/04/27/cuckoo-public-honeynet-org/yara-problems-wit
h-cuckoo-0-6/f/5732666116

https://groups.google.com/forum/#!topic/yara-project/CQlMGS0sJaM

I've installed Yara 1.7, Python 2.7 and Volatility from svn and have tried to 
duplicate this bug, but for some reason it won't manifest for me :-(

Putting these links here so we don't lose them.

Original issue reported on code.google.com by jamie.l...@gmail.com on 19 Sep 2013 at 3:45

[GoogleCodeExporter]

2.2 Standalone exe works (I had wrongly mentioned earlier the opposite).

With 2.3, last night's build, I found the issue to be that match is simple a 
1-byte string of "n".

type(match) = <type 'str'>
len(match) = 1
match = n
match.encode('hex') = 6e
match.strings =  AttributeError: 'str' object has no attribute 'strings'

If I manually code what I see in malfind.py:

>>> rules = yara.compile(sources = {'n':'rule r1 {strings: $a = "SPLIT" 
condition: $a}'})
>>> matches = rules.match('e:\VMs\WinXP_Malware\WinXP_Malware.vmem')
>>> matches
{'n': [{'meta': {}, 'tags': [], 'matches': True, 'strings': [{'flags': 19, 
'identifier': '$a', 'data': 'SPLIT', 'offset': 57805694L}, {'flags': 19, 
'identifier': '$a', 'data': 'SPLIT', 'offset': 57805710L},  ...

>>> for i in matches.iterkeys(): print i
...
n
>>> for i in matches.itervalues(): print i
...
[{'meta': {}, 'tags': [], 'matches': True, 'strings': [{'flags': 19, 
'identifier': '$a', 'data': 'SPLIT', 'offset': 57805694L},  ...

I have no experience with the yara library, unfortunately, so the amount of 
debugging I can do is limited. I do have the memory image available (512MB raw) 
that I can FTP somewhere.

Original comment by brian@thebaskins.com on 19 Sep 2013 at 9:01

[GoogleCodeExporter]

Brian,

Where did you get your Yara code and how did you install it? For example did 
you checkout the svn trunk from 
https://code.google.com/p/yara-project/source/checkout or did you get the 
downloads from https://code.google.com/p/yara-project/downloads/list? 

Can you run "yara -v" on command line and see what the version string says? 

According to your output: 

"""
>>> matches
{'n': [{'meta': {}, 'tags': [],....
""""

the match() API is returning a dictionary, but ever since the beginning of Yara 
(January 2009) up to *and including* 1.7 (at least the installs that I've seen) 
return a list (aka array) not a dictionary. From the 1.6 documentation (there 
is no 1.7 document because I don't believe there were any significant changes 
to report) it says "The match method returns a list of instances of the class 
Match". An API change to return a dictionary instead of a list would be a 
significant piece of info to report (if not in the documentation at least in 
the change log and I don't see anything about it there either).  

Original comment by michael.hale@gmail.com on 20 Sep 2013 at 10:55

[GoogleCodeExporter]

On my Windows box, yara was installed via 'pip':
C:\Windows\system32>pip install yara --upgrade
Requirement already up-to-date: yara in c:\python27\lib\site-packages
Cleaning up...

C:\Windows\system32>pip install yara --upgrade --force
Downloading/unpacking yara
  Downloading yara-1.7.5.tar.gz (392kB): 392kB downloaded
  Running setup.py egg_info for package yara
    Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.25.tar.gz
    Extracting in c:\users\brian\appdata\local\temp\tmphngzy_
    Now working in c:\users\brian\appdata\local\temp\tmphngzy_\distribute-0.6.25
    Building a Distribute egg in c:\users\brian\appdata\local\temp\pip-build-Brian\yara
    c:\users\brian\appdata\local\temp\pip-build-Brian\yara\distribute-0.6.25-py2.7.egg

Installing collected packages: yara
  Found existing installation: yara 1.7.5
    Uninstalling yara:
      Successfully uninstalled yara
  Running setup.py install for yara
    SyntaxError: name 'scanner' is local and global (webapp.py, line 32)

    Installing yara-ctypes-script.py script to c:\python27\Scripts
    Installing yara-ctypes.exe script to c:\python27\Scripts
Successfully installed yara
Cleaning up...

C:\Windows\system32>python
Python 2.7.4 (default, Apr  6 2013, 19:55:15) [MSC v.1500 64 bit (AMD64)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import yara
>>> yara.version.__version__
'1.7.5'

However, I forced an install of 1.6.0 (oldest available on pip), and malfind.py 
had the same issue:
E:\Development\volatility>.\vol.py -f e:\VMs\WinXP_Malware\WinXP_Malware.vmem 
yarascan -Y "SPLIT"
Volatile Systems Volatility Framework 2.3_beta
Import yara version: 1.6.0
<type 'str'>
1
n
6e
Traceback (most recent call last):
  File "E:\Development\volatility\vol.py", line 186, in <module>
    main()
  File "E:\Development\volatility\vol.py", line 177, in main
    command.execute()
  File "E:\Development\volatility\volatility\commands.py", line 111, in execute
    func(outfd, data)
  File "E:\Development\volatility\volatility\plugins\malware\malfind.py", line 469, in render_text
    for o, addr, hit, content in data:
  File "E:\Development\volatility\volatility\plugins\malware\malfind.py", line 461, in calculate
    for hit, address in scanner.scan():
  File "E:\Development\volatility\volatility\plugins\malware\malfind.py", line 324, in scan
    for match in BaseYaraScanner.scan(self, vad.Start, vad.Length):
  File "E:\Development\volatility\volatility\plugins\malware\malfind.py", line 304, in scan
    for moffset, _name, _value in match.strings:
AttributeError: 'str' object has no attribute 'strings'

On my OSX box, pip was from the download list (yara-python-1.7.tar.gz).

Original comment by brian@thebaskins.com on 21 Sep 2013 at 1:41

[GoogleCodeExporter]

Ah, so if you get "yara" through pip, you actually get yara-ctypes 
(https://github.com/mjdorma/yara-ctypes) which is a bit different than 
yara-python. The yara-ctypes is a third party wrapper for libyara and appears 
to use slightly different APIs than the yara-python from the project's main 
download page. 

I would suggest installing these two packages since you are using x64 windows 
and python 2.7: 

https://yara-project.googlecode.com/files/yara-1.7-win64.zip
https://yara-project.googlecode.com/files/yara-python-1.7.win-amd64-py2.7.exe

They are both just click-through installers, but you may need to pip uninstall 
yara first to clean your system of the other files. Very strange that 
yara-python's match() API returns a list and yara-ctypes match() API returns a 
dictionary. 

We could support yara-ctypes easily in volatility's yarascan by just checking 
if match() returns a dict or list and handling it appropriately, but I would 
probably want to write the authors of yara and yara-ctypes first and make sure 
its not an accident on their part that the two APIs don't return consistent 
values. 

Original comment by michael.hale@gmail.com on 21 Sep 2013 at 10:43

[GoogleCodeExporter]

That was it!

I uninstalled pip's yara and installed the yara-python via the link above. 
Volatility is now working as expected:

E:\Development\volatility>.\vol.py -f e:\VMs\WinXP_Malware\WinXP_Malware.vmem 
yarascan -Y "SPLIT"
Volatile Systems Volatility Framework 2.3_beta
Rule: r1
Owner: Process java.exe Pid 1920
0x2abadbec  53 50 4c 49 54 03 03 03 69 70 3d 77 77 77 2e 6d   SPLIT...ip=www.m
0x2abadbfc  61 6c 77 61 72 65 2e 63 6f 6d 53 50 4c 49 54 09   alware.comSPLIT.
0x2abadc0c  09 09 09 09 09 09 09 09 70 61 73 73 3d 70 61 73   ........pass=pas
0x2abadc1c  73 77 6f 72 64 53 50 4c 49 54 0e 0e 0e 0e 0e 0e   swordSPLIT......

Original comment by brian@thebaskins.com on 21 Sep 2013 at 11:39

[GoogleCodeExporter]

Original comment by michael.hale@gmail.com on 23 Sep 2013 at 1:15

• Changed state: Done