is_dll() function from pe module is not working properly on yara-python versions >= 4.0.0 (up to current 4.0.4)

[N0fix]

Hello,

I have noticed that is_dll() function in pe module from yara project is not working properly on yara-python from version 4.0.0 up to current one (4.0.4).

It seems like for the same version of yara-python and yara (the standalone executable), different results occur when using this specific function.

Reproducible example

$ yara -v 
4.0.2
$ cat dummy.yar
import "pe"
rule match_dll {
  condition: pe.is_dll()
}
$ yara -r dummy.yar kernel32.dll # any dll should give the same result
match_dll kernel32.dll

$ python3.8 -m pip install yara-python==4.0.2
#[installation]
$ cat test_script.py                                                      ~
import yara

r = yara.compile(source="""
import "pe" 
rule a{ 
    condition: pe.is_dll() 
}""")
print(r.match("kernel32.dll")) 
$ python3.8 test_script.py
[] # this should be returning a match

I got the kernel32.dll file used for the example from here. Its SHA256 shasum is : 0b2e367d671073efd70641a198c340b12b1bd813263050ae16b9c48b414775e0.

Further notes

I have been checking version number inside the library used by yara-python after installation (strings -a /home/myuser/myvenv/lib/python3.8/site-packages/yara.cpython-38-x86_64-linux-gnu.so | grep -E '4\.[0-9]' ) and the version of the library used by yara, and they match (4.0.2).
This behavior has been noticed on other computers too, and is not distribution related from what I could test. I tried to install yara in various ways (source code & distribution's package installer), and both confirm the behavior stated above.

After some testing with yara-python different versions packaged to pip, I noticed that this behavior start happening after version 4.0.0 included.

Cheers, Nofix

[wxsBSD]

Thanks for the detailed repro. I'll investigate this tonight.

[N0fix]

Further investigation might indicate that the problem is located here. The characteristics field would not be retrieved (returning YR_UNDEFINED), thus leading to the flag FILE_IS_DLL not to be read.

[wxsBSD]

I think you're close but probably not for the right reasons. I can replicate this problem. Here's what I've discovered so far:

• If you checkout yara-python at 4fa2fe8 (and make sure to update the yara submodule) you can replicate this problem.
• If you checkout yara-python at c1490e, which is the commit right before the bad one (and make sure to update the yara submodule) the problem isn't there.
• If you check the message value passed to yara_callback in yara-python.c the values passed in when it works are: import module (4), module imported (5), rule matched (1), scan finished (3). If you check when it doesn't work the values are: import module (4), module imported (5), rule not matched (2), scan finished (3). Given the logic of rule matching is done in libyara this indicates yara-python.c is not a problem.
• At yara-python 4fa2fe8 (the broken revision) the submodule is at b15d6b. If you check out b15d6b in the yara repro you can not replicate the problem there. This indicates it isn't a problem in libyara when built normally.
• At work we are currently on 4.0.2 (waiting on 4.1.0 release) but we build it with dynamic linking and we don't have the problem there. This tells me there is something funky going on when using yara-python 4.0.0 or higher and NOT using dynamic linking (ie: yara-python is building the submodule itself).
• When it breaks, it is because the is_undefined() check pointed out is returning true, despite the "characteristics" field always being set properly. Given that I can not trigger this when building libyara normally I suspect this is in the way the compiler is invoked when using setup.py.

I'm tired tonight, I'll keep digging tomorrow night.

[plusvic]

I found out that if you change the order of include statements in pe.c it works fine. Including yara/modules.h before yara/pe.h is enough to fix it, but didn't find the root cause yet.

[wxsBSD]

@metthal mentioned that this might be related to https://github.com/VirusTotal/yara-python/pull/159/ and he is right. I checked out 4fa2fe8 and then manually applied his patch to it and it's working as expected.

[N0fix]

Closing this issue since #159 has been accepted and merged into master. 4.0.5 seems to be working perfectly well. Thanks for your researches to find the source of this bug.