Overflow error when using externals with large integers

[tlansec]

Hey,

I recently started using yara-python with some custom externals, one of which is the size of a file which can frequently be quite large.

When the size of the file is really large, YARA fails to evaluate this condition, here's some code and a rule to illustrate:

Rule - tmp.yar

rule tmp
{
    condition:
        size > 100000
}

Python - note that tmp.txt can be an empty file for the purposes of this test

import yara, sys 

sys.maxsize

externals = {
    'size' : 3352428544
}

rules = yara.compile("tmp.yar", externals=externals)

matches = rules.match("tmp.txt")

Results

λ python poc.py
Traceback (most recent call last):
  File "poc.py", line 9, in <module>
    rules = yara.compile("tmp.yar", externals=externals)
OverflowError: Python int too large to convert to C long

To verify, the same external when passed to the binary will work as expected.

This bug occurs on Win10x64, Python 3.6.8.

Cheers, Tom

[tlansec]

Also, to clarify, i realise filesize is available in regular YARA rules, but for this particular example I am reading in a large file containing metadata about files and applying rules based on this metadata and dont have copies of the files themselves.

[wxsBSD]

This is because you're overflowing the largest value for your system. We could switch it to use PyLong_FromUnsignedLong() (https://docs.python.org/3/c-api/long.html#c.PyLong_FromUnsignedLong) which would get us the upper bit but there is still a fundamental limit we would hit there too. Also, switching it to treating the external as an unsigned value may break people's usage of externals. At the end of the day, I'm not sure there is a good answer here...

[wxsBSD]

I did a bit more digging here after talking to Tom offline. I can not trigger this on a OS X system but installing yara-python 3.11.0 on Windows I can trigger this. It looks like on Windows the call to PyLong_As Long() at https://github.com/VirusTotal/yara-python/blob/master/yara-python.c#L994 throws if the value is 0x80000000 or higher but not on OS X. I suspect we can switch that to PyLong_AsLongLong and the Windows behavior will match OS X? I don't have a development environment setup for Windows setup to test it out though. :(

[plusvic]

Using PyLong_AsLongLong looks like the solution, I'm going to test it.