Closed jgrunzweig closed 3 years ago
I think this boils down to the isalnum()
checks in https://github.com/VirusTotal/yara/blob/master/libyara/scan.c#L672-L677 which are locale aware. I haven't figured out what locale python is using that is triggering this, but if you try setting LC_CTYPE=POSIX
in your environment before running the python script it will work.
One possible solution is to not use isalnum()
here and to roll our own version of it. Curious if @plusvic has thoughts on it, as I'm happy to do the implementation.
That's interesting, I couldn't imagine that isalnum()
is locale-aware. I agree on implementing our own version of isalnum()
, go head with it if you want.
I wrote a short script that checks to see what values isalnum()
cares about on my system and the results are fascinating to me: https://gist.github.com/wxsBSD/5cc0aab5fc99eccaf20eb95b94ed79e8. All the lines with matches are because isalnum()
did not consider that byte alphanumeric. Things behave as I would expect for parts of it. That is, 0x30-0x39, 0x41-0x5a and 0x61-0x7a are all considered alphanumeric, but things get weird around 0xaa and later when it starts NOT matching (because isalnum()
is considering those bytes alphanumeric.
An interesting point is that https://gist.github.com/wxsBSD/e817cef3444b16a0669e52cbd9a7a958 shows a simple C program behaves as I would expect. I'm sure this is some unicode thing that I am terrible at but I'll go ahead and implement a check for 0x30-39, 0x41-0x5a and 0x61-0x7a only.
I did just notice this also lives in https://github.com/VirusTotal/yara/blob/master/libyara/re.c#L105 so I'll also investigate that. I suspect it is also misbehaving but only when running inside python.
I think I have dealt with something similar (or maybe the same) some time ago (2 years I think). This is a comment I left in the code after the fix.
// Enforce C locale because if we are called from Python bindings
// UTF-8 locale seems to be used. Mixing default <cctype> functions
// with non-default locales seems to be not portable across different systems.
//
// Linux: worked as usual
// macOS: expected 'char' instead of 'unsigned char'
// Windows: asserted on signed types
static std::locale cLocale("C");
Then, when calling std::isalnum()
we do it as std::isalnum(x, cLocale);
. So calling this function instead - https://en.cppreference.com/w/cpp/locale/isalnum. I'm not exactly sure how to enforce locale in C because I'm mostly versed in C++ but it might help.
@metthal - I think you're right but rather than deal with it and any edge cases that may come up from my very limited understanding of locales I figure it is probably best to just be very explicit on what we consider alphanumeric (anything in 0-9a-zA-Z range). I realize this is very English centric but I think that was the original intention of the code.
I've discovered unexpected results while using the 'fullword' keyword modifier in yara-python on OSX. Note that this issue likely only effects OSX, as I was unable to reproduce on Linux and Windows environments.
Example rule:
Example file (Also attached): yara_issue.bin.zip
Steps to reproduce:
Removing the fullword modifier results in a match. It appears as though yara-python is incorrectly identifying the surrounding characters around 'Load' and inappropriately believing them to be alphanumeric.
Appreciate any help in resolving this issue.