XOR key value is zero when string to match is less than 5 characters

[melomac]

I am very excited by the new XOR key option and noticed the XOR key is always zero in my tests with the yara-python project. I realized this actually is a YARA problem as the CLI won't print the XOR key either:

yara -s -X test_rule.yara $TMPDIR/CodeRunner/Untitled 
oneTwoThree /var/folders/4n/4cnph2ps2t77b9xms19fsqrc0000gn/T//CodeRunner/Untitled
0x3fb0:$:xor(0x00): 123
0x3fb4:$:xor(0x00): abc

Here you have the original story from the now closed issue on the yara-python project:

I am very excited by StringMatch and StringMatchInstance objects in yara-python version 4.3 release candidate and also the XOR key property.

I needed to support the new objects and also wanted to implement the XOR property in some of my code.

Out of curiosity, I created this Mach-O test file using CodeRunner for example:

#import <Foundation/Foundation.h>

int main(int argc, char *argv[]) {
    @autoreleasepool {
        NSString *a = @"123";
        NSString *b = @"abc";
    }
}

and this YARA test rule:

rule oneTwoThree {
    strings:
        $ = "123" xor
    condition:
        any of them
}

The string 123 should match the NSString a C string with XOR key 0 and the NSString b C string with XOR key P:

>>> from Crypto.Cipher import XOR
>>> XOR.new(b"P").encrypt(b"123")
b'abc'

On compiling the rule file and looking for matches in the compiled Mach-O file, I am getting the two instances as expected:

>>> matches[0]
oneTwoThree
>>> matches[0].strings[0]
$
>>> matches[0].strings[0].is_xor()
True
>>> matches[0].strings[0].instances
[123, abc]

But the XOR key value is 0 in both cases:

>>> matches[0].strings[0].instances[0].xor_key
0 / 0x0  # expected
>>> matches[0].strings[0].instances[1].xor_key
0 / 0x0  # not expected: this should be the ordinal value of P i.e. 80 / 0x50

Would you please be so kind to consider this as a problem to fix for version 4.3 final candidate?

Many thanks for this new feature that will be very interesting to work with in a near future!

[melomac]

This looks like XOR key is working great when the string to match length is greater than 4.

Test file:

#import <Foundation/Foundation.h>

int main(int argc, char *argv[]) {
    @autoreleasepool {
        NSString *a = @"12345";
        NSString *b = @"abcde";
        NSString *d = @"dummy";
        NSString *c = @"etllxfwoo{";
    }
}

Test 1 ok ✅

rule oneTwoThree {
    strings:
        $a = "abcde" xor
    condition:
        all of them
}

rule test {
    strings:
        $a = "dummy" xor
    condition:
        $a
}

yara -s -X test_rules.yara $TMPDIR/CodeRunner/Untitled

oneTwoThree /var/folders/4n/4cnph2ps2t77b9xms19fsqrc0000gn/T//CodeRunner/Untitled
0x3f98:$a:xor(0x50): 12345
0x3f9e:$a:xor(0x00): abcde

test /var/folders/4n/4cnph2ps2t77b9xms19fsqrc0000gn/T//CodeRunner/Untitled
0x3fa4:$a:xor(0x00): dummy
0x3faa:$a:xor(0x01): etllx
0x3faf:$a:xor(0x02): fwoo{

Test 2 ko ❌

rule oneTwoThree {
    strings:
        $a = "abcd" xor
    condition:
        all of them
}

rule test {
    strings:
        $a = "dummy" xor
    condition:
        $a
}

yara -s -X test_rules.yara $TMPDIR/CodeRunner/Untitled

oneTwoThree /var/folders/4n/4cnph2ps2t77b9xms19fsqrc0000gn/T//CodeRunner/Untitled
0x3f9c:$a:xor(0x00): 1234
0x3fa1:$a:xor(0x00): abcd

test /var/folders/4n/4cnph2ps2t77b9xms19fsqrc0000gn/T//CodeRunner/Untitled
0x3fa6:$a:xor(0x00): dummy
0x3fac:$a:xor(0x01): etllx
0x3fb1:$a:xor(0x02): fwoo{

On compiling YARA with --with-debug-verbose=2, the output reveals _yr_scan_xor_compare() isn't invoked when a XOR'd string to match is less than 5 characters. Attaching output.

Terminal Saved Output.txt

[wxsBSD]

This is definitely a bug, thanks for finding it and for the detailed analysis. It looks like if the string fits in a single atom we do not record the XOR key properly. I'm working on a fix for this now.

[wxsBSD]

Just pushed a fix for this in #1849. I didn't want to put it in a different branch since this branch was really helpful in seeing the plaintext while debugging.

This bug has likely existed since the xor modifier went in, and you were right that it only happens on short strings (which was a significant clue for what is going on).

[melomac]

Many thanks for your reactivity, this sets the tone of a promising year 👏