Magic module tests can sometimes fail depending on libmagic version

[MatejKastak]

Hi,

the following condition in tests fails: https://github.com/VirusTotal/yara/blob/ee6b2363f7d3bcb6b6bbe655ad8a71a451cc3af8/tests/test-magic.c#L26

because file identifies binary differently on different systems:

Centos7

[root@6c45817bcf04 /]# file --version
file-5.11
magic file from /etc/magic:/usr/share/misc/magic

# Running file on the binary blob saved to a file using python script
[root@6c45817bcf04 yara]# file /tmp/test.pe32
/tmp/test.pe32: MS-DOS executable

# Adding a prinft to https://github.com/VirusTotal/yara/blob/ee6b2363f7d3bcb6b6bbe655ad8a71a451cc3af8/libyara/modules/magic/magic.c#L150
MS-DOS executable

Arch linux

λ file --version
file-5.41
magic file from /usr/share/file/misc/magic
seccomp support included

λ file /tmp/test.pe32
/tmp/test.pe32: MS-DOS executable PE Unknown PE signature 0x103 Intel 80386, for MS Windows

# Printf in magic.c
# Only this output passes the test!
MS-DOS executable PE32 executable (GUI) Intel 80386, for MS Windows 

:question: In this output we can also see that the results differ from each other. The data in buffer is identified differently to data extracted to a file. I found that this might be caused by libmagic (see relevant part of their man page).

BUGS top The results from magic_buffer() and magic_file() where the buffer and the file contain the same data can produce different results, because in the magic_file() case, the program can lseek(2) and stat(2) the file descriptor.

There are maybe two questions:

• We are always calling the magic_buffer() function in magic.c module, so the results of the file command and yara output might be different
• Maybe we can add a mention to Magic module documentation that scanning buffers instead of files can have different results?

  There are two functions in this module: type() and mime_type(). The first one returns the descriptive string returned by file

Solution?

• We can change the PE32_FILE from blob.h in tests to something more reliable (maybe a new binary?) but I am not sure if this would work for all file versions
• We can relax the condition in tests?

Repro

FROM centos:7

# Install YARA build deps
RUN yum groupinstall -y "Development tools"
RUN yum install -y \
    automake \
    file-devel \
    jansson-devel \
    libtool \
    git \
    make

RUN cd /tmp \
    && git clone https://github.com/VirusTotal/yara \
    && cd yara \
    && ./bootstrap.sh \
    && ./configure --enable-magic \
    && make clean \
    && make \
    && make check

run with:

docker build -t yara-test .

Output

PASS: test-atoms
PASS: test-api
PASS: test-rules
PASS: test-pe
PASS: test-elf
PASS: test-version
PASS: test-bitmask
PASS: test-math
PASS: test-stack
PASS: test-re-split
PASS: test-async
PASS: test-exception
PASS: test-dotnet
FAIL: test-magic                     # <------------------------------------- FAILED MAGIC MODULE
make[4]: Entering directory `/tmp/yara'
Making all in libyara
make[5]: Entering directory `/tmp/yara/libyara'
make  all-am
make[6]: Entering directory `/tmp/yara/libyara'
make[6]: Nothing to be done for `all-am'.
make[6]: Leaving directory `/tmp/yara/libyara'
make[5]: Leaving directory `/tmp/yara/libyara'
make[5]: Entering directory `/tmp/yara'
make[5]: Nothing to be done for `all-am'.
make[5]: Leaving directory `/tmp/yara'
make[4]: Leaving directory `/tmp/yara'
============================================================================
Testsuite summary for yara 4.2.0
============================================================================
# TOTAL: 16
# PASS:  15
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to vmalvarez@virustotal.com
============================================================================
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory `/tmp/yara'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory `/tmp/yara'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/tmp/yara'
make: *** [check-recursive] Error 1
The command '/bin/sh -c cd /tmp     && git clone https://github.com/VirusTotal/yara     && cd yara     && ./bootstrap.sh     && ./configure --enable-magic     && make clean     && make     && make check' returned a non-zero code: 2

[plusvic]

I would relax the test to something like magic.type() contains \"MS-DOS executable\" which seems to be present in all cases. I didn't know about this discrepancy between magic_buffer() and magic_file(), it does't make much sense to me, my expectations are that magic_file() just reads the content of the file and passes it to magic_buffer().