Differentiate between a crash and unexpected behavior

[ameily]

We need to differentiate between a debloated binary crashing, which indicates that the debloater failed, and the binary performing unexpectedly (e.g.- a comparator detecting a difference). This may not be straightforward because we can't solely rely on the process exit code. We most likely need to detect a signal such as segfault or an error such as running an undefined opcode.

[ameily]

I have a small proof concept that uses os.waitpid to get the exit status of a program, which includes whether the process was terminated because of a signal and, if so, which signal. I've verified that exiting with the same exit code can be differentiated from one that exited with a signal. The downside is that we can't use Popen.wait and have to have our own loop that eventually times out.

[ameily]

This example uses the binrec eq2 which segfaults if no arguments are specified. In this case, the exit code is 139

import os
import signal
import subprocess
import time

BINARY = '../binrec-benchmark/samples/bin/x86/binrec/eq2'

def run(args):
    proc = subprocess.Popen([BINARY] + args)
    for i in range(5):
        pid, status = os.waitpid(proc.pid, os.WNOHANG)
        if pid == proc.pid:
            break
        time.sleep(0.5)

    print('result:', status)
    if status and os.WIFSIGNALED(status):
        sig = os.WTERMSIG(status)
        print('>> signal =', sig, '::', signal.Signals(sig).name)

if __name__ == '__main__':
    run(['a', 'b'])
    run([])

Output:

arguments are NOT equal
B
result: 0  # normal run, we are ok
result: 139  # segfault run
>> signal = 11 :: SIGSEGV