Tracing - QEMU stuck on help screen?

[stevenagy]

Hey!

I posted an issue for angr (https://github.com/angr/angr/issues/2786#issuecomment-872264348) but I figured I'd post here as well since it's becoming a bit more archr-related. I'm basically trying to replicate the tracing code from here (https://github.com/angr/angr/pull/2441) but it doesn't seem that QEMU is actually invoking the target binary. Any ideas?

Best, -Steve

[Kyle-Kyle]

It looks like we are feeding some unknown arguments to QEMU when invoking it.

1. Can you check the version of your shellphish-qemu? We recently updated it and it has to be reinstalled to work correctly. If you install it through pip, please reinstall it. If you install it through source code, plz pull the latest code and run the rebuild.sh script to rebuild the binaries.
2. Can you share the full QEMU command? It can be done by adding logging.getLogger("archr.analyzers.qemu_tracer").setLevel("DEBUG") to your script.

[stevenagy]

Thanks for the reply! I'm on latest src and trying to run the rebuild script, but am running into the following build errors:

    [1546/1546] Generating interop man pages with a custom command
    Running Sphinx v1.8.5
    loading pickled environment... done
    building [mo]: targets for 0 po files that are out of date
    building [man]: all manpages
    updating environment: 12 added, 0 changed, 0 removed
    reading sources... [  8%] bitmaps
    reading sources... [ 16%] dbus
    reading sources... [ 25%] dbus-vmstate
    reading sources... [ 33%] index
    reading sources... [ 41%] live-block-operations
    reading sources... [ 50%] pr-helper
    reading sources... [ 58%] qemu-ga
    reading sources... [ 66%] qemu-ga-ref
    reading sources... [ 75%] qemu-qmp-ref
    reading sources... [ 83%] vhost-user
    reading sources... [ 91%] vhost-user-gpu
    reading sources... [100%] vhost-vdpa

    looking for now-outdated files... none found
    pickling environment... done
    checking consistency... done
    writing... qemu-ga.8 { } qemu-ga-ref.7 { } qemu-qmp-ref.7 { }
    build succeeded.

    The manual pages are in docs.
    Configuring CGC tracer qemu...
    Building CGC tracer qemu...
    Configuring CGC nxtracer qemu...
    Building CGC nxtracer qemu...
    Configuring CGC base qemu...
    Building CGC base qemu...
    Configuring Linux qemu...
    Building Linux qemu...
    running egg_info
    error: [Errno 13] Permission denied
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/bigdata/angr/shellphish-qemu/setup.py'"'"'; __file__='"'"'/mnt/bigdata/angr/shellphish-qemu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix= Check the logs for full command output.

[Kyle-Kyle]

That's weird. Are you installing it in the system python environment?(non-virtualenv environment) If yes, you should need to give sudo permission to rebuild.sh. Because it will be added into the system python packages, which requires root privilege. The preferred way to install shellphish-qemu is to run rebuild.sh in your angr virtualenv. (Sorry I forgot to mention this)

[stevenagy]

Sorry for the late reply! Yeah I'm doing it all on bare-metal. I re-ran the rebuild.sh (had to manually run the two commands inside it as sudo). Everything seems to have installed successfully but I still get the same No such file or directory: '/tmp/tracer-cq5ss3co.trace' error when I run my script.

What's the best way to check which shellphish-qemu is being invoked? There might be some artifacts leftover from when I was trying out Driller a few years back but I'm not sure how to verify that.

[Kyle-Kyle]

The best way is is to set its logger to DEBUG level using logging.getLogger("archr.analyzers.qemu_tracer").setLevel("DEBUG") If shellphish-qemu is invoked, it will print out the command line invoking shellphish-qemu. Then you can use the command line to verify whether it is working as intended. I hope this helps

[stevenagy]

This is what the debug log is showing:

DEBUG   | 2021-07-06 09:39:09,725 | archr.analyzers.qemu_tracer | launch QEMU with command: /tmp/archr_local_0rthxkrl/shellphish_qemu/fire /tmp/archr_local_0rthxkrl/shellphish_qemu/shellphish-qemu-linux-x86_64 -C /tmp/tracer_target_j95aw84c -d nochain,exec,page -D /tmp/tracer-fwa8je3z.trace -- ./sbof

Shouldn't it be loading the QEMU from where I built archr? I'm not sure why this tmp folder is showing up.

[Kyle-Kyle]

We invoke shellphish-qemu from a temporary folder because it is convenient to support both local and docker environments. One typical use of it is to trace programs built inside a docker container.

This folder exists within the archr.target context. What you can do is to add a stop at the end of the context and then invoke the command to see what went wrong.

And also, if you can share the program and then input, I can have a look at it.

[stevenagy]

Thanks for your help!

So running the shellphish-qemu binary directly (./shellphish-qemu/shellphish_qemu/bin/shellphish-qemu-linux-x86_64 -C tmp_coredump_dir -d nochain,exec,page -D qemu_trace.txt -- ./sbof) does work. I've attached the trace log (qemu_trace.txt) and the original binary (sbof.zip).

I'm not sure why the Python wrapper isn't working. But my understanding is that, to pass this off to angr.exploration_techniques.Tracer, all I really need (as the starting point for my concolic execution) is a list of the basic block addresses executed .

The trace log has a ton of other info -- is it safe to assume I can strip this down to just the block addrs?

[stevenagy]

Just thinking: is there any way to force the tracer to invoke from a non-tmp dir?

Alternatively, if I generate the trace file without the python wrapper, what archr functions are relevant to processing it? I've been going through the src but it's admittedly a little tough to understand.

[Kyle-Kyle]

You command works, which implies archr invokes a wrong shellphish-qemu. So, you do have some old Driller leftovers. This script is how we bundle shellphish-qemu and ship it to /tmp. If you do want to find out where the leftover is, you can follow what the script does. (python -c "import shellphish_qemu; print(shellphish_qemu.qemu_base())")

The trace log has a ton of other info -- is it safe to assume I can strip this down to just the block addrs?

No. tracer needs other information to create a relatively accurate state, e.g. memory mapping, reason of crashes.

is there any way to force the tracer to invoke from a non-tmp dir?

There is no documented ways to do it. But you can modify the source code here to achieve that for your specific case. Warning: this only works in LocalTarget.

Alternatively, if I generate the trace file without the python wrapper, what archr functions are relevant to processing it? I've been going through the src but it's admittedly a little tough to understand.

I agree that this "fire" mechanism makes the source code have a steep learning curve. If you want to "hijack" the trace results, here is what you want to modify.

[stevenagy]

So my old angr/driller/shellphish-qemu were installed with python2, yet the current ones are all installed with python3. The above command shows the correct shellphish-qemu when I run it with python3, but when running with python2 it (expectedly) shows the old leftovers. I've since uninstalled the old stuff with pip2, but given that I've been using python3 this whole time, how could it anything python2-installed have been interfering?

[stevenagy]

So now re-running my script (with python3 as I have been) shows the following:

$ python3 test.py
/usr/local/lib/python3.8/dist-packages/itanium_demangler-1.0-py3.8.egg/itanium_demangler/__init__.py:635: SyntaxWarning: "is not" with a literal. Did you mean "!="?
/usr/local/lib/python3.8/dist-packages/itanium_demangler-1.0-py3.8.egg/itanium_demangler/__init__.py:637: SyntaxWarning: "is not" with a literal. Did you mean "!="?
/usr/local/lib/python3.8/dist-packages/itanium_demangler-1.0-py3.8.egg/itanium_demangler/__init__.py:639: SyntaxWarning: "is not" with a literal. Did you mean "!="?
/usr/local/lib/python3.8/dist-packages/itanium_demangler-1.0-py3.8.egg/itanium_demangler/__init__.py:635: SyntaxWarning: "is not" with a literal. Did you mean "!="?
/usr/local/lib/python3.8/dist-packages/itanium_demangler-1.0-py3.8.egg/itanium_demangler/__init__.py:637: SyntaxWarning: "is not" with a literal. Did you mean "!="?
/usr/local/lib/python3.8/dist-packages/itanium_demangler-1.0-py3.8.egg/itanium_demangler/__init__.py:639: SyntaxWarning: "is not" with a literal. Did you mean "!="?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named shellphish_qemu
cp: -r not specified; omitting directory '/bin'
cp: -r not specified; omitting directory '/boot'
cp: -r not specified; omitting directory '/cdrom'
cp: -r not specified; omitting directory '/dev'
cp: -r not specified; omitting directory '/etc'
cp: -r not specified; omitting directory '/home'
cp: cannot stat '/initrd.img': No such file or directory
cp: cannot stat '/initrd.img.old': No such file or directory
cp: -r not specified; omitting directory '/lib'
cp: -r not specified; omitting directory '/lib32'
cp: -r not specified; omitting directory '/lib64'
cp: -r not specified; omitting directory '/libx32'
cp: -r not specified; omitting directory '/lost+found'
cp: -r not specified; omitting directory '/media'
cp: -r not specified; omitting directory '/mnt'
cp: -r not specified; omitting directory '/opt'
cp: -r not specified; omitting directory '/proc'
cp: -r not specified; omitting directory '/root'
cp: -r not specified; omitting directory '/run'
cp: -r not specified; omitting directory '/sbin'
cp: -r not specified; omitting directory '/snap'
cp: -r not specified; omitting directory '/srv'
cp: cannot open '/swapfile' for reading: Permission denied
cp: -r not specified; omitting directory '/sys'
cp: -r not specified; omitting directory '/tmp'
cp: -r not specified; omitting directory '/usr'
cp: -r not specified; omitting directory '/var'
cp: cannot stat '/vmlinuz': No such file or directory
cp: cannot stat '/vmlinuz.old': No such file or directory

After doing some digging in pip it seems to me that it's looking for python2 shellphish-qemu?

$ pip2 list | grep -F shellphish
$ pip3 list | grep -F shellphish
shellphish-qemu              0.12.3               /usr/local/lib/python3.8/dist-packages

EDIT: this seems similar to https://github.com/angr/archr/issues/12, yet my default python interpreter is python3.

[stevenagy]

After some re-installs it seems to work now :).