Support latest kernel and compilers

[benlaurie]

I get this:

DEBUG:qsym.afl:Temp directory=/tmp/tmpwCabXk
DEBUG:qsym.afl:Cannot find pintool. Maybe updating?
Traceback (most recent call last):
  File "bin/run_qsym_afl.py", line 47, in <module>
    main()
  File "bin/run_qsym_afl.py", line 41, in main
    e.run()
  File "/usr/local/google/home/benl/git-work/qsym/venv/local/lib/python2.7/site-packages/qsym-0.1-py2.7.egg/qsym/afl.py", line 355, in run
    self.run_file(fp)
  File "/usr/local/google/home/benl/git-work/qsym/venv/local/lib/python2.7/site-packages/qsym-0.1-py2.7.egg/qsym/afl.py", line 359, in run_file
    check_so_file()
  File "/usr/local/google/home/benl/git-work/qsym/venv/local/lib/python2.7/site-packages/qsym-0.1-py2.7.egg/qsym/afl.py", line 61, in check_so_file
    FATAL("Cannot find SO file!")
NameError: global name 'FATAL' is not defined

[insuyun]

Thanks, Ben. I will add Dockerfile to ease your setup. Sorry for inconvinience!

[insuyun]

@benlaurie Hi, Ben. Could you check the latest README.md and Dockerfile? Thanks.

[tunz]

I think you need to run python setup.py build to build the libqsym.so file. It works for me.

[insuyun]

@tunz I think pip install . will do that for you. no?

[tunz]

yes, it seems like pip install . works well.

[benlaurie]

I'd rather not add docker to the list of stuff I have to mess with.

Are you saying there's a step I missed? Where am I supposed to run what?

[benlaurie]

Ah, I see the revised instructions. Now I get:

  g++ -DBIGARRAY_MULTIPLIER=1 -Wall -Werror -Wno-unknown-pragmas -fno-stack-protector -DTARGET_IA32E -DHOST_IA32E -fPIC -DTARGET_LINUX  -I/tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/source/include/pin -I/tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/source/include/pin/gen -I/tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/extras/components/include -I/tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/extras/xed-intel64/include -I/tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/source/tools/InstLib -O3 -fomit-frame-pointer -fno-strict-aliasing  -I/tmp/pip-req-build-o3KA32/qsym/pintool -g -Wno-error=unused-function -std=c++11 -DCONFIG_CONTEXT_SENSITIVE -I/tmp/pip-req-build-o3KA32/third_party/z3/include  -c -o obj-intel64/dependency.o dependency.cpp
  In file included from /tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/source/include/pin/pin.H:41:0,
                   from analysis_instruction.h:4,
                   from analysis_instruction.cpp:2:
  /tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/source/include/pin/compiler_version_check2.H:102:2: error: #error The C++ ABI of your compiler does not match the ABI of the pin kit.
   #error The C++ ABI of your compiler does not match the ABI of the pin kit.
    ^~~~~
  /tmp/pip-req-build-o3KA32/third_party/pin-2.14-71313-gcc.4.4.7-linux/source/include/pin/compiler_version_check2.H:117:2: error: #error This kit requires gcc 3.4 or later
   #error This kit requires gcc 3.4 or later
    ^~~~~

$ g++ --version
g++ (Debian 7.3.0-5) 7.3.0

[benlaurie]

This is when doing pip install .

[insuyun]

Could you let me know your environment to build this system? QSYM is only tested in Ubuntu 16.04 and 14.04 with compilers that are installed by apt.

[benlaurie]

This is Google's internal version of Debian. I have apt if I need to install different versions of things (probably), or I can take a look at debugging.

[insuyun]

Currently, QSYM is based on PIN 2.14 which is old, but stable version. PIN 3.x has some problems in supporting external libraries (https://github.com/JonathanSalwan/Triton/issues/324) and dynamo rio has problems in supporting z3 (https://github.com/DynamoRIO/dynamorio/issues/1881).

That's why I choose PIN 2.14 as our DBT, but it has several dependencies on compilers. I hope you can fix the issue, if not, I think it would be better to use Docker or VM. Sorry for that.

[benlaurie]

What version of gcc/g++ are you using?

[insuyun]

gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)

Quiet old one.

[benlaurie]

Hmm. Well, I made it build with gcc 7, but now when I run qsym it doesn't do anything. The root cause seems to be this file is missing: /usr/local/google/home/benl/git-work/qsym/venv/lib/python2.7/site-packages/qsym/../third_party/pin-2.14-71313-gcc.4.4.7-linux/pin.sh

[benlaurie]

Tests all still fail, by the way.

[insuyun]

@benlaurie Could you check your conf.py is the latest one? I fix some bug in conf.py. It should check other path if pin.sh does not exist in the path.

NOTE: https://github.com/sslab-gatech/qsym/blob/master/qsym/conf.py#L3

[benlaurie]

It would have been helpful to note that I have to redo pip install . to make that change work.

Anyway, now I hit:

Pin 2.14 kit 71293
E:  4.16 is not a supported linux release

This appears to originate from something that is supplied as a binary, so I can't easily fix it.

Presumably docker will not help with this.

[insuyun]

True. My bad.

[insuyun]

This issue is related to #9

[Jinmo]

There's -ifeellucky option for 4.x kernel in Pin 2.x version, it enables execution on unsupported linux versions. Maybe this can be a workaround? I didn't see any incompatible case on 4.4.0.

[insuyun]

Currently, I am using -ifeellucky option to support Ubuntu 16.04. But it seems like 4.17 is not supportable even with the option.

[benlaurie]

Not sure what has changed, but I'm now getting a different error:

timeout -k 5 90 /usr/local/google/home/benl/git-work/qsym/venv/lib/python2.7/site-packages/qsym/../../../../third_party/pin-2.14-71313-gcc.4.4.7-linux/pin.sh -ifeellucky -t /usr/local/google/home/benl/git-work/qsym/venv/lib/python2.7/site-packages/qsym/pintool/obj-intel64/libqsym.so -logfile /tmp/tmpaEmmUX/qsym-out-11/pin.log -i /usr/local/google/home/benl/git-work/qsym/afl-out/4/.cur_input -f 1 -o /tmp/tmpaEmmUX/qsym-out-11 -l 1 -b afl-out/4/bitmap -- ../boringssl/fuzz/fuzz/pkcs12 /usr/local/google/home/benl/git-work/qsym/afl-out/4/.cur_input
E: Unable to load /usr/local/google/home/benl/git-work/qsym/venv/lib/python2.7/site-packages/qsym/pintool/obj-intel64/libqsym.so: /usr/local/google/home/benl/git-work/qsym/venv/lib/python2.7/site-packages/qsym/pintool/obj-intel64/libqsym.so: undefined symbol: _ZN10LEVEL_BASE9KNOBVALUEImE10FromStringERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

[insuyun]

That is the function in the pin's static library. But I don't know why it cannot find symbol.

[richinseattle]

The comment on https://github.com/sslab-gatech/qsym/issues/14#issuecomment-422041964 says you are on Pin 2.x because of libz3 incompatibility. Have you tried boolector or STP as alternatives? They are actually more performant on bitvector solving in many cases.

[insuyun]

@richinseattle No. I have never tried them. I know that boolector and STP's are good for bitvectors, but z3 is easy to use for me.