Support directed fuzzing

[spencerwuwu]

Hi! I'm trying to use magma to evaluate the performance of directed fuzzing. I think the idea of having patches with ground-truth bugs really makes it a wonderful data set.

I made some changes to make magma work for directed fuzzers. I use the 'patches/bugs' as goals for directed fuzzers to reach. You can find them in this fork: https://github.com/usc-isi-bass/magma/tree/aflgo

It is not yet completed though. I'm opening this issue to see if you guys are interested in this feature, and I can create a PR later.

Currently my fork can run aflgo on libpng, libxml2 without any issues. For other targets, I'm still trying to compile them. Also, the 'distance calculation' phase of aflgo for libxml2 takes extremely long time to compute. I might open an issue at aflgo to see if the authors can help for these problems.

[adrianherrera]

Cool! Definitely interested in this :) I actually did something very similar for an older version of Magma, but never got around to porting this to the current version.

Please let me know if you have any issues with the remaining targets; happy to help out where I can.

Are there other directed fuzzers we could add to compare against AFLGo? I know of Hawkeye, but afaik it was never open sourced.

[spencerwuwu]

Hi Adrian,

Great! And thank you! After some testing I believe the current scripts work for most targets except openssl & sqlite3 (regardless of the extremely long distance calculation for some of them).

Would you mind helping me to see why they fail to compile with AFLGo when you have time? The $TARGET/build.sh script failed for both target. I'd tried a few possibilities but couldn't make them work yet. I attached the captainrc to run one instance for both targets at the end of this comment.

For sqlite3, somehow the last step (compiling ossfuzz.c) cannot link external symbols properly.

For openssl, I'm getting errors like

clang (LLVM option parsing): for the -outdir option: may only occur zero or one times!
clang (LLVM option parsing): for the -targets option: may only occur zero or one times!

which are the additional flags set for AFLGo. The errors come from setting $CFLAGS in the cmd ./config --debug enable-fuzz-libfuzzer...... However, if I don't set these flags (by changing $CFLAGS to $COPY_CFLAGS, the original flags created in fuzzers/aflgo/instrument.sh), the errors become:

clang (LLVM option parsing): Unknown command line argument '-targets=/magma_out/temp/BBtargets.txt'.  Try: 'clang (LLVM option parsing) -help'
clang (LLVM option parsing): Did you mean '-stats=/magma_out/temp/BBtargets.txt'?

Do you know which level should we set these additional flags then?

---

For other directed fuzzers, I honestly don't have any in mind now that's patch-testing oriented and opensource like AFLGo. There is directed concolic tester based on KLEE though called KATCH. I guess it won't be so hard to include as there are already scripts for KLEE in Magma.

Btw, I may be trying to evaluate some hybrid fuzzers (Driller, QSYM, etc) with Magma as well soon. So scripts for those fuzzers can be expected. We can see if you would like to include them by then.

---

WORKDIR=./workdir
REPEAT=1
TIMEOUT=1m
CACHE_ON_DISK=0
CANARY_MODE=1
FUZZERS=()

DFUZZERS=(aflgo)
aflgo_TARGETS=(openssl sqlite3)

aflgo_openssl_BUGS=(AAH054)
aflgo_sqlite3_BUGS=(JCH214)

aflgo_libtiff_AAH009_PROGRAMS=(tiff_read_rgba_fuzzer)
aflgo_openssl_AAH054_PROGRAMS=(asn1)

[adrianherrera]

Cool!

I just checked out your repo and aflgo branch to try to build the various targets. Currently libpng fails to build with an Empty Ftargets.txt error. I'll look into it myself, but if you have any pointers that would be helpful :)

~Adrian

[spencerwuwu]

Ah yes, that's something I set up in purpose. If aflgo fails to locate the target function, it returns first instead of running various things and exit later.

It happens quite a lot in targeting different patches. But I think it should be able to locate all patches for libpng. Are there any other errors above in the log?

[adrianherrera]

Ah I think it was my fault. When running TARGET=libpng FUZZER=aflgo ./build.sh I forgot to also specify a bug. When I rerun with TARGET=libpng FUZZER=aflgo BUG=AAH001 ./build.sh it works fine.

Is the intention to have a separate target per bug?

[adrianherrera]

Okay so I got a bit further with sqlite3 by changing the compile command at the end of targets/sqlite3/build.sh to:

$CC $CFLAGS -I. \
    "$TARGET/repo/test/ossfuzz.c" \
    -o "$OUT/sqlite3_fuzz" \
    $LDFLAGS .libs/libsqlite3.a $LIBS -pthread -ldl -lz

I still get an Empty Ftargets.txt error, but at least it's building! Can you please incorporate this fix in your repo and I will check out openssl.

[adrianherrera]

Re. openssl: I had a quick look, and for some reason CFLAGS is being expanded twice, resulting in the error message about -outdir and targets. Expanded, the compile command that fails is:

/magma/fuzzers/aflgo/repo/afl-clang-fast  -I. -Iinclude -Iapps/include  -fPIC -pthread -m64 -fno-omit-frame-pointer -g -Wa,--noexecstack -Qunused-arguments -include /magma/magma/src/canary.h -DMAGMA_ENABLE_CANARIES    -g -O0 -targets=/magma_out/temp/BBtargets.txt -outdir=/magma_out/temp -flto -fuse-ld=gold -Wl,-plugin-opt=save-temps -include /magma/magma/src/canary.h -g -O0 -targets=/magma_out/temp/BBtargets.txt -outdir=/magma_out/temp -flto -fuse-ld=gold -fno-sanitize=alignment -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib/engines-3\"" -DMODULESDIR="\"/usr/local/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DPEDANTIC -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -DMAGMA_ENABLE_CANARIES -MMD -MF apps/lib/libapps-lib-app_params.d.tmp -MT apps/lib/libapps-lib-app_params.o -c -o apps/lib/libapps-lib-app_params.o apps/lib/app_params.c

However, I have no idea why CFLAGS is being expanded twice here 😕 At first I thought it might be colliding with CXXFLAGS, but this doesn't seem to be the issue. I don't have much time to look into it for the next few days, so if you find the reason for this please let me know!

[spencerwuwu]

Ah I think it was my fault. When running TARGET=libpng FUZZER=aflgo ./build.sh I forgot to also specify a bug. When I rerun with TARGET=libpng FUZZER=aflgo BUG=AAH001 ./build.sh it works fine.

Is the intention to have a separate target per bug?

Yes. It is necessary to build separately for each bug, at least for AFLGo. AFLGo compiles target program twice: first to generate CFGs and calculate distances with given locations, then it takes the results to compile the final instrumented binary. (This is one of the fews things I find AFLGo tricky to use: double compiling time plus time-consuming distance calculation)

Okay so I got a bit further with sqlite3 by changing the compile command at the end of targets/sqlite3/build.sh to:

$CC $CFLAGS -I. \
    "$TARGET/repo/test/ossfuzz.c" \
    -o "$OUT/sqlite3_fuzz" \
    $LDFLAGS .libs/libsqlite3.a $LIBS -pthread -ldl -lz

I still get an Empty Ftargets.txt error, but at least it's building! Can you please incorporate this fix in your repo and I will check out openssl.

Thanks! I'll update the scripts and try to run them.

For openssl I'll see if I can fix it and let you know.

Thanks again!

[scanakci]

Hi guys,

I have been also integrating AFLGO into MAGMA repo. For openssl, my quick fix is given as below:

#CONFIGURE_FLAGS=""
#if [[ $CFLAGS = *sanitize=memory* ]]; then
  CONFIGURE_FLAGS="no-asm"
#fi

# the config script supports env var LDLIBS instead of LIBS
export LDLIBS="$LIBS"

./config --debug enable-fuzz-libfuzzer enable-fuzz-afl disable-tests -DPEDANTIC \
    -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION no-shared no-module \
    enable-tls1_3 enable-rc5 enable-md2 enable-ec_nistp_64_gcc_128 enable-ssl3 \
    enable-ssl3-method enable-nextprotoneg enable-weak-ssl-ciphers \
    $CFLAGS -fno-sanitize=alignment $CONFIGURE_FLAGS

replacedCFLAG=$(grep -w "CFLAGS=" repo/Makefile | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("\n")}')
replacedCXXFLAG=$(grep -w "CXXFLAGS=" repo/Makefile | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("\n")}')
sed -i '/CFLAGS=-include/c\'"$replacedCFLAG"'' Makefile
sed -i '/CXXFLAGS=-include/c\'"$replacedCXXFLAG"'' Makefile

As you already noticed, some CCFLAGS are expanded twice and LLVM pops up error for AFLGO specific options (e.g., output-dir flag). I just removed duplication after generating the Makefile. Also, no-asm flag needs to be there.

So far, I could make libpng, libxml2, openssl, libtiff, sqlite3 work. Currently, I am working on PHP, and sometimes later I will give a shot to poppler.

"For sqlite3, somehow the last step (compiling ossfuzz.c) cannot link external symbols properly."

Yes, I also observed this. The problem is that the plugin-opt flag makes some external symbols internal for a reason and this results in a linker error. It is most probably originating from the clang version (it is very old). There is probably a bug (maybe still exists) in clang but I did not spend much time on it. For now, you can simply ignore the linker error since you do not have to create the final executable in the first step. Even if you do not link properly, you will still generate all necessary files to generate the necessary CFGs. I updated some scripts to make sure that docker does not stop after facing this error for sqlite3. (please check build.sh changes in https://github.com/scanakci/magma/commit/d6bea7f5b5f279332bf757a90ff58b79f967f994)

My fork is provided below (aflgo branch). libsqlite3 and libpng should work fine but it is better not to use it for now until I verify that I added all of my local changes and test in a clean environment. I should probably fix if any issue exists during this weekend.

https://github.com/scanakci/magma/tree/aflgo

[adrianherrera]

@scanakci cool! Do you also get the same problem where some bugs result in empty Ftargets files, or is this due to the approach taken by @spencerwuwu?

Also, does anyone know why CFLAGS get duplicated? Is this particular to the afl-clang-fast wrapper or something in the openssl build process?

[scanakci]

" Do you also get the same problem where some bugs result in empty Ftargets files, or is this due to the approach taken by @spencerwuwu?"

I did not get any problem for around 25 bugs that I tested so far. From libpng, I tested 5 bugs and they all went well.

"Also, does anyone know why CFLAGS get duplicated? Is this particular to the afl-clang-fast wrapper or something in the openssl build process?"

I do not know. I walk around in some openssl build scripts but it did not look like an easy task :).

[scanakci]

@adrianherrera Related to this, I get "/magma/targets/openssl/corpus/server/001c159fc891438181a8255e5587c48e25efb408: exit_code 137" for around 250 inputs from the corpus folder when using openssl library with aflgo. In your evaluation, did you use -m 100M for all fuzzers and for all programs (afl-based ones)? For libpng, libxml2, sqlite3, 100M was enough but I guess it is not for openssl. It may be related to aflgo, I will check with afl, moptafl soon.

[scanakci]

Php also worked fine on my end. I got a linker error in the first stage similar to sqlite3 but it did not cause any problem when generating CFGs in the next step. I tested openssl with afl and moptafl with 100M and got exit_code 137 (i.e. out of memory) error. I believe memory is not enough in aflgo case because of the additional instrumentation in the code. I will increase 100M and see if there is any change.

poppler is the last benchmark that needs to be fixed. I got some compilation errors that I can spend time fixing the following week.

[adrianherrera]

@scanakci yeah for our evaluation we used the configurations provided in the v1.1 branch. Try to increase the memory limit and let me know what happens.

[hazimeh]

https://twitter.com/mboehme_/status/1372884402020122627

It seems an updated version of AFLGo has been released, and it's a lot faster

[scanakci]

That is great. It is also perfect that they switched to LLVM 11. All libraries worked fine with AFLGo with the previous version. To make poppler work, I needed to update build.sh

https://github.com/scanakci/magma/commit/6f02e3a9c99034d49da4b8517dd5697e43c9ac43

I used -M 500M for openssl, but it was not enough.

Unfortunately, I do not have time to test recent AFLGo in near future but I can probably test after finalizing some other works on my end.

[guestzz]

@spencerwuwu Hi! I'm currently using your work to test AFLGo, but I've encountered some issues. I've deployed your project directly, with only modifications to the captainrc and the Docker user ID. I've tested AAH001-AAH007 on libpng. Everything related to compilation, instrumentation, and building went smoothly. And I correctly generated the distance files. However, during the fuzzing process, there are no crashes produced. After testing libpng, I found that the final libpng_read_fuzzer binary is around 800KB in size, and no matter what input data I provide, there is no output or crash; it exits immediately. I think this issue may be related to the last compilation process:

$CXX $CXXFLAGS -std=c++11 -I. \
     contrib/oss-fuzz/libpng_read_fuzzer.cc \
     -o libpng_read_fuzzer \
     $LDFLAGS .libs/libpng16.a $LIBS -lz

Could you please provide some assistance? Thank you.

Edit: Finished. Thanks.

[ravenolf]

Hello! I am also working on integrating AFLGo into magma and fuzzing some of the bugs. I am having an issue with the empty Ftargets.txt on sqlite3 similar to what @adrianherrera and @spencerwuwu have mentioned. I tested with the changes suggested in this issue but I am still getting an empty Ftargets file. Could you please provide the solution you used for it?