search

HexHive / magma

A ground-truth fuzzing benchmark suite based on real programs with real bugs.
https://hexhive.epfl.ch/magma
286 stars 83 forks source link

Magma build failure (and other issues) #24

Closed cryptomadco closed 3 years ago

cryptomadco commented 4 years ago

Hello .

I tried many time to setup magma .

I want mainly :

  1. Setup magma to run campaigns but unfortunately seems scripts are not working properly, see this log :
crypto@crypto-Standard-PC-i440FX-PIIX-1996:~/magma/tools/captain$ cat workdir/log/afl_libpng_build.log 
++ id -u root
++ id -g root
+ docker build -t magma/afl/libpng --build-arg fuzzer_name=afl --build-arg target_name=libpng --build-arg USER_ID=0 --build-arg GROUP_ID=0 --build-arg canaries=1 --build-arg isan=1 -f /home/crypto/magma/docker/Dockerfile /home/crypto/magma
Sending build context to Docker daemon  147.2MB
Step 1/59 : FROM ubuntu:18.04
 ---> 6526a1858e5d
Step 2/59 : RUN apt-get update && apt-get install -y sudo
 ---> Using cache
 ---> 5dc954e1e6e1
Step 3/59 : ARG magma_root=./
 ---> Using cache
 ---> 27be792df8c5
Step 4/59 : ENV MAGMA_R /magma
 ---> Using cache
 ---> 94e87f0117e6
Step 5/59 : ENV OUT             /magma_out
 ---> Using cache
 ---> 4c3fafcb1c5a
Step 6/59 : ENV SHARED  /magma_shared
 ---> Using cache
 ---> 17a7c6c4c785
Step 7/59 : ENV CC  /usr/bin/gcc
 ---> Using cache
 ---> f9804c69b838
Step 8/59 : ENV CXX /usr/bin/g++
 ---> Using cache
 ---> 28da6f77ed91
Step 9/59 : ENV LD /usr/bin/ld
 ---> Using cache
 ---> 3f7266ea28d7
Step 10/59 : ENV AR /usr/bin/ar
 ---> Using cache
 ---> 242350988509
Step 11/59 : ENV AS /usr/bin/as
 ---> Using cache
 ---> 55e88f84dcd4
Step 12/59 : ENV NM /usr/bin/nm
 ---> Using cache
 ---> add1d13f6a47
Step 13/59 : ENV RANLIB /usr/bin/ranlib
 ---> Using cache
 ---> d27110b1dc9b
Step 14/59 : ARG USER_ID=1000
 ---> Using cache
 ---> dacf05462bcb
Step 15/59 : ARG GROUP_ID=1000
 ---> Using cache
 ---> e2e8c6738607
Step 16/59 : RUN mkdir -p /home &&      groupadd -g ${GROUP_ID} magma &&        useradd -l -u ${USER_ID} -K UMASK=0000 -d /home -g magma magma &&      chown magma:magma /home
 ---> Running in c87a6f122e87
groupadd: GID '0' already exists
The command '/bin/sh -c mkdir -p /home &&       groupadd -g ${GROUP_ID} magma &&        useradd -l -u ${USER_ID} -K UMASK=0000 -d /home -g magma magma &&      chown magma:magma /home' returned a non-zero code: 4

This is the log after running ./run.sh in the captain dir and I don't know why it's not running .

  1. I want to build a vulnerable target (say libpng) with my own fuzzer and have reports as in the magma documentation, how is this possible ? (As easy as changing scripts or the fuzzer should be built into a docker ?)

  2. While setting up symcc_afl I got the following error :

[7/12] Building CXX object CMakeFiles/Symbolize.dir/compiler/Main.cpp.o
FAILED: /usr/bin/c++   -DSymbolize_EXPORTS -isystem /usr/lib/llvm-11/include -DNDEBUG  -std=c++17 -Wredundant-decls -Wcast-align -Wmissing-include-dirs -Wswitch-default -Wextra -Wall -Winvalid-pch -Wredundant-decls -Wformat=2 -Wmissing-format-attribute -Wformat-nonliteral -Werror -fPIC   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -MMD -MT CMakeFiles/Symbolize.dir/compiler/Main.cpp.o -MF CMakeFiles/Symbolize.dir/compiler/Main.cpp.o.d -o CMakeFiles/Symbolize.dir/compiler/Main.cpp.o -c ../compiler/Main.cpp
In file included from /usr/include/llvm-11/llvm/PassSupport.h:27:0,
                 from /usr/include/llvm-11/llvm/Pass.h:318,
                 from /usr/include/llvm-11/llvm/IR/LegacyPassManager.h:19,
                 from ../compiler/Main.cpp:15:
/usr/include/llvm-11/llvm/ADT/StringRef.h:22:23: fatal error: string_view: No such file or directory
compilation terminated.
[7/12] Performing configure step for 'SymRuntime'
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler using: Ninja
-- Check for working CXX compiler using: Ninja -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/crypto/magma/fuzzers/symcc_afl/symcc/build/SymRuntime-prefix/src/SymRuntime-build
ninja: build stopped: subcommand failed.

How to solve and build symcc_afl ?

Thanks!

hazimeh commented 4 years ago

Please check the Troubleshooting section on the Magma website as a first step when facing problems.

  1. It seems you've run the ./run.sh command with the root user (possibly via sudo?). Try instead running it with a non-root user. Make sure that user is part of the docker group. Refer to the relevant troubleshooting item on this topic.
  2. Add a fuzzer configuration under the fuzzers directory. As a start, clone the fuzzers/afl configuration to fuzzers/myfuzzer, and modify the included scripts to fetch and build your fuzzer and instrument the target correctly. Refer to the Technical Reference section for details about expected files and environment variables. The report generation tools are still being merged and polished, but you could check the reports branch for scripts inside the tools/benchd directory. These allow you to generate a JSON file summary of the campaigns. Usage: 
    
pip3 install pandas

mkdir out_dir python3 loggen.py --workers 16 /path/to/fuzz_workdir ./out_dir python3 logparse.py ./out_dir/log --out-format json --out-file ./data.json


3. Based on (1), I'm assuming you're building it outside the docker container. It also seems that the compiler installed on your system is GCC 5.4.0. GCC versions before 7 have only experimental support for C++17 (which includes `std::string_view`). Try upgrading your system's compiler, or point the script to a more modern compiler through the `CC` and `CXX` flags. Ideally, these scripts should only be run inside the docker container.
hazimeh commented 4 years ago

Update: I've gone ahead and updated the tools/benchd toolset (currently just one script), and pushed it to the master branch.

Check out the documentation for requirements, usage, and sample output.

cryptomadco commented 4 years ago

Hello @hazimeh !

Thanks for your reply .

After a lot of investigation I came to three commands which basically solved my problem on docker permission things .

so, I highly recommend you to add these commands to top of this getting started page, so no-one will get disappointed from this permission problem stuff :

sudo usermod -aG docker ${USER}
su -s ${USER}
sudo chmod 666 /var/run/docker.sock

If you can add these commands to : https://hexhive.epfl.ch/magma/docs/getting-started.html

I'll appreciate you!

I mostly interested in running this interesting bench tool as a local tool so if you can also make just one bash script to install all dependencies need for all these toolset, it's much appreciated, running too much scripts one by one is a bit cumbersome .

Also, if you can provide all those vulnerable targets as a corpus for those of people who are interested to run them against their fuzzers out of MAGMA toolset (exactly Like LAVA) it would be nice!

I'll let you know the results of running a test with magma and if that was going fine, I'll close this issue.

Thanks again

cryptomadco commented 4 years ago

From the output I got from Magma, it seems it runs well on afl and aflplusplus but failed at running and building symcc .

For all of these stuff, I used captain because it was easier to run .

Does using captain mean that the tests will be done at the local machine or it just mean in an unmanaged manner ?

Also I would like to know, how is it possible to generate beautiful reports from MAGMA like this for campaigns ?

https://hexhive.epfl.ch/magma/reports/sample/

Thanks

hazimeh commented 4 years ago

I mostly interested in running this interesting bench tool as a local tool so if you can also make just one bash script to install all dependencies need for all these toolset, it's much appreciated, running too much scripts one by one is a bit cumbersome .

Also, if you can provide all those vulnerable targets as a corpus for those of people who are interested to run them against their fuzzers out of MAGMA toolset (exactly Like LAVA) it would be nice!

Will do. Thanks for the feedback.

From the output I got from Magma, it seems it runs well on afl and aflplusplus but failed at running and building symcc .

For all of these stuff, I used captain because it was easier to run .

Could you attach the full build log for the symcc_afl image?

Does using captain mean that the tests will be done at the local machine or it just mean in an unmanaged manner ?

The captain toolset automates the process of building, running, and scheduling Magma Docker images and containers. The experiments run on the local machine, but within a containerized process. This could induce some syscall overhead, but all fuzzers are evaluated within the same environment (a docker container), so they're all subject to the same overhead. syscall-heavy fuzzers are technically at a disadvantage, but that's not due to the containerization; it's just because they're syscall-heavy.

Also I would like to know, how is it possible to generate beautiful reports from MAGMA like this for campaigns ?

I'm still in the process of reviewing PR #22. Currently, the generated reports are tightly coupled to the CSS stylesheets on Magma's website. I'll need to look into a more suitable way to publish these reports.

cryptomadco commented 4 years ago

Thanks for your explanations .

Could you attach the full build log for the symcc_afl image? Yes, sure, asap.

I'm still in the process of reviewing PR #22. Currently, the generated reports are tightly coupled to the CSS stylesheets on Magma's website. I'll need to look into a more suitable way to publish these reports.

I don't know when you're going to make change and make working scripts for report generation but I used the reports branch in the current magma and after successful running of afl and aflplus against libpng for report generation I have the following problems :


crypto@fuzzer3:~/magma/tools/report/WebPages$ python3 main.py /home/crypto/magma/tools/benchd/myfile.json 
Load json
Create useful directories
Generate plots
Traceback (most recent call last):
  File "main.py", line 78, in <module>
    main()
  File "main.py", line 46, in main
    plots.generate()
  File "/home/crypto/magma/tools/report/WebPages/plotGenerator.py", line 36, in generate
    self.line_plot_unique_bugs(self.REACHED)
  File "/home/crypto/magma/tools/report/WebPages/plotGenerator.py", line 817, in line_plot_unique_bugs
    campaign_data = self.get_minimum_bugs(library, metric)
  File "/home/crypto/magma/tools/report/WebPages/plotGenerator.py", line 689, in get_minimum_bugs
    bugs = self.get_fuzzer_lib_bugs(fuzzer, library)
  File "/home/crypto/magma/tools/report/WebPages/plotGenerator.py", line 665, in get_fuzzer_lib_bugs
    for p_data in self.data[fuzzer][library].values():
KeyError: 'libpng'

The json generated files with benchd scripts is as :

{"logs": {"libpng": {"libpng_read_fuzzer": {"1": {"reached": {"AAH005": 15, "AAH007": 15, "AAH003": 10, "AAH001": 15, "AAH004": 15, "AAH008": 15}, "triggered": {"AAH003": 15, "AAH008": 515}}, "2": {"reached": {"AAH005": 15, "AAH007": 15, "AAH003": 10, "AAH001": 15, "AAH004": 15, "AAH008": 15}, "triggered": {"AAH003": 15}}, "0": {"reached": {"AAH005": 15, "AAH007": 15, "AAH003": 10, "AAH001": 15, "AAH004": 15, "AAH008": 15}, "triggered": {"AAH003": 15}}}}, "libtiff": {"tiffcp": {"1": {"reached": {"AAH011": 10, "AAH020": 5, "AAH015": 415, "AAH022": 415}, "triggered": {}}, "2": {"reached": {"AAH011": 10, "AAH020": 5, "AAH015": 305, "AAH022": 305}, "triggered": {}}, "0": {"reached": {"AAH011": 10, "AAH020": 5, "AAH015": 405, "AAH022": 405}, "triggered": {}}}, "tiff_read_rgba_fuzzer": {"1": {"reached": {"AAH011": 15, "AAH020": 10, "AAH015": 35, "AAH022": 35}, "triggered": {}}, "2": {"reached": {"AAH011": 15, "AAH020": 10, "AAH015": 35, "AAH022": 35}, "triggered": {"AAH015": 590}}, "0": {"reached": {"AAH011": 15, "AAH020": 10, "AAH015": 35, "AAH022": 35}, "triggered": {"AAH015": 275}}}}}, "tmp": {}}

Is it mean that the scripts are not working as expected or there are other problems around ?

Thanks!

hazimeh commented 4 years ago

Have you tried using the new "exp2json.py" script instead? Just point it at the /path/to/workdir and it should generate a valid JSON summary.

I think the issue with the attached JSON file is that the "logparse.py" script may have been pointed at the wrong root dir.

Try the "exp2json.py" script and let me know how it goes.

cryptomadco commented 4 years ago

Using the "exp2json.py", I have the following erros : (using branch v1.0.4)

crypto@magma-1:~/magma/tools/benchd$ python3 exp2json.py  /home/crypto/magma/tools/benchd/workdir2 ./outfile.json
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/x509/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/client/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/asn1/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/server/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/asn1parse/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/client/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/server/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/x509/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/asn1parse/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/server/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/x509/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/asn1parse/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/asn1/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/bignum/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/bignum/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/asn1/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/client/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdfimages/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/openssl/bignum/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdf_fuzzer/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdfimages/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdftoppm/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libtiff/tiff_read_rgba_fuzzer/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdf_fuzzer/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdfimages/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdftoppm/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libtiff/tiff_read_rgba_fuzzer/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdf_fuzzer/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libtiff/tiffcp/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libtiff/tiffcp/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/poppler/pdftoppm/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libtiff/tiffcp/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libtiff/tiff_read_rgba_fuzzer/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libpng/libpng_read_fuzzer/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libxml2/libxml2_xml_read_memory_fuzzer/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libxml2/xmllint/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/sqlite3/sqlite3_fuzz/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libpng/libpng_read_fuzzer/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/json/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libpng/libpng_read_fuzzer/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libxml2/xmllint/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libxml2/libxml2_xml_read_memory_fuzzer/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/json/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/unserialize/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/json/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/sqlite3/sqlite3_fuzz/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/unserialize/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libxml2/libxml2_xml_read_memory_fuzzer/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/exif/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/sqlite3/sqlite3_fuzz/1
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/unserialize/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/exif/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/exif/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/libxml2/xmllint/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/parser/2
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/parser/0
Processing /home/crypto/magma/tools/benchd/workdir2/ar/aflplusplus/php/parser/1
Traceback (most recent call last):
  File "exp2json.py", line 193, in <module>
    main()
  File "exp2json.py", line 183, in main
    summary = get_experiment_summary(experiment)
  File "exp2json.py", line 173, in get_experiment_summary
    reached, triggered = get_ttb_from_df(df)
  File "exp2json.py", line 152, in get_ttb_from_df
    bugs = set(x[:-2] for x in df.columns)
AttributeError: 'NoneType' object has no attribute 'columns'

I think a complete documentation is also necessary for that report and bench tools stuff .

cryptomadco commented 4 years ago

Another test with the latest branch :

crypto@magma2:~/magma/tools/benchd$ python3 exp2json.py ./workdir/ ./outfule
Processing ./workdir/ar/aflplusplus_lto/sqlite3/sqlite3_fuzz/1
Processing ./workdir/ar/aflplusplus_lto/sqlite3/sqlite3_fuzz/2
Processing ./workdir/ar/aflplusplus_lto/libpng/libpng_read_fuzzer/0
Processing ./workdir/ar/aflplusplus_lto/libtiff/tiff_read_rgba_fuzzer/1
Processing ./workdir/ar/aflplusplus_lto/libtiff/tiff_read_rgba_fuzzer/0
Processing ./workdir/ar/aflplusplus_lto/libpng/libpng_read_fuzzer/2
Processing ./workdir/ar/aflplusplus_lto/sqlite3/sqlite3_fuzz/0
Processing ./workdir/ar/aflplusplus_lto/libpng/libpng_read_fuzzer/1
Processing ./workdir/ar/aflplusplus_lto/libtiff/tiff_read_rgba_fuzzer/2
Processing ./workdir/ar/aflplusplus_lto/libtiff/tiffcp/0
Processing ./workdir/ar/symcc_afl/sqlite3/sqlite3_fuzz/1
Processing ./workdir/ar/aflplusplus_lto/libtiff/tiffcp/1
Processing ./workdir/ar/symcc_afl/sqlite3/sqlite3_fuzz/2
Processing ./workdir/ar/aflplusplus_lto/libtiff/tiffcp/2
Processing ./workdir/ar/symcc_afl/sqlite3/sqlite3_fuzz/0
Processing ./workdir/ar/symcc_afl/libpng/libpng_read_fuzzer/0
Processing ./workdir/ar/symcc_afl/libtiff/tiff_read_rgba_fuzzer/1
Processing ./workdir/ar/symcc_afl/libpng/libpng_read_fuzzer/1
Processing ./workdir/ar/symcc_afl/libtiff/tiff_read_rgba_fuzzer/0
Processing ./workdir/ar/symcc_afl/libtiff/tiff_read_rgba_fuzzer/2
Processing ./workdir/ar/symcc_afl/libpng/libpng_read_fuzzer/2
Processing ./workdir/ar/symcc_afl/libtiff/tiffcp/0
Processing ./workdir/ar/symcc_afl/libtiff/tiffcp/1
Processing ./workdir/ar/symcc_afl/libtiff/tiffcp/2
Traceback (most recent call last):
  File "exp2json.py", line 193, in <module>
    main()
  File "exp2json.py", line 183, in main
    summary = get_experiment_summary(experiment)
  File "exp2json.py", line 173, in get_experiment_summary
    reached, triggered = get_ttb_from_df(df)
  File "exp2json.py", line 152, in get_ttb_from_df
    bugs = set(x[:-2] for x in df.columns)
AttributeError: 'NoneType' object has no attribute 'columns'

And about symcc issue to install locally, I installed a new clean ubuntu 18 vm and after using fetch.sh and build.sh scripts things are as : (Have nothing inside, just used magma scripts to install everything but seems not working) 

crypto@magma2:~/magma/fuzzers/symcc_afl/symcc/build$ ./symcc
/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
cryptomadco commented 4 years ago

@hazimeh Any recommendations / possible fixes / solutions so far ?

hazimeh commented 4 years ago

It's possibly an edge case where the monitor log is empty (premature campaign). I'll address it when I get the time.

hazimeh commented 4 years ago

@cryptomadco Could you please archive the workdir and attach it here? I'd like to reproduce the error.

cryptomadco commented 4 years ago

@hazimeh Here is the workdir :

workdir.zip

If you get a working report of this , please let me know how to setup for that report .

Thank you

hazimeh commented 4 years ago

Alright, it seems the error was due to an empty monitor log for symcc_afl/sqlite3/sqlite3_fuzz/{0,1,2}. This could be due to the very short campaign lengths (1 minute). I've modified the script to gracefully ignore these errors (but still output error messages). Try with longer campaigns and let me know if the issue with symcc_afl/sqlite3 persists.

cryptomadco commented 4 years ago

Alright, it seems the error was due to an empty monitor log for symcc_afl/sqlite3/sqlite3_fuzz/{0,1,2}. This could be due to the very short campaign lengths (1 minute). I've modified the script to gracefully ignore these errors (but still output error messages). Try with longer campaigns and let me know if the issue with symcc_afl/sqlite3 persists.

@hazimeh In that test I just didn't want to run test against sqlite3, why the problem is from that ? I excluded sqlite3 from captainrc file !

hazimeh commented 4 years ago

Could you also attach the captainrc file you used?

cryptomadco commented 4 years ago

Yes, this was my captainrc :


# This file contains the configuration for the run.sh script. It follows the
# Bash syntax and is sourced by the script to access the variables. Variables
# are mandatory unless marked with [brackets].

###
## Configuration parameters
###

# WORKDIR: path to directory where shared volumes will be created
WORKDIR=./workdir

# REPEAT: number of campaigns to run per program (per fuzzer)
REPEAT=2

# [WORKER_MODE]: defines the type of CPU resources to allocate (default: 1)
# - 1: logical cores (possibly SMT-enabled)
# - 2: physical cores
# - 3: physical sockets (1 worker per CPU socket)
# WORKER_MODE=1

# [WORKERS]: number of worker threads (default: all cores)
WORKERS=3

# [WORKER_POOL]: a space-separated list of logical cores to allocate
# WORKER_POOL="1 3 5 7 9"

# [CAMPAIGN_WORKERS]: number of workers to allocate for a campaign (default: 1)
# CAMPAIGN_WORKERS=1

# [TIMEOUT]: time to run each campaign. This variable supports one-letter
# suffixes to indicate duration (s: seconds, m: minutes, h: hours, d: days)
# (default: 1m)
TIMEOUT=15m

# [POLL]: time (in seconds) between polls (default: 5)
POLL=5

# [CACHE_ON_DISK]: if set, the cache workdir is mounted on disk instead of
# in-memory (default: unset)
# CACHE_ON_DISK=1

# [NO_ARCHIVE]: if set, campaign workdirs will not be tarballed (default: unset)
# NO_ARCHIVE=1

# [TMPFS_SIZE]: the size of the tmpfs mounted volume. This only applies when
# CACHE_ON_DISK is not set (default: 50g)
# TMPFS_SIZE=16g

# [MAGMA]: path to magma root (default: ../../)
# MAGMA=/path/to/magma/

# [CANARY_MODE]: defines the mode of canaries at compile time (default: 1)
# - 1: without fixes, with canaries
# - 2: without fixes, without canaries
# - 3: with fixes, without canaries
# CANARY_MODE=3

# [ISAN]: if set, build the benchmark with ISAN/fatal canaries (default: unset)
# ISAN=1

# [HARDEN]: if set, build the benchmark with hardened canaries (default: unset)
# HARDEN=1

# [POC_EXTRACT]: if set, run the extract.sh script after the campaign is done
# (default: unset)
# POC_EXTRACT=1

###
## Campaigns to run
###

# FUZZERS: an array of fuzzer names (from magma/fuzzers/*) to evaluate
FUZZERS=(aflplusplus_lto symcc_afl)

# [fuzzer_TARGETS]: an array of target names (from magma/targets/*) to fuzz with
# `fuzzer` (default: all targets)
afl_TARGETS=(libpng libtiff libxml2)

# [fuzzer_target_PROGRAMS]: an array of program names (from
# magma/targets/target/configrc) to use as execution drivers when fuzzing the
# `target`
# afl_libtiff_PROGRAMS=(tiffcp)

# [fuzzer_CAMPAIGN_WORKERS]: overrides the global CAMPAIGN_WORKERS setting
# afl_CAMPAIGN_WORKERS=3
hazimeh commented 4 years ago

The TARGETS configuration parameter requires a fuzzer prefix which matches the fuzzers being evaluated. In your case, you would need to specify them as follows:

aflplusplus_lto_TARGETS=(libpng libtiff libxml2)
symcc_afl_TARGETS=(libpng libtiff libxml2)
cryptomadco commented 4 years ago

aflplusplus_lto_TARGETS=(libpng libtiff libxml2) symcc_afl_TARGETS=(libpng libtiff libxml2)

Thanks a lot, but I didn't find that keywords in the documentation, I just thought all will be done with afl_TARGETS . just think that it's better to add one more keyword for this in captainrc file .

Now going to run that .

@hazimeh Do you possibly know approximately, how much does it takes for you to check and fix that web report tool to a working version ?

Thanks!

hazimeh commented 4 years ago

I am currently in the process of merging it. It will take a while, however, as I need to proofread the code and make sure it works correctly. I'll ping you when it's done.

cryptomadco commented 4 years ago

Thanks about that, further discussion about that report tool also can be take through mail, some days ago I pinged you over you@epf.ch , just don't know if you receive that and get time to read that .

Thanks!

cryptomadco commented 4 years ago

Now, seems the exp2json.py script is working well 👍

There is just one problem reaming that is related to that report tool :

python3 main.py myoutfilex.json 
Load json
Create useful directories
Generate plots
Traceback (most recent call last):
  File "main.py", line 78, in <module>
    main()
  File "main.py", line 46, in main
    plots.generate()
  File "/home/crypto/reports/tools/report/WebPages/plotGenerator.py", line 36, in generate
    self.line_plot_unique_bugs(self.REACHED)
  File "/home/crypto/reports/tools/report/WebPages/plotGenerator.py", line 817, in line_plot_unique_bugs
    campaign_data = self.get_minimum_bugs(library, metric)
  File "/home/crypto/reports/tools/report/WebPages/plotGenerator.py", line 689, in get_minimum_bugs
    bugs = self.get_fuzzer_lib_bugs(fuzzer, library)
  File "/home/crypto/reports/tools/report/WebPages/plotGenerator.py", line 665, in get_fuzzer_lib_bugs
    for p_data in self.data[fuzzer][library].values():
KeyError: 'libtiff'

As I tested Magma in recent days, I think magma with a working version that web report tool, is a cool fuzzer evaluation tool!

hazimeh commented 4 years ago

@cryptomadco I have merged and partially tested the report-generation code. In its current form, it is only able to generate plots:

pip3 install --user pandas lifelines scipy seaborn scikit_posthocs
cd /path/to/magma/tools/report_df
mkdir -p 'output/data'
python3 main.py /path/to/data.json

I hope I am not missing any requirements.

Edit: It's on the dev branch now.

cryptomadco commented 4 years ago

@hazimeh

Too much appreciate Ahmad! the plot generation is working well, tested it with different scenarios, works like a charm! thank you very much !

I think that the Symcc_afl had some modifications so I have problems after running the campaign with ./run.sh , it's the log of symcc_afl_sqlite3_build.log failure :

[21/21] Linking CXX shared library libSymRuntime.so
qsym_backend/CMakeFiles/SymRuntime.dir/Runtime.cpp.o: In function `_sym_initialize':
Runtime.cpp:(.text+0x459): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
[12/14] No install step for 'SymRuntime'
[13/14] Completed 'SymRuntime'
[13/14] Testing the system...
-- Testing: 16 tests, 16 workers --
PASS: compiler :: regression/cxa_vector.ll (1 of 16)
FAIL: compiler :: switch.c (2 of 16)
******************** TEST 'compiler :: switch.c' FAILED ********************
Script:
--
: 'RUN: at line 15';   /magma/fuzzers/symcc_afl/symcc/build/test/../symcc -O2 /magma/fuzzers/symcc_afl/symcc/test/switch.c -o /magma/fuzzers/symcc_afl/symcc/build/test/Output/switch.c.tmp
: 'RUN: at line 16';   echo -ne "\x00\x00\x00\x05" | /magma/fuzzers/symcc_afl/symcc/build/test/Output/switch.c.tmp 2>&1 | FileCheck --check-prefix=QSYM --check-prefix=ANY /magma/fuzzers/symcc_afl/symcc/test/switch.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 15"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/../symcc" "-O2" "/magma/fuzzers/symcc_afl/symcc/test/switch.c" "-o" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/switch.c.tmp"
# command stderr:
Warning: losing track of symbolic expressions at inline assembly   %23 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %17) #4, !srcloc !8

$ ":" "RUN: at line 16"
$ "echo" "-ne" "\x00\x00\x00\x05"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/Output/switch.c.tmp"
$ "FileCheck" "--check-prefix=QSYM" "--check-prefix=ANY" "/magma/fuzzers/symcc_afl/symcc/test/switch.c"
# command stderr:
/magma/fuzzers/symcc_afl/symcc/test/switch.c:56:19: error: QSYM-COUNT: expected string not found in input (1 out of 2)
 // QSYM-COUNT-2: SMT
                  ^
<stdin>:1:1: note: scanning from here
This is SymCC running with the QSYM backend
^
<stdin>:1:9: note: possible intended match here
This is SymCC running with the QSYM backend
        ^

error: command failed with exit status: 1

--

********************
FAIL: compiler :: loop.c (3 of 16)
******************** TEST 'compiler :: loop.c' FAILED ********************
Script:
--
: 'RUN: at line 15';   /magma/fuzzers/symcc_afl/symcc/build/test/../symcc -O2 /magma/fuzzers/symcc_afl/symcc/test/loop.c -o /magma/fuzzers/symcc_afl/symcc/build/test/Output/loop.c.tmp
: 'RUN: at line 16';   echo -ne "\x00\x00\x00\x05" | /magma/fuzzers/symcc_afl/symcc/build/test/Output/loop.c.tmp 2>&1 | FileCheck --check-prefix=QSYM --check-prefix=ANY /magma/fuzzers/symcc_afl/symcc/test/loop.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 15"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/../symcc" "-O2" "/magma/fuzzers/symcc_afl/symcc/test/loop.c" "-o" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/loop.c.tmp"
# command stderr:
Warning: losing track of symbolic expressions at inline assembly   %22 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %16) #5, !srcloc !8

$ ":" "RUN: at line 16"
$ "echo" "-ne" "\x00\x00\x00\x05"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/Output/loop.c.tmp"
$ "FileCheck" "--check-prefix=QSYM" "--check-prefix=ANY" "/magma/fuzzers/symcc_afl/symcc/test/loop.c"
# command stderr:
/magma/fuzzers/symcc_afl/symcc/test/loop.c:38:19: error: QSYM-COUNT: expected string not found in input (1 out of 5)
 // QSYM-COUNT-5: New testcase
                  ^
<stdin>:1:1: note: scanning from here
This is SymCC running with the QSYM backend
^
<stdin>:1:21: note: possible intended match here
This is SymCC running with the QSYM backend
                    ^

error: command failed with exit status: 1

--

********************
FAIL: compiler :: pointers.c (4 of 16)
******************** TEST 'compiler :: pointers.c' FAILED ********************
Script:
--
: 'RUN: at line 15';   /magma/fuzzers/symcc_afl/symcc/build/test/../symcc -O2 /magma/fuzzers/symcc_afl/symcc/test/pointers.c -o /magma/fuzzers/symcc_afl/symcc/build/test/Output/pointers.c.tmp
: 'RUN: at line 16';   echo -ne "\x00\x00\x00\x05\x12\x34\x56\x78\x90\xab\xcd\xef" | /magma/fuzzers/symcc_afl/symcc/build/test/Output/pointers.c.tmp 2>&1 | FileCheck --check-prefix=QSYM --check-prefix=ANY /magma/fuzzers/symcc_afl/symcc/test/pointers.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 15"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/../symcc" "-O2" "/magma/fuzzers/symcc_afl/symcc/test/pointers.c" "-o" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/pointers.c.tmp"
# command stderr:
Warning: losing track of symbolic expressions at inline assembly   %25 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %19) #4, !srcloc !8

$ ":" "RUN: at line 16"
$ "echo" "-ne" "\x00\x00\x00\x05\x12\x34\x56\x78\x90\xab\xcd\xef"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/Output/pointers.c.tmp"
$ "FileCheck" "--check-prefix=QSYM" "--check-prefix=ANY" "/magma/fuzzers/symcc_afl/symcc/test/pointers.c"
# command stderr:
/magma/fuzzers/symcc_afl/symcc/test/pointers.c:51:10: error: ANY: expected string not found in input
 // ANY: different
         ^
<stdin>:8:20: note: scanning from here
[INFO] New testcase: /magma/fuzzers/symcc_afl/symcc/build/test/SymccOutput/000000
                   ^
<stdin>:8:54: note: possible intended match here
[INFO] New testcase: /magma/fuzzers/symcc_afl/symcc/build/test/SymccOutput/000000
                                                     ^

error: command failed with exit status: 1

--

********************
PASS: compiler :: large_alloc.c (5 of 16)
PASS: compiler :: strings.c (6 of 16)
PASS: compiler :: read.c (7 of 16)
PASS: compiler :: integers.c (8 of 16)
PASS: compiler :: floats.c (9 of 16)
FAIL: compiler :: structs.c (10 of 16)
******************** TEST 'compiler :: structs.c' FAILED ********************
Script:
--
: 'RUN: at line 15';   /magma/fuzzers/symcc_afl/symcc/build/test/../symcc -O2 /magma/fuzzers/symcc_afl/symcc/test/structs.c -o /magma/fuzzers/symcc_afl/symcc/build/test/Output/structs.c.tmp
: 'RUN: at line 16';   echo -ne "\x00\x00\x00\x05" | /magma/fuzzers/symcc_afl/symcc/build/test/Output/structs.c.tmp 2>&1 | FileCheck --check-prefix=QSYM --check-prefix=ANY /magma/fuzzers/symcc_afl/symcc/test/structs.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 15"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/../symcc" "-O2" "/magma/fuzzers/symcc_afl/symcc/test/structs.c" "-o" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/structs.c.tmp"
# command stderr:
Warning: losing track of symbolic expressions at inline assembly   %22 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %16) #4, !srcloc !8

$ ":" "RUN: at line 16"
$ "echo" "-ne" "\x00\x00\x00\x05"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/Output/structs.c.tmp"
$ "FileCheck" "--check-prefix=QSYM" "--check-prefix=ANY" "/magma/fuzzers/symcc_afl/symcc/test/structs.c"
# command stderr:
/magma/fuzzers/symcc_afl/symcc/test/structs.c:50:19: error: QSYM-COUNT: expected string not found in input (1 out of 2) // QSYM-COUNT-2: SMT
                  ^
<stdin>:1:1: note: scanning from here
This is SymCC running with the QSYM backend
^
<stdin>:1:9: note: possible intended match here
This is SymCC running with the QSYM backend
        ^

error: command failed with exit status: 1

--

********************
PASS: compiler :: arrays.c (11 of 16)
FAIL: compiler :: memcpy.c (12 of 16)
******************** TEST 'compiler :: memcpy.c' FAILED ********************
Script:
--
: 'RUN: at line 15';   /magma/fuzzers/symcc_afl/symcc/build/test/../symcc -O2 /magma/fuzzers/symcc_afl/symcc/test/memcpy.c -o /magma/fuzzers/symcc_afl/symcc/build/test/Output/memcpy.c.tmp
: 'RUN: at line 16';   echo -ne "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03" | /magma/fuzzers/symcc_afl/symcc/build/test/Output/memcpy.c.tmp 2>&1 | FileCheck --check-prefix=QSYM --check-prefix=ANY /magma/fuzzers/symcc_afl/symcc/test/memcpy.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 15"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/../symcc" "-O2" "/magma/fuzzers/symcc_afl/symcc/test/memcpy.c" "-o" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/memcpy.c.tmp"
# command stderr:
Warning: losing track of symbolic expressions at inline assembly   %26 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %20) #4, !srcloc !8
Warning: losing track of symbolic expressions at inline assembly   %49 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %43) #4, !srcloc !9
Warning: losing track of symbolic expressions at inline assembly   %72 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %66) #4, !srcloc !10

$ ":" "RUN: at line 16"
$ "echo" "-ne" "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/Output/memcpy.c.tmp"
$ "FileCheck" "--check-prefix=QSYM" "--check-prefix=ANY" "/magma/fuzzers/symcc_afl/symcc/test/memcpy.c"
# command stderr:
/magma/fuzzers/symcc_afl/symcc/test/memcpy.c:60:19: error: QSYM-COUNT: expected string not found in input (1 out of 2)
 // QSYM-COUNT-2: SMT
                  ^
<stdin>:1:1: note: scanning from here
This is SymCC running with the QSYM backend
^
<stdin>:1:9: note: possible intended match here
This is SymCC running with the QSYM backend
        ^

error: command failed with exit status: 1

--

********************
PASS: compiler :: if.c (13 of 16)
FAIL: compiler :: file_input.c (14 of 16)
******************** TEST 'compiler :: file_input.c' FAILED ********************
Script:
--
: 'RUN: at line 15';   /bin/echo -ne "\x00\x00\x00\x05aaaa" > /magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.input
: 'RUN: at line 16';   /magma/fuzzers/symcc_afl/symcc/build/test/../symcc -O2 /magma/fuzzers/symcc_afl/symcc/test/file_input.c -o /magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.tmp
: 'RUN: at line 17';   env SYMCC_INPUT_FILE=/magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.input /magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.tmp /magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.input 2>&1 | FileCheck --check-prefix=QSYM --check-prefix=ANY /magma/fuzzers/symcc_afl/symcc/test/file_input.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 15"
$ "/bin/echo" "-ne" "\x00\x00\x00\x05aaaa"
$ ":" "RUN: at line 16"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/../symcc" "-O2" "/magma/fuzzers/symcc_afl/symcc/test/file_input.c" "-o" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.tmp"
# command stderr:
Warning: losing track of symbolic expressions at inline assembly   %33 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %27) #4, !srcloc !8
Warning: losing track of symbolic expressions at inline assembly   %120 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %114) #4, !srcloc !9

$ ":" "RUN: at line 17"
$ "env" "SYMCC_INPUT_FILE=/magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.input" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.tmp" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/file_input.c.input"
$ "FileCheck" "--check-prefix=QSYM" "--check-prefix=ANY" "/magma/fuzzers/symcc_afl/symcc/test/file_input.c"
# command stderr:
/magma/fuzzers/symcc_afl/symcc/test/file_input.c:70:10: error: ANY: expected string not found in input
 // ANY: Not sure
         ^
<stdin>:7:20: note: scanning from here
[INFO] New testcase: /magma/fuzzers/symcc_afl/symcc/build/test/SymccOutput/000000
                   ^
<stdin>:9:56: note: possible intended match here
Warning: input file opened multiple times; this is not yet supported
                                                       ^

error: command failed with exit status: 1

--

********************
FAIL: compiler :: globals.c (15 of 16)
******************** TEST 'compiler :: globals.c' FAILED ********************
Script:
--
: 'RUN: at line 15';   /magma/fuzzers/symcc_afl/symcc/build/test/../symcc -O2 /magma/fuzzers/symcc_afl/symcc/test/globals.c -o /magma/fuzzers/symcc_afl/symcc/build/test/Output/globals.c.tmp
: 'RUN: at line 16';   echo -ne "\x00\x00\x00\x05" | /magma/fuzzers/symcc_afl/symcc/build/test/Output/globals.c.tmp 2>&1 | FileCheck --check-prefix=QSYM --check-prefix=ANY /magma/fuzzers/symcc_afl/symcc/test/globals.c
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 15"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/../symcc" "-O2" "/magma/fuzzers/symcc_afl/symcc/test/globals.c" "-o" "/magma/fuzzers/symcc_afl/symcc/build/test/Output/globals.c.tmp"
# command stderr:
Warning: losing track of symbolic expressions at inline assembly   %22 = call i32 asm "bswap $0", "=r,0,~{dirflag},~{fpsr},~{flags}"(i32 %16) #6, !srcloc !8

$ ":" "RUN: at line 16"
$ "echo" "-ne" "\x00\x00\x00\x05"
$ "/magma/fuzzers/symcc_afl/symcc/build/test/Output/globals.c.tmp"
$ "FileCheck" "--check-prefix=QSYM" "--check-prefix=ANY" "/magma/fuzzers/symcc_afl/symcc/test/globals.c"
# command stderr:
/magma/fuzzers/symcc_afl/symcc/test/globals.c:73:19: error: QSYM-COUNT: expected string not found in input (1 out of 2) // QSYM-COUNT-2: SMT
                  ^
<stdin>:1:1: note: scanning from here
This is SymCC running with the QSYM backend
^
<stdin>:1:9: note: possible intended match here
This is SymCC running with the QSYM backend
        ^

error: command failed with exit status: 1

--

********************
PASS: compiler :: bswap.c (16 of 16)
Testing Time: 0.79s
********************
Failing Tests (7):
    compiler :: file_input.c
    compiler :: globals.c
    compiler :: loop.c
    compiler :: memcpy.c
    compiler :: pointers.c
    compiler :: structs.c
    compiler :: switch.c

  Expected Passes    : 9
  Unexpected Failures: 7
FAILED: test/CMakeFiles/check 
cd /magma/fuzzers/symcc_afl/symcc/build/test && lit --verbose --path=/usr/lib/llvm-9/bin /magma/fuzzers/symcc_afl/symcc/build/test
ninja: build stopped: subcommand failed.
The command '/bin/sh -c ${FUZZER}/fetch.sh && ${FUZZER}/build.sh' returned a non-zero code: 1

failed for other targets for symcc_afl as well .

Thanks!

hazimeh commented 4 years ago

@cryptomadco I forgot to mention, the campaign duration used for plot generation is currently hard-coded (in seconds) in BenchmarkData.py:63. Don't forget to change it to match your configrc file. In the meantime, I'll be working on finishing the report generation pipeline.

cryptomadco commented 4 years ago

Hi . As I noticed there are problems in building libxml2 target with different fuzzers . I faced this multiple times .

On the other hand, whenever I want to generate reports, I get the following :

  cbar_ax = hax.figure.add_axes(cbar_ax_bbox or [0.95, 0.35, 0.04, 0.3])
/home/crypto/magma/tools/report_df/MatplotlibPlotter.py:109: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  fig.tight_layout(pad=2.0)

Currently, I run the test in 2 hours cycles, also I fixed what you mentioned about seconds in BenchmarkData.py:63 . so what's the meaning of the above warning especially this part :

This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.

Thanks!