search

MelbourneFuzzingHub / aflteam

AFLTeam Collaborative Parallel Fuzzing
https://thuanpv.github.io/publications/AFLTeam-ASE21-NIER.pdf
Apache License 2.0
72 stars 11 forks source link

aflteam-manager.py crash when fuzz tcpdump #4

Open zr950624 opened 2 years ago

zr950624 commented 2 years ago

Hi there, I tried to fuzz tcpdump with aflteam. After running a while, aflteam-manager.py got crashed but the master AFL process still run. I would like to know how to fix this issue. Please see following backtrace.

  File "/home/microfuzz_benchmark/aflteam/aflteam-manager.py", line 267, in <module>
    main(args.binary, args.afl_binary, args.horsefuzz_binary, args.profiling_binary, args.gcov_binary, args.gcov_folder, args.pre_arguments, args.post_arguments, args.seed_corpus, args.out_folder, args.dict, args.dot_file, args.func_ids, args.func_bbs, args.cores, args.algorithm, args.total_timeout, args.scanning_timeout, args.expl_timeout)
  File "/home/microfuzz_benchmark/aflteam/aflteam-manager.py", line 175, in main
    lukes.partition(CG, main_v, v_fname_dict, fname_src_dict, fname_bbs_dict, cores - 1, taskDir)
  File "/home/microfuzz_benchmark/aflteam/tasks/lukes_partitioning.py", line 48, in partition
    total_branches = total_branches + CG.nodes[v]['btotal']
KeyError: 'btotal'
thuanpv commented 2 years ago

Thanks for reporting the issue. According to the error message, the current node v does not have 'btotal' attribute. I think you may fix it by adding a check for that corner case.

MelbourneFuzzingHub commented 2 years ago

I have tried to fix this in the recent commit so I close this issue. Please feel free to reopen it if the issue still persists.

zr950624 commented 2 years ago

Hi, I want to reopen this issue because the tcpdump is still crashing when I apply your update. After looking at your commit, I think it is not a TypeError, but a KeyError.

Also, there are some other crash points like following

 File "/tmp/fuzz_benchmarks/aflteam/aflteam-manager.py", line 267, in <module>
    main(args.binary, args.afl_binary, args.horsefuzz_binary, args.profiling_binary, args.gcov_binary, args.gcov_folder, args.pre_arguments, args.post_arguments, args.seed_corpus, args.out_folder, args.dict, args.dot_file, args.func_ids, args.func_bbs, args.cores, args.algorithm, args.total_timeout, args.scanning_timeout, args.expl_timeout)
  File "/tmp/fuzz_benchmarks/aflteam/aflteam-manager.py", line 175, in main
    lukes.partition(CG, main_v, v_fname_dict, fname_src_dict, fname_bbs_dict, cores - 1, taskDir)
  File "/tmp/fuzz_benchmarks/aflteam/tasks/lukes_partitioning.py", line 57, in partition
    score = (CG.nodes[v]['bcovered_cur'] - CG.nodes[v]['bcovered_pre'] + 1) * (CG.nodes[v]['btotal'] - CG.nodes[v]['bcovered_cur'] + 1)
KeyError: 'bcovered_cur'
thuanpv commented 2 years ago

Thanks! Would you mind sharing with me the detailed instructions to reproduce the issue so that I can apply a more appropriate patch instead of a workaround? I cannot reproduce the issue with LibPNG, which is my common subject for testing.

zr950624 commented 2 years ago

Hi, I just clone the latest tcpdump and run ./configure in tcpdump with the same command in https://github.com/MelbourneFuzzingHub/aflteam/blob/main/experiments/Makefile#L91. There is no special configure flag. Then command line I use to run aflteam is following:

HF_BINARY=tcpdump $AFLTEAM/aflteam-manager.py -bn tcpdump -ab $SUBJECTS/tcpdump-afl/tcpdump -hb $SUBJECTS/tcpdump-horsefuzz/tcpdump -pb $SUBJECTS/tcpdump-horsefuzz-profiling/tcpdump -gb $SUBJECTS/tcpdump-cov/tcpdump -gf $SUBJECTS/tcpdump-cov -d $SUBJECTS/tcpdump-wllvm/tcpdump.dot -i ~/tmp/testcases/tcpdump -x $AFLSMART/dictionaries/webp.dict -f /tmp/tcpdump/func_ids.log -b /tmp/tcpdump/func_bbs.log -c 10 -o $RESULTS/out-tcpdump-aflteam -a lukes -tt 36000 -st 60 -et 60 -ea1 "" -ea2 " -r"

The seed is just same random pacp files.

thuanpv commented 2 years ago

Thanks! It would be great if you could provide the detailed Makefile entries for different tcpdump binaries, like what we did for LibPNG and other subjects. I would really appreciate that since it could save my time.

zr950624 commented 2 years ago

Hi, thanks for replying to me. However, I do not have a Makefile, I just ran the commands one by one.

thuanpv commented 2 years ago

Would you mind copying all the commands you used to compile and run tcpdump with AFLTeam here? I can convert them to Makefile format. I did try to compile tcpdump but I got some errors. I have very limited time but I want to fix this issue so I would really appreciate your help!

zr950624 commented 2 years ago

Hi, I wrote a google zx script to install tcpdump. It will compile tcpdump with aflteam in /tmp/fuzz_benchmarks. Hope this can help you well.

#!/usr/bin/env zx

let WORKDIR = "/tmp/fuzz_benchmarks"
let FUZZ_TARGET = WORKDIR + "/targets"
let RESULTS = WORKDIR + "/results"
let AFLTEAM_PATH = WORKDIR + "/aflteam"
let HF_PATH = WORKDIR + "/horsefuzz"
let AFL_PATH = WORKDIR + "/afl"
let AFLSMART_PATH = WORKDIR + "/aflsmart"

await $`sudo apt-get -y install clang-6.0 tcl`
await $`sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-6.0 1000`
await $`sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-6.0 1000`

$.env.WORKDIR = WORKDIR
$.env.LLVM_CONFIG = "llvm-config-6.0"
$.env.LLVM_COMPILER = "clang"
$.env.PATH = [$.env.PATH, AFL_PATH, HF_PATH, AFLTEAM_PATH].join(":")

/// clone aflteam
await $`rm -rf ${WORKDIR} && git clone https://github.com/MelbourneFuzzingHub/aflteam.git ${AFLTEAM_PATH}`

/// aflteam setup
cd(WORKDIR)
await $`mkdir ${FUZZ_TARGET} && mkdir ${WORKDIR}/results`
await $`make -f ${AFLTEAM_PATH}/experiments/Makefile prerequisites`
await $`make -f ${AFLTEAM_PATH}/experiments/Makefile afl`
await $`make -f ${AFLTEAM_PATH}/experiments/Makefile aflsmart`
await $`make -f ${AFLTEAM_PATH}/experiments/Makefile horsefuzz`

async function setupTarget(TargetNames, RepoList, RepoCommand, ConfigureFile, ConfigFlags, MakeCommand,
    MakeFlags, WllvmMakeCommand = "", HorseMakeCommand = "") {

    let TargetName = TargetNames[0]
    let TARGET_PATH = path.join(FUZZ_TARGET, TargetName)
    await $`mkdir ${TARGET_PATH}`
    cd(TARGET_PATH)

    for (var idx = 0; idx < TargetNames.length; idx++) {
        await $`git clone ${RepoList[idx]} ${TARGET_PATH}/${TargetNames[idx]}`
        for await (let command of RepoCommand[idx]) {
            console.log("current: " + TARGET_PATH + "/" + TargetNames[idx])
            await $`cd ${path.join(TARGET_PATH, TargetNames[idx])} && ${command}`
        }
    }

    /// Instrument with afl
    within(async () => {
        await $`mkdir ${TARGET_PATH}/bld_afl`
        cd(path.join(TARGET_PATH, "/bld_afl"))

        await $`CC=${AFL_PATH}/afl-clang-fast CXX=${AFL_PATH}/afl-clang-fast++ \
    ${TARGET_PATH}/${TargetName}/${ConfigureFile} ${ConfigFlags} && ${MakeCommand} ${MakeFlags}`
    })

    /// Instrument with wllvm
    within(async () => {
        await $`mkdir ${TARGET_PATH}/bld_wllvm`
        cd(path.join(TARGET_PATH, "/bld_wllvm"))

        let TmpWllvmMake = MakeCommand
        if (WllvmMakeCommand.length != 0) {
            TmpWllvmMake = WllvmMakeCommand
        }
        await $`CC=wllvm CXX=wllvm++  ${TARGET_PATH}/${TargetName}/${ConfigureFile} ${ConfigFlags} && \
                ${TmpWllvmMake} ${MakeFlags}`
        await $`extract-bc -l llvm-link-6.0 ${TargetName} && \
             opt-6.0 -analyze -dot-callgraph < ${TargetName}.bc && \
             mv callgraph.dot ${TargetName}.dot`
    })

    /// Instrument with horsefuzz
    within(async () => {
        // Instrument horsefuzz log
        await $`mkdir ${TARGET_PATH}/bld_horsefuzz`
        cd(path.join(TARGET_PATH, "/bld_horsefuzz"))

        let TmpHorseFuzzMake = MakeCommand
        if (HorseMakeCommand.length != 0)
            TmpHorseFuzzMake = HorseMakeCommand

        await $`rm -rf /tmp/hf_${TargetName} && mkdir /tmp/hf_${TargetName}`
        await $`HF_BINARY=hf_${TargetName}  LD_LIBRARY_PATH=/usr/lib/llvm-6.0/lib \
                CC=${HF_PATH}/horsefuzz-clang-fast \
                CXX=${HF_PATH}/horsefuzz-clang-fast++ \
                ${TARGET_PATH}/${TargetName}/${ConfigureFile} ${ConfigFlags}`
        await $`HF_BINARY=hf_${TargetName}  LD_LIBRARY_PATH=/usr/lib/llvm-6.0/lib \
                ${TmpHorseFuzzMake} ${MakeFlags}`
        await $`mv /tmp/hf_${TargetName} ${TargetName}-horsefuzz-logs`

        // Instrument horsefuzz profiling
        await $`mkdir ${TARGET_PATH}/bld_horsefuzz_profiling`
        cd(path.join(TARGET_PATH, "/bld_horsefuzz_profiling"))

        await $`rm -rf /tmp/hf_${TargetName} && mkdir /tmp/hf_${TargetName}`
        await $`HF_BINARY=hf_${TargetName}  HORSEFUZZ_CG_PROFILING=1 \
                LD_LIBRARY_PATH=/usr/lib/llvm-6.0/lib \
                CC=${HF_PATH}/horsefuzz-clang-fast \
                CXX=${HF_PATH}/horsefuzz-clang-fast++ \
                ${TARGET_PATH}/${TargetName}/${ConfigureFile} ${ConfigFlags}`
        await $`HF_BINARY=hf_${TargetName}  HORSEFUZZ_CG_PROFILING=1 \
                LD_LIBRARY_PATH=/usr/lib/llvm-6.0/lib \
                ${TmpHorseFuzzMake} ${MakeFlags}`
        await $`mv /tmp/hf_${TargetName} ${TargetName}-horsefuzz-logs`
    })

    /// Instrument with cov
    within(async () => {
        await $`mkdir ${TARGET_PATH}/bld_cov`
        cd(path.join(TARGET_PATH, "/bld_cov"))
        await $`CFLAGS="-g -fprofile-arcs -ftest-coverage" \
                ${TARGET_PATH}/${TargetName}/${ConfigureFile} ${ConfigFlags} && \
                ${MakeCommand} ${MakeFlags}`
    })

}

/// Setup tcpdump
let TcpDumpTargets = ["tcpdump", "libpcap"]
let TcpDumpRepoLists = ["https://github.com/the-tcpdump-group/tcpdump.git",
    "https://github.com/the-tcpdump-group/libpcap.git"]
let TcpDumpRepoCommand = [["true"], ["./configure", ["make", "-j"]]]
let TcpDumpConfigureFile = "configure"
let TcpDumpConfigureFlags = []
let TcpDumpMakeCommand = "make"
let TcpDumpMakeFlags = ["-j"]
setupTarget(TcpDumpTargets, TcpDumpRepoLists, TcpDumpRepoCommand, TcpDumpConfigureFile, TcpDumpConfigureFlags,
    TcpDumpMakeCommand, TcpDumpMakeFlags)
thuanpv commented 2 years ago

I can compile and run AFLTeam with tcpdump now. Thanks! I have noticed that there is an issue in your command to run AFLTeam. The " -r" should be passed to ea1 instead of ea2 (-ea1 " -r" -ea2 ""). A quick run with the updated arguments gives me no exceptions. I have started a longer run for 10 hours to see how it goes.

thuanpv commented 2 years ago

My 10-hr experiment is completed and there are no exceptions. Could you try again, using the newest code & correct command, and let me know if the issue persists? If there are issues, please share with me the file aflteam.log and the error message. Thanks.

zr950624 commented 2 years ago

My 10-hr experiment is completed and there are no exceptions. Could you try again, using the newest code & correct command, and let me know if the issue persists? If there are issues, please share with me the file aflteam.log and the error message. Thanks.

Hi, it still crashed with the following message after using the latest code.

File "/tmp/fuzz_benchmarks/aflteam/aflteam-manager.py", line 267, in <module>
    main(args.binary, args.afl_binary, args.horsefuzz_binary, args.profiling_binary, args.gcov_binary, args.gcov_folder, args.pre_arguments, args.post_arguments, args.seed_corpus, args.out_folder, args.dict, args.dot_file, args.func_ids, args.func_bbs, args.cores, args.algorithm, args.total_timeout, args.scanning_timeout, args.expl_timeout)
  File "/tmp/fuzz_benchmarks/aflteam/aflteam-manager.py", line 175, in main
    lukes.partition(CG, main_v, v_fname_dict, fname_src_dict, fname_bbs_dict, cores - 1, taskDir)
  File "/tmp/fuzz_benchmarks/aflteam/tasks/lukes_partitioning.py", line 57, in partition
    score = (CG.nodes[v]['bcovered_cur'] - CG.nodes[v]['bcovered_pre'] + 1) * (CG.nodes[v]['btotal'] - CG.nodes[v]['bcovered_cur'] + 1)
KeyError: 'bcovered_cur'

CommandLine I used

HF_BINARY=tcpdump /tmp/fuzz_benchmarks/aflteam/aflteam-manager.py -bn tcpdump  -ab /tmp/fuzz_benchmarks/targets/tcpdump/bld_afl/tcpdump  -hb /tmp/fuzz_benchmarks/targets/tcpdump/bld_horsefuzz/tcpdump  -pb /tmp/fuzz_benchmarks/targets/tcpdump/bld_horsefuzz_profiling/tcpdump  -gb /tmp/fuzz_benchmarks/targets/tcpdump/bld_cov/tcpdump  -gf /tmp/fuzz_benchmarks/targets/tcpdump/bld_cov  -d /tmp/fuzz_benchmarks/targets/tcpdump/bld_wllvm/tcpdump.dot  -i /home/yongheng/zr/MicroFuzz/test_bins/tcpdump_corpus  -x /tmp/fuzz_benchmarks/aflsmart/dictionaries/webp.dict  -f /tmp/tcpdump/func_ids.log  -b /tmp/tcpdump/func_bbs.log -c 5  -o /tmp/fuzz_benchmarks/results/out-tcpdump  -a lukes -tt 36000 -st 60 -et 60 -ea1 " -r" -ea2 ""

Aflteam version

commit c0dd8faad7739f0b6d14e1390dfe1a43379f7896 (HEAD -> main, origin/main, origin/HEAD)
Author: Thuan Pham <thuan.pham@unimelb.edu.au>
Date:   Mon Sep 5 15:40:29 2022 +1000

    Fix a wrong exception type

commit 36948dc814656d9729299ef79461479cb400649d
Author: Thuan Pham <thuan.pham@unimelb.edu.au>
Date:   Sun Aug 28 18:38:13 2022 +1000

    An attemp to fix Issue #2

commit 3a438b9e44de47bf68e88b7a3f682eb3edf6e71b
Author: Thuan Pham <thuan.pham@unimelb.edu.au>
Date:   Sun Aug 28 15:22:08 2022 +1000

    Fix a TypeError (Issue #4) and a File-not-found bug

Follows aflteam.log

DEBUG:root:AFLTeam starts
 DEBUG:root:extract_callgraph starts at: 05/09/2022 21:29:31
 DEBUG:root:extract_callgraph ends at: 05/09/2022 21:29:39
 DEBUG:root:Monitor command: afl-fuzz -m 1G -t 20000+ -x /tmp/fuzz_benchmarks/aflsmart/dictionaries/webp.dict -o /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs -i /tmp/fuzz_benchmarks/results/out- tcpdump/active_runs/seeds_origin -S monitor -T AFLTeam-Monitor /tmp/fuzz_benchmarks/targets/tcpdump/bld_afl/tcpdump -r @@
 DEBUG:root:Monitor PID: 2536112
 DEBUG:root:Round_1, Exploitation: False
 DEBUG:root:Is CG acyclic?: False
 DEBUG:root:update_callgraph starts at: 05/09/2022 21:29:39
 DEBUG:root:update_callgraph ends at: 05/09/2022 21:29:40
 DEBUG:root:prune_callgraph starts at: 05/09/2022 21:29:40
 DEBUG:root:Nodes-before-prunning-1: 934
 DEBUG:root:Edges-before-prunning-1: 2624
 DEBUG:root:Nodes-before-prunning-2: 632
 DEBUG:root:Edges-before-prunning-2: 1040
 DEBUG:root:Nodes-before-prunning-3: 529
 DEBUG:root:Edges-before-prunning-3: 1040
 DEBUG:root:Nodes-after-all-prunning: 42
 DEBUG:root:Edges-after-all-prunning: 58
 DEBUG:root:prune_callgraph ends at: 05/09/2022 21:29:40
 DEBUG:root:Fuzzer command: afl-fuzz -m 1G -t 20000+ -x /tmp/fuzz_benchmarks/aflsmart/dictionaries/webp.dict -o /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs -i /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/seeds -S fuzzer_1 /tmp/fuzz_benchmarks/targets/tcpdump/bld_afl/tcpdump -r @@
 DEBUG:root:PID: 2537863
 DEBUG:root:Fuzzer command: afl-fuzz -m 1G -t 20000+ -x /tmp/fuzz_benchmarks/aflsmart/dictionaries/webp.dict -o /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs -i /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/seeds -S fuzzer_2 /tmp/fuzz_benchmarks/targets/tcpdump/bld_afl/tcpdump -r @@
 DEBUG:root:PID: 2546378
 DEBUG:root:Fuzzer command: afl-fuzz -m 1G -t 20000+ -x /tmp/fuzz_benchmarks/aflsmart/dictionaries/webp.dict -o /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs -i /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/seeds -S fuzzer_3 /tmp/fuzz_benchmarks/targets/tcpdump/bld_afl/tcpdump -r @@
 DEBUG:root:PID: 2556365
 DEBUG:root:Fuzzer command: afl-fuzz -m 1G -t 20000+ -x /tmp/fuzz_benchmarks/aflsmart/dictionaries/webp.dict -o /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs -i /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/seeds -S fuzzer_4 /tmp/fuzz_benchmarks/targets/tcpdump/bld_afl/tcpdump -r @@
 DEBUG:root:PID: 2567273
 DEBUG:root:Round_2, Exploitation: True
 DEBUG:root:Saving: /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/fuzzer_1
 DEBUG:root:Saving to: /tmp/fuzz_benchmarks/results/out-tcpdump/backup_round_1
 DEBUG:root:Saving: /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/fuzzer_2
 DEBUG:root:Saving to: /tmp/fuzz_benchmarks/results/out-tcpdump/backup_round_1
 DEBUG:root:Saving: /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/fuzzer_3
 DEBUG:root:Saving to: /tmp/fuzz_benchmarks/results/out-tcpdump/backup_round_1
 DEBUG:root:Saving: /tmp/fuzz_benchmarks/results/out-tcpdump/active_runs/fuzzer_4
 DEBUG:root:Saving to: /tmp/fuzz_benchmarks/results/out-tcpdump/backup_round_1
 DEBUG:root:Is CG acyclic?: True
 DEBUG:root:Nodes-before-update: 42
 DEBUG:root:Edges-before-update: 58
 DEBUG:root:update_callgraph starts at: 05/09/2022 21:31:00
 DEBUG:root:update_callgraph ends at: 05/09/2022 21:31:03
 DEBUG:root:Nodes-after-update: 139
 DEBUG:root:Edges-after-update: 171
DEBUG:root:Nodes-before-prunning-2: 105
 DEBUG:root:Edges-before-prunning-2: 130
 DEBUG:root:Nodes-before-prunning-3: 104
 DEBUG:root:Edges-before-prunning-3: 130
 DEBUG:root:Nodes-after-all-prunning: 42
 DEBUG:root:Edges-after-all-prunning: 58
 DEBUG:root:prune_callgraph ends at: 05/09/2022 21:31:03
 DEBUG:root:do_partitioning_lukes starts at: 05/09/2022 21:31:03
 DEBUG:root:Running minimum_spanning_arborescence algorithm ...
zr950624 commented 2 years ago

My 10-hr experiment is completed and there are no exceptions. Could you try again, using the newest code & correct command, and let me know if the issue persists? If there are issues, please share with me the file aflteam.log and the error message. Thanks.

The python exceptions will appear and disappear in a short time(maybe 5s), and the afl-fuzz master process will flush the screen so you may see nothing. This is the reason I change to -st 60 -et 60.

thuanpv commented 2 years ago

Interesting! In fact I tried both, with long and short "-st/-et" options. Would you mind sharing with me the seed corpus as well? This could be the only thing that is different between our setups.

zr950624 commented 2 years ago

tcpdump_corpus.tar.gz Please see the compressed file. Will the corpus affect CG? This is a little bit out of my mind..

seaslug2 commented 5 months ago

Hi there, I encountered the same issue. Have you found a solution? If so, could you please share the steps you followed to resolve it? Thank you!