search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 270 forks source link

Seed experiment #2025

Open tokatoka opened 3 months ago

tokatoka commented 3 months ago

This is a experiment to see how five fuzzers (afl, aflpp, libfuzzer, honggfuzz, libafl) behaves when supplied with a empty seeds and saturated seed corpus.

First I want do an empty seed experiment. Later i will change it to use ossfuzz corpus

tokatoka commented 3 months ago

The command is 

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

@DonggeLiu Could you run the command?

DonggeLiu commented 3 months ago

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-empty_seeds --fuzzers aflplusplus libafl honggfuzz libfuzzer afl aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

Sure, but there are 2 issues:

  1. Could you please follow the same date format in experiment names? 2024-08-14. This help us keep the cloud storage more organized.
  2. Like last time, this experiment requests too many fuzzers. Could you please split them into 2 experiments. Let's keep #fuzzers <= 5 in each exp.
tokatoka commented 3 months ago

ok i edited and fixed the issues https://github.com/google/fuzzbench/pull/2025#issuecomment-2286274645

tokatoka commented 3 months ago

this is dependent on #2023 #2021

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

DonggeLiu commented 3 months ago

Ops, experiment name cannot contain _.

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty-seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

tokatoka commented 3 months ago

it looks like it didn't run can you check what is the reason?

DonggeLiu commented 3 months ago

2024-08-14-empty-seeds-2

it looks like it didn't run can you check what is the reason?

I think it is because of the same issue I fixed in #2023 Now that you have updated your branch, let me re-run the exp.

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

DonggeLiu commented 3 months ago

Experiment 2024-08-17-empty-seeds-1 data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

Experiment 2024-08-17-empty-seeds-2 data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

DonggeLiu commented 3 months ago

Hi @tokatoka, I started two test exp above. If they work well, could you please rebase to adopt this change? I forgot to revert temp changes in a previous PR.

Thanks!

tokatoka commented 3 months ago

hi, i merged the master branch to revert it. now it's ready

tokatoka commented 3 months ago

now it's truly okay (with the green ci) could you run the same command? @DonggeLiu

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

tokatoka commented 3 months ago

hmm the experiment-data has data in its folder but the report is not generated do you know the reason? @DonggeLiu

DonggeLiu commented 2 months ago

Let's debug this on the other PR #2028 This experiment also had many runtime crashes, for example.

tokatoka commented 2 months ago

I updated libafl commit here too. Can we start the experiment here?

DonggeLiu commented 2 months ago

I updated libafl commit here too. Can we start the experiment here?

Sure, I wonder if that heisenbug will haunt us here. Let's run for once, and if the bug reappears, I will create a new PR with my debug logs.

DonggeLiu commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

tokatoka commented 2 months ago

No it doesn't work. Just like before. The experiment-data is there but the report is missing

tokatoka commented 2 months ago

@DonggeLiu How can I help to debug this stuff..?

DanBlackwell commented 2 months ago

@DonggeLiu How can I help to debug this stuff..?

https://github.com/google/fuzzbench/pull/2000#issuecomment-2272406883 Could it be related to this? maybe you can try run just zlib on it's own and see if there's the same error?

tokatoka commented 2 months ago

@DonggeLiu Can you create a debug branch for this too..? For this one, 0 reports were created

DonggeLiu commented 2 months ago

yep I saw a lot of "Fuzz target binary not found." errors: image

It's likely the same error again. Weird that it stopped occurring for a year and suddenly came back.

DonggeLiu commented 2 months ago

@DonggeLiu How can I help to debug this stuff..?

#2000 (comment) Could it be related to this? maybe you can try run just zlib on it's own and see if there's the same error?

Thanks for helping @DanBlackwell . We will investigate more in a new PR.

DonggeLiu commented 2 months ago

@DonggeLiu Can you create a debug branch for this too..? For this one, 0 reports were created

I will need your help to resolve the conflict before I can rebase and create a new PR : )

Auto-merging fuzzers/libafl/builder.Dockerfile
CONFLICT (content): Merge conflict in fuzzers/libafl/builder.Dockerfile
error: could not apply 8b14d84f... Update LibAFL (#2036)
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 8b14d84f... Update LibAFL (#2036)

Or you could cherry-pick my commits from https://github.com/google/fuzzbench/pull/2038.

tokatoka commented 2 months ago

resolved it