search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.09k stars 266 forks source link

Orchesctra: A Collaborative Fuzzer Framework #1904

Closed gtt1995 closed 5 months ago

gtt1995 commented 1 year ago

1859

Hello dongge, @Alan32Liu I've improved my fuzzer framework based on the data from the last experiment, and it's time to reopen this experiment. Other than that, to avoid duplicating the experiment overhead, I've only submitted the necessary fuzzers, so please review it and get it running.

Best regards, Taotao.

DonggeLiu commented 1 year ago

OK, just to be extra safe, would it be alright if we split them into the following two groups and run them separately? Group 1:

- orchestra_236
- orchestra_339
- orchestra_4310
- orchestra_219
- orchestra_116

Group 2:

- orchestra_aflplusplus
- orchestra_honggfuzz
- orchestra_libfuzzer
- pastis

If you'd like to see their result in one table, we can download the raw CSV data (data.csv.gz) from the end of the report, which can be used to regenerate the combined report with selected fuzzers and benchmarks.

DonggeLiu commented 1 year ago

If you agree, the commands are: (future note for myself)

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-2 --fuzzers orchestra_aflplusplus orchestra_honggfuzz orchestra_libfuzzer pastis
DonggeLiu commented 1 year ago

BTW, just a minor future request if it is not too much trouble: Instead of editing service/experiment-requests.yaml, would there be any chance that I could invite you to write the experiment request in this format? You can swap the --experiment-name and --fuzzers parameters with your values. I would really appreciate that : )

The file service/experiment-requests.yaml was no longer used to trigger experiments, as it requires merging PRs. The new way (via gcbrun comments) allows us to run experiments and modify code multiple times on the same PR for better maintenance. I believe this improvement will benefit both of us.

My comment above is another example of this : )

gtt1995 commented 1 year ago

I totally agree with you, dongge. 😄

---Original--- From: "Dongge @.> Date: Tue, Sep 26, 2023 08:31 AM To: @.>; Cc: "taotao @.**@.>; Subject: Re: [google/fuzzbench] Orchesctra: A Collaborative Fuzzer Framework(PR #1904)

OK, just to be extra safe, would it be alright if we split them into the following two groups and run them separately? Group 1:

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

gtt1995 commented 1 year ago

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116

gtt1995 commented 1 year ago

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-2 --fuzzers orchestra_aflplusplus orchestra_honggfuzz orchestra_libfuzzer pastis

DonggeLiu commented 1 year ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116

DonggeLiu commented 1 year ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-27-orchestra-1 --fuzzers orchestra_116 orchestra_216 orchestra_236 orchestra_339 orchestra_4310

DonggeLiu commented 1 year ago

Experiment 2023-09-27-orchestra-1 data and results will be available later at: The experiment data. The experiment report.

DonggeLiu commented 1 year ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-2 --fuzzers orchestra_aflplusplus orchestra_honggfuzz orchestra_libfuzzer pastis

DonggeLiu commented 1 year ago

Experiment 2023-09-25-orchestra-2 data and results will be available later at: The experiment data. The experiment report.

DonggeLiu commented 1 year ago

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116

I reckon orchestra_219 is actually orchestra_216?

gtt1995 commented 1 year ago

Yes, dongge. The config of 216 is same as 219 when the total time budget less than 23h.

---Original--- From: "Dongge @.> Date: Wed, Sep 27, 2023 10:41 AM To: @.>; Cc: "taotao @.**@.>; Subject: Re: [google/fuzzbench] Orchesctra: A Collaborative Fuzzer Framework(PR #1904)

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116

I reckon orchestra_219 is actually orchestra_216?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

DonggeLiu commented 1 year ago

Yes, dongge. The config of 216 is same as 219 when the total time budget less than 23h.

I meant there isn't a 219, is this intended?

gtt1995 commented 1 year ago

Yes, you can delete 219, thanks.

---Original--- From: "Dongge @.> Date: Wed, Sep 27, 2023 18:34 PM To: @.>; Cc: "taotao @.**@.>; Subject: Re: [google/fuzzbench] Orchesctra: A Collaborative Fuzzer Framework(PR #1904)

Yes, dongge. The config of 216 is same as 219 when the total time budget less than 23h.

I meant there isn't a 219, is this intended?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>