search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.09k stars 266 forks source link

[DO NOT MERGE] A standard experiment #1929

Open DonggeLiu opened 9 months ago

DonggeLiu commented 9 months ago

This reverts commit c734a74ba59deada3cf4d7e37009125e26d51701 and adds a dummy change to run experiments.

DonggeLiu commented 9 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-08-standard-cov --fuzzers afl aflfast aflplusplus aflsmart centipede eclipser fairfuzz honggfuzz libafl libfuzzer mopt

DonggeLiu commented 9 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-08-standard-bug --fuzzers afl aflfast aflplusplus aflsmart centipede eclipser fairfuzz honggfuzz libafl libfuzzer mopt --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

DonggeLiu commented 9 months ago

Experiment 2023-12-08-standard-cov data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-08-standard-bug data and results will be available later at: The experiment data. The experiment report.

jonathanmetzman commented 9 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-09-standard-cov --fuzzers afl aflfast aflplusplus aflsmart centipede eclipser fairfuzz honggfuzz libafl libfuzzer mopt

jonathanmetzman commented 9 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-08-standard-bug --fuzzers afl aflfast aflplusplus aflsmart centipede eclipser fairfuzz honggfuzz libafl libfuzzer mopt --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

jonathanmetzman commented 9 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-08-standard-bug --fuzzers afl aflfast aflplusplus aflsmart centipede eclipser fairfuzz honggfuzz libafl libfuzzer mopt --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

DonggeLiu commented 9 months ago

These two do not seem to work either:

Experiment 2023-12-09-standard-cov data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-08-standard-bug (forgot to change the name? : )) data and results will be available later at: The experiment data. The experiment report.

In particular, the new 2023-12-09-standard-cov's experiment data dir is empty, but the old 2023-12-08-standard-cov was not.

DonggeLiu commented 9 months ago

Oh I just have a theory about why 2023-12-09-standard-cov has not data:

Did you, by any chance, happen to comment the bug-based exp command right after the cov-based one? This will interrupt the cov experiment.

DonggeLiu commented 9 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-10-standard-cov --fuzzers afl aflfast aflplusplus aflsmart centipede eclipser fairfuzz honggfuzz libafl libfuzzer mopt

DonggeLiu commented 9 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-10-standard-bug --fuzzers afl aflfast aflplusplus aflsmart centipede eclipser fairfuzz honggfuzz libafl libfuzzer mopt --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

DonggeLiu commented 9 months ago

Experiment 2023-12-10-standard-cov data and results will be available later at: The experiment data. The experiment report

Experiment 2023-12-10-standard-bug data and results will be available later at: The experiment data. The experiment report.

jonathanmetzman commented 9 months ago

Dongge, FYI the URLs are wrong (a silly feature we have doesn't put "standard experiments" in experimental). They actually are: https://storage.googleapis.com/www.fuzzbench.com/reports/2023-12-10-standard-bug/index.html https://storage.googleapis.com/www.fuzzbench.com/reports/2023-12-10-standard-cov/index.html

DonggeLiu commented 9 months ago

Dongge, FYI the URLs are wrong (a silly feature we have doesn't put "standard experiments" in experimental). They actually are: https://storage.googleapis.com/www.fuzzbench.com/reports/2023-12-10-standard-bug/index.html https://storage.googleapis.com/www.fuzzbench.com/reports/2023-12-10-standard-cov/index.html

Ah I see, thanks, that explains it. I was wondering why we have the experiment data but not the report.

Is it a one-time thing or does it mean we will stop using experimental dir? I will edit my script to generate this comment:

Experiment 2023-12-10-standard-bug data and results will be available later at: