Testing AFL++ variant Fish++-nonLTO

[kdsjZh]

Hi dongge,

@Alan32Liu I developed a variant of FishFuzz (USENIX Security23) to make it compatible with fuzzbench (the original version in the paper rely on LTO mode, which fails/timeout on loots of fuzzbench targets), and I would like to request an evaluation to see if it works. Could you help me run the fuzzer aflplusplus_ff_cmp, aflplusplus_fishfuzz and aflplusplus_fishfuzz_allbb?

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-04-ff --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

Thanks! Han

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

[DonggeLiu]

Hi @kdsjZh Thanks for writing down the command! That makes my work a lot easier : )

Just two minor notes:

1. The --experiment-name and --fuzzers parameters need to be swapped with your values (See the example command above)
2. We need to make a trivial modification to service/gcbrun_experiment.py to launch experiments in this PR. Here is an example to add a dummy comment : )

Please feel free to ping me once you finished 2. Thanks!

[kdsjZh]

Hi dongge,

Thanks for the reminding. I've finished the dummy comment.

@Alan32Liu

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

[DonggeLiu]

Experiment 2023-10-05-fishfuzz data and results will be available later at: The experiment data. The experiment report.

[kdsjZh]

Hello Dongge @Alan32Liu,

I fixed some build errors in libpcap/zlib and optimized the exploration stage, could you help me run the aflplusplus_ff_cmp, aflplusplus_fishfuzz_allbb and aflplusplus_fishfuzz_exp again?

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-12-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_exp aflplusplus_fishfuzz_allbb

Thanks!

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-12-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_exp aflplusplus_fishfuzz_allbb

[DonggeLiu]

Experiment 2023-10-12-fishfuzz data and results will be available later at: The experiment data. The experiment report.

[kdsjZh]

Hi Dongge @Alan32Liu ,

I fix the builder script/exploration stage and updated the llvm-12 to llvm-15.0.0 (to be coherent with fuzzbench's). Besides, existing fuzzers are working on ASan version binary, I want to include non-ASan to compare with existing main results. Could you help me start another campaign? Thank!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-19-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_allbb aflplusplus_fishfuzz_noasan aflplusplus_fishfuzz_noasan_all

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-21-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_allbb aflplusplus_fishfuzz_noasan aflplusplus_fishfuzz_noasan_all

[DonggeLiu]

Experiment 2023-10-21-fishfuzz data and results will be available later at: The experiment data. The experiment report.

[kdsjZh]

Hi Dongge @Alan32Liu ,

I'm plaining to do an ablation study about how each component works, with aflpp's tracepc option only (I found one possible bug with cmplog feature, so I opt for the tracepc only). Therefore I'm wondering could you help me with that? Many thanks for your patience and help!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-25-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-25-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

[DonggeLiu]

Sure! Experiment 2023-10-25-fishfuzz data and results will be available later at: The experiment data. The experiment report.

[kdsjZh]

Hi @Alan32Liu Dongge,

I profile the fuzzer and find out that the sampling in exploitation stage has super high overhead, therefore I reduce the sampling frequency a bit and want another round of campaign if possible. Thank!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-01-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-01-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

[DonggeLiu]

Experiment 2023-11-01-fishfuzz data and results will be available later at: The experiment data. The experiment report.

[kdsjZh]

Hi Dongge,

@Alan32Liu Recently, I contacted Marc and located the cmplog issue. I integrated the fix and rewritten fishpp based on the latest AFL++. Could you help me with a dry run to test if my fix/latest integration is correct?

BTW, recently Marc tried the Fish++ vs AFL++ on bug benchmark, results shows that Fish++ is able to improve based on AFL++ baseline (ff_cmp_3). So I would like to request a bug evaluation as well.

# coverage
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-04-fishfuzz --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp
# bug
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-04-fishfuzz-bug --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Thanks!

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz-bug --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[DonggeLiu]

Experiment 2023-12-06-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-06-fishfuzz-bug data and results will be available later at: The experiment data. The experiment report.

[jonathanmetzman]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp

[jonathanmetzman]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz-bug --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[kdsjZh]

Hi @Alan32Liu ,

I analyzed the data from this experiment and found that the path frequency sampling introduced notable overhead (in execution speed), while Fishpp already implemented a cheaper alternative in its exploitation mode. Therefore I added an explore mode Fishpp to avoid this overhead.

Could you help me run another campaign to verify? Thanks!

# cov (we only need cmplog version)

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz --fuzzers fishpp_new fishpp_new_exp aflplusplus 

# bug (nocmp seems to be better in some scenarios, therefore we need both cmp and nocmp for Fishpp explore)
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz-bug --fuzzers fishpp_new_exp aflplusplus fishpp_new_nocmp_exp aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz --fuzzers fishpp_new fishpp_new_exp aflplusplus

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz-bug --fuzzers fishpp_new_exp aflplusplus fishpp_new_nocmp_exp aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[DonggeLiu]

Experiment 2023-12-11-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-11-fishfuzz-bug data and results will be available later at: The experiment data. The experiment report.

[kdsjZh]

Hi Dongge @Alan32Liu ,

I checked the data in existing reports and found out that h264 is not run.

The reports (both standard-cov and recent ff reports) are using cached data. e.g., in data.csv.gz from 2023-12-11-standard-cov, the data of openh264_decoder_fuzzer afl is copied from 2023-09-21-libafl. while aflfast's data are copied from 2023-03-20-ecofuzz.

BTW, I also notice that the aflplusplus is also not evaluated in 2023-12-11-standard-cov. and in the report, they cached results from 2023-12-10-aflpp, which uses a different aflplusplus commit. I don't know if it's an intended behavior. So this is just a head up.

[kdsjZh]

I just updated the fishpp to the AFL++ 4.09c. could you help us do another round of test? Thanks!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-15-fishfuzz --fuzzers fishpp_new fishpp_new_exp fishpp_new_fast aflplusplus

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-15-fishfuzz-bug --fuzzers fishpp_new_exp aflplusplus fishpp_new_fast fishpp_new --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[DonggeLiu]

The reports (both standard-cov and recent ff reports) are using cached data. e.g., in data.csv.gz from 2023-12-11-standard-cov, the data of openh264_decoder_fuzzer afl is copied from 2023-09-21-libafl. while aflfast's data are copied from 2023-03-20-ecofuzz.

BTW, I also notice that the aflplusplus is also not evaluated in 2023-12-11-standard-cov. and in the report, they cached results from 2023-12-10-aflpp, which uses a different aflplusplus commit. I don't know if it's an intended behavior. So this is just a head up.

Using data from old experiments is expected and default behavior. If a fuzzer-benchmark pair is not modified, there is no solid reason for re-generating its result, which can be expensive and time-consuming.

If you'd like to re-run all fuzzer-benchmark pairs, then you need to change this config. However, this is strongly discouraged due to the reasons above.

If you'd like to use a different version of a fuzzer, then please commit/merge the version into this PR. This protects your experiment from unexpected changes from other fuzzers, and ensures a clean and easy-to-access experiment environment/baseline for each report.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-15-fishfuzz --fuzzers fishpp_new fishpp_new_exp fishpp_new_fast aflplusplus

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-15-fishfuzz-bug --fuzzers fishpp_new_exp aflplusplus fishpp_new_fast fishpp_new --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[DonggeLiu]

Experiment 2023-12-15-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-15-fishfuzz-bug data and results will be available later at: The experiment data. The experiment report.

[DonggeLiu]

There seems to be a build failure with openh264_decoder_fuzzer in 2023-12-11-standard-cov. Checkout the build log from the experiment data URL I posted, there is no build log of this benchmark with any fuzzers.

BTW, I presume you have tested building benchmarks with your fuzzer locally?

[kdsjZh]

Thanks for the quick reply!

Is the config changed recently? As far as I understand, every command should run all the benchmarks for the fuzzers specified in the command. e.g., in the current scenario, fishpp-related fuzzers should be evaluated on 23 benchmarks. But now only 22 benchmarks are evaluated. (it's not because openh264 build or run failed, If it failed, at least I could find a build log, but now the build log didn't exist, e.g., 2023-12-14-libafl)

If a fuzzer-benchmark pair is not modified, there is no solid reason for re-generating its result

I totally agree, but the issue is that new tested fuzzers (e.g., fishpp or other aflpp/libafl variants) didn't have the openh264 results (not because it failed to build with openh264, the openh264 seems not included in the configure). Therefore when merging with the existing results, baseline fuzzers will have 23 benchmark's results while the new fuzzer only have 22. This issue seems to appear not only in my evaluation but also on other evaluation.

[kdsjZh]

Yes, I've tested it and in previous evaluation, this issue is not encountered.

And as far as I understood, if the benchmark build failed, it will at least have a build log in the data. But now non of them have this log (not only in fishpp, but also in latest aflpp and libafl's evaluation)

[kdsjZh]

e.g., in https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2023-12-14-libafl/build-logs/, the libafl is a standard setup, so the openh264 should be built correctly, but now the log didn't exists as well. and the current ongoing report didn't have the data for openh264 as well.

[DonggeLiu]

Is the config changed recently? As far as I understand, every command should run all the benchmarks for the fuzzers specified in the command.

Nope, this has always been the default config.

fishpp-related fuzzers should be evaluated on 23 benchmarks. But now only 22 benchmarks are evaluated. (it's not because openh264 build or run failed, If it failed, at least I could find a build log, but now the build log didn't exist, e.g., 2023-12-14-libafl)

This could be caused by a different issue: We modified the database a bit in past few days, which affected some experiments.fishfuzz might be one of them : ( The new experiment should be fine, as the change has been reverted.

If a fuzzer-benchmark pair is not modified, there is no solid reason for re-generating its result

I totally agree, but the issue is that new tested fuzzers (e.g., fishpp or other aflpp/libafl variants) didn't have the openh264 results (not because it failed to build with openh264, the openh264 seems not included in the configure). Therefore when merging with the existing results, baseline fuzzers will have 23 benchmark's results while the new fuzzer only have 22. This issue seems to appear not only in my evaluation but also on other evaluation.

Did it happen in the past or only in the last few days? If that happened repeatedly in the past, could you please share some examples with us? Much appreciated : ) (We are focusing on another project at the moment and might not be able to fix it right now, but will certainly add it to the TODO list). Thanks!

[kdsjZh]

Thanks for the clarification!

Did it happen in the past or only in the last few days?

I only notice it in recent evaluations, but I'm reading other reports and logs as well. I'll come back to you once there are more cases.

Thanks

[DonggeLiu]

Thanks for the clarification!

Did it happen in the past or only in the last few days?

I only notice it in recent evaluations, but I'm reading other reports and logs as well. I'll come back to you once there are more cases.

Thanks

Fantastic! Thanks for your help!

[kdsjZh]

Hi Dongge @Alan32Liu ,

I drafted the SBFT24 submission and want to tune a bit, could you help me run a test? Thanks!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[DonggeLiu]

e.g., in https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2023-12-14-libafl/build-logs/, the libafl is a standard setup, so the openh264 should be built correctly, but now the log didn't exists as well. and the current ongoing report didn't have the data for openh264 as well.

Fixing OpenH264 here. Once it passes all CIs, I will merge it into the master branch so that you can rebase your PR on it (or merge it into your PR). Feel free to ping me if I forget : )

[DonggeLiu]

Hi Dongge @Alan32Liu ,

I drafted the SBFT24 submission and want to tune a bit, could you help me run a test? Thanks!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Let's ask @phi-go if the competition allows requesting experiments to tune fuzzers now : )

[phi-go]

Hey, yes please feel free to tune fuzzers. However, the mutation measurer is still in progress. @Alan32Liu let me check with my co-chairs if we should also allow access to the mutation measurer results once that is possible. As it is already getting close to the deadline I would expect it to be fairer if no one gets access. Though, as this PR is publicly available these results could be gotten privately, so I'm not quite sure.

[kdsjZh]

Hi Dongge @Alan32Liu , I drafted the SBFT24 submission and want to tune a bit, could you help me run a test? Thanks!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-19-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Let's ask @phi-go if the competition allows requesting experiments to tune fuzzers now : )

Thanks for your help regarding the openh264's fix and inquiry. And Thanks for Philipp's quick reply!

For this tuning, I don't need mutation analysis and only want to check a bit the final results, so if possible, could you help me start the campaign to see the results? Thanks @Alan32Liu

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-26-tunefuzz --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-26-tunefuzz-bug --fuzzers tunefuzz tunefuzz_fast tunefuzz_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[DonggeLiu]

Experiment 2023-12-26-tunefuzz data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-26-tunefuzz-bug data and results will be available later at: The experiment data. The experiment report.

[kdsjZh]

Thanks for your help!

[kdsjZh]

Hi @vanhauser-thc ,

I reuse this PR given it's not closed yet. I would like to test both coverage and bug

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-01-24-aflpp-sk --fuzzers aflplusplus_sk aflplusplus_sk_near aflplusplus

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-01-24-aflpp-sk-bug --fuzzers aflplusplus_sk aflplusplus_sk_near aflplusplus --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb