test new afl++ variant

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-08-31-aflpp --fuzzers aflplusplus aflplusplus_pendfav

[vanhauser-thc]

@Alan32Liu the experiment is not visible in gs://fuzzbench-data/ ... did I do something wrong or is it not working at the moment?

[DonggeLiu]

I think you might have forgotten to edit service/gcbrun_experiment.py, which allows launching experiments in PRs : ) Recall your previous example to add a dummy comment.

[vanhauser-thc]

right ... damn I do this too rarely :)

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-08-31-aflpp --fuzzers aflplusplus aflplusplus_pendfav

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-02-aflpp --fuzzers aflplusplus_frida aflplusplus_qemu aflplusplus_frida_perf aflplusplus_qemu_tcgcov

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-03-aflpp --fuzzers aflplusplus aflplusplus_pendfav

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-04-aflpp --fuzzers aflplusplus aflplusplus_pendfav

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-04-aflpp2 --fuzzers aflplusplus aflplusplus_pendfav

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-06-aflpp --fuzzers aflplusplus aflplusplus_pendfav

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-06-aflpp2 --fuzzers aflplusplus aflplusplus_pendfav

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-12-aflpp --fuzzers aflplusplus aflplusplus_o0 aflplusplus_o1 aflplusplus_o2

[jonathanmetzman]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-12-aflpp --fuzzers aflplusplus aflplusplus_o0 aflplusplus_o1 aflplusplus_o2

Can you run these experiments?

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name test-trigger-by-quote --fuzzers aflplusplus

Testing why quoting the gcb command still triggers experiments. Ideally, we only want to trigger exps if gcb is in plain text to avoid repeatedly triggering them when quoting them.

[DonggeLiu]

Just a sanity check: A normal comment will not trigger experiments.

[DonggeLiu]

How about this /gcbrun run_experiment.py --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name test-trigger-in-text ?

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-ff --fuzzers aflplusplus_ff_comp aflplusplus_fishfuzz

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-28-ff --fuzzers aflplusplus_ff_comp aflplusplus_fishfuzz aflplusplus_ff_comp2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-06 --fuzzers aflplusplus aflplusplus_early

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-06 --fuzzers aflplusplus_fishfuzz aflplusplus_ff_comp3

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-07-aflpp --fuzzers aflplusplus aflplusplus_early

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-08-aflpp --fuzzers aflplusplus_llvm17 aflplusplus_early

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-08-aflpp2 --fuzzers aflplusplus_llvm17 aflplusplus_early

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-08-aflpp-ff --fuzzers aflplusplus_ff_comp3 aflplusplus_fishfuzz

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-09-aflpp-nocmplog --fuzzers aflplusplus_llvm17 aflplusplus_early

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-10-aflpp-nocmplog --fuzzers aflplusplus_llvm17 aflplusplus_early

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-23-aflpp --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-23-aflpp2 --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-29-fish --fuzzers aflplusplus_fishfuzz aflplusplus_ff_comp3 --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-29-aflpp --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-01-aflpp --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-05-aflpp --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-05-aflpp --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-07-aflpp --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-08-aflpp --fuzzers aflplusplus aflplusplus_nou8

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-10-aflpp --fuzzers aflplusplus aflplusplus_explore aflplusplus_noclassify

[vanhauser-thc]

@Alan32Liu no experiments are currently going through it seems:

gs://fuzzbench-data/2023-12-07-standard/
gs://fuzzbench-data/2023-12-08-aflpp/
gs://fuzzbench-data/2023-12-08-standard-bug/
gs://fuzzbench-data/2023-12-08-standard-cov/
gs://fuzzbench-data/2023-12-10-aflpp/

but none of them are visible at https://www.fuzzbench.com/reports/experimental/index.html so there seems to be an issue with running the experiments.

[DonggeLiu]

@Alan32Liu no experiments are currently going through it seems:

Thanks for the report, @vanhauser-thc !

Could you please try again with the latest commit from Jonathan (dc03c08100933a3335cf95c6575f651c8dbd50c8)? We believe that should have fixed the problem. Actually, I am not that confident that it will work now : (

Sorry about the inconvenience : )

[vanhauser-thc]

can you please ping me when you know it is working again? also with the information if I can use this older state or if I need to restart from current state. thank you :)

[kdsjZh]

Looks like it works, the latest campaign's report is available at https://fuzzbench.com/reports/experimental/2023-12-10-aflpp/index.html. @vanhauser-thc

[DonggeLiu]

Looks like it works, the latest campaign's report is available at https://fuzzbench.com/reports/experimental/2023-12-10-aflpp/index.html. @vanhauser-thc

Yep!

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-12-aflpp --fuzzers aflplusplus aflplusplus_cxx

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-13-aflpp --fuzzers aflplusplus aflplusplus_cxx

[vanhauser-thc]

@Alan32Liu there still seem to be issues, the last run stopped, the index.html shows no update for the last 12 hours ... I restarted it but I think this is something that should be inspected.

[DonggeLiu]

@Alan32Liu there still seem to be issues, the last run stopped, the index.html shows no update for the last 12 hours ... I restarted it but I think this is something that should be inspected.

Not sure what 2023-12-12-aflpp looks like earlier, but it seems we have 10+ fuzzer instances successfully completed 23h-exp for all benchmarks?

We don't have all fuzzer instances, but I presume they were re-running now (hence the subtitle says experiment incomplete/still running...). Recall that we use preemptible instances so it's possible that some instances were preempted and restarted. We cannot see their progress in the plots until they finish, because the existing results in the plot are already at 23h.

I also double-checked the logs and can confirm some instances of 2023-12-12-aflpp is running at this moment.

[vanhauser-thc]

yes it hung for about 18 hours or so, then progressed again. weird.

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-15-aflpp --fuzzers aflplusplus_408 aflplusplus_409