retry tests with current fb

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-10-aflpp2 --fuzzers aflplusplus_reg2 aflplusplus_reg4 aflplusplus_reg3 --benchmarks bloaty_fuzz_target libpcap_fuzz_both proj4_proj_crs_to_crs_fuzzer

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-11-aflpp --fuzzers aflplusplus_reg0 aflplusplus_reg1 aflplusplus_reg3 --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http lcms_cms_transform_fuzzer libpcap_fuzz_both mbedtls_fuzz_dtlsclient openthread_ot-ip6-send-fuzzer proj4_proj_crs_to_crs_fuzzer systemd_fuzz-link-parser

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-12-aflpp --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-22-aflpp --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-23-aflpp --fuzzers aflplusplus_aweight3 aflplusplus_aweight2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-23-aflpp2 --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-25-aflpp --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --oss-fuzz-corpus --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-28-aflpp --fuzzers aflplusplus_exold aflplusplus_exnew --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[vanhauser-thc]

/gcbrun run_experiment.py -a --oss-fuzz-corpus --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-30-aflpp --fuzzers aflplusplus_exold aflplusplus_exnew --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-01-aflpp --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-01-aflpp2 --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-19-aflpp --fuzzers aflplusplus aflplusplus_wp aflplusplus_wp2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-19-aflpp2 --fuzzers aflplusplus aflplusplus_wp aflplusplus_wp2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-20-aflpp --fuzzers aflplusplus aflplusplus_wp2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-21-aflpp --fuzzers aflplusplus aflplusplus_wp

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-23-aflpp --fuzzers aflplusplus libafl

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-23-aflpp2 --fuzzers aflplusplus libafl

[vanhauser-thc]

@DonggeLiu benchmarks are not starting anymore since the 22nd/23rd. be benchmarking binaries are build according to the build logs but nothing else. can you have a look please? thank you!

[DonggeLiu]

@DonggeLiu benchmarks are not starting anymore since the 22nd/23rd. be benchmarking binaries are build according to the build logs but nothing else. can you have a look please? thank you!

Sorry that I am occupied by other tasks and won't be able to look into this for awhile. Do you mean they do not run anymore in this PR? How about testing it in a new one? Thanks!

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-02-aflpp --fuzzers aflplusplus libafl

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-02-aflpp2 --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

@DonggeLiu I updated to the current fuzzbench state but still only benchmark measures are being built (checking with gsutil ls gs://fuzzbench-data/2024-08-02-aflpp/build-logs/ )

building locally works fine.

[DonggeLiu]

Thanks for reporting this, @vanhauser-thc. A similar issue has been reported by https://github.com/google/fuzzbench/pull/2000#issuecomment-2265045685. We can discuss more over there.

For now, I will delete all experiments since they are not producing any result.

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-04-aflpp --fuzzers aflplusplus aflplusplus_new

[DonggeLiu]

I suppose this is running as expected and not affected by issues in https://github.com/google/fuzzbench/pull/2002 and https://github.com/google/fuzzbench/pull/2000?

Experiment 2024-08-04-aflpp data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-aflpp --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-18-aflpp --fuzzers aflplusplus aflplusplus_new

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-19-aflpp --fuzzers aflplusplus_new libafl

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-02-aflpp --fuzzers aflplusplus wingfuzz

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-03-aflpp --fuzzers aflplusplus libafl_fuzz

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-04-aflpp --fuzzers libafl_fuzz --benchmarks re2_fuzzer

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-05-aflpp --fuzzers aflplusplus libafl_fuzz

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-09-aflpp --fuzzers aflplusplus_nocmplog libafl_fuzz

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-19-aflpp --fuzzers aflplusplus_nocmplog libafl_fuzz

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-20-aflpp --fuzzers aflplusplus_nocmplog aflplusplus_vp0 aflplusplus_vp1

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-20-aflpp2 --fuzzers aflplusplus_nocmplog aflplusplus_vp0 aflplusplus_vp1

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-20-aflpp3 --fuzzers aflplusplus_nocmplog aflplusplus_vp0 aflplusplus_vp1

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-21-aflpp --fuzzers aflplusplus_nocmplog aflplusplus_vp0 aflplusplus_vp1

[vanhauser-thc]

@DonggeLiu the last 3 experiments are not starting. There were built successfully say the logs though and I updated to the newest commit. Do you know why?

[vanhauser-thc]

@DonggeLiu I implemented value profile for afl++ (without queue pollution, but currently less solving mechanisms, want to test it first) - and would like to test it ... :)

[DonggeLiu]

Thanks for pinging me on this: I wanted to take look earlier but was busy with other things...

Let me try triggering the exp again in case it is flaky, will look into the logs if that fails.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-24-aflpp --fuzzers aflplusplus_nocmplog aflplusplus_vp0 aflplusplus_vp1

[DonggeLiu]

Hmm, I think 2024-09-21-aflpp works fine? https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2024-09-21-aflpp/index.html

The log console also shows it terminated successfully:

(Taking a wild guess) Maybe you checked the non-experimental link? IIUC, FuzzBench uses the experimental link for non-core fuzzers (e.g., the ones above) and non-experimental link for core fuzzers (afl++, libfuzzer, etc.) The difference is the experimental/ subdir in the link.

[vanhauser-thc]

Weird now they are all there, but they weren’t on the weekend

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-09-25-aflpp --fuzzers aflplusplus_nocmplog aflplusplus_vp0 aflplusplus_vp1

[vanhauser-thc]

@DonggeLiu OK I see the same thing again - the report is not showing up. However I see that it is running: https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-09-25-aflpp/experiment-folders/

but it is neither here https://www.fuzzbench.com/reports/experimental/index.html nor there https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2024-09-25-aflpp/index.html

My guess is it will only be visible once the experiment is completed (which is why it showed up days later and are visible now, but not the days after I started them). This is a new thing.

[DonggeLiu]

but it is neither here https://www.fuzzbench.com/reports/experimental/index.html nor there https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2024-09-25-aflpp/index.html

Is this using a lot of space? I saw messages like Retrying on experiment.runner.TrialRunner.archive_and_save_corpus failed with [Errno 28] No space left on device. Raise.:

My guess is it will only be visible once the experiment is completed (which is why it showed up days later and are visible now, but not the days after I started them). This is a new thing.

I don't think we have changed the report upload logic. It should be the same as before.

[vanhauser-thc]

No, nothing does create a lot of data, not more than usual. but you see that it is running, and new report is coming up?

[vanhauser-thc]

@DonggeLiu now that the test is completed the report is up … and two fuzzers are missing, and they produced coverage so they should be there.

Also the results look suspect - no cmplog is better than with cmplog? No …

[DonggeLiu]

Also the results look suspect - no cmplog is better than with cmplog? No …

I don't know the internal of AFL++ as well as you so I have no clue why that is better.

and two fuzzers are missing, and they produced coverage so they should be there.

Which 2 are missing? I can search for their logs