search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 269 forks source link

Debug PR #2028 #2039

Closed DonggeLiu closed 2 months ago

DonggeLiu commented 2 months ago

Debug two new errors observed in #2028 with the logs from #2038.

Error 1: cloud build timeout

Error 2: Fuzz target binary not found.

DonggeLiu commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-25-2039-libafl --fuzzers libafl

DonggeLiu commented 2 months ago

Experiment 2024-08-25-2039-libafl data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

DonggeLiu commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-25-2039-libafl-1 --fuzzers libafl

DonggeLiu commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-25-2039-bases-1 --fuzzers afl aflplusplus libfuzzer

DonggeLiu commented 2 months ago

Experiment 2024-08-25-2039-libafl-1 data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

Experiment 2024-08-25-2039-bases-1 data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

addisoncrump commented 2 months ago

This is starting to look like a Heisenbug...

DonggeLiu commented 2 months ago

This is starting to look like a Heisenbug...

LOL, let's merge your PR if these two work fine. We can always revert if the bug reappears.

I will merge my PR to add more logs so that we can better observe the bug once it reoccurs.

tokatoka commented 2 months ago

For the 1st one some of them stopped at 15 minutes For the 2nd one some of them stopped at 6h so it was not really working

tokatoka commented 2 months ago

Now it is showing the final report. But for the 2nd one in the final report, None of {afl aflplusplus libfuzzer} is listed in the report, despite they are the fuzzers being tested

tokatoka commented 2 months ago

and as i said, for the 2nd report, https://storage.googleapis.com/www.fuzzbench.com/reports/2024-08-25-2039-bases-1/data.csv.gz the data.csv.gz does contain the data for afl, aflplusplus, and libfuzzer. but all their experiment are terminated after 6 hours (= 21600 seconds)

for the 1st one, too. at least for this one, we have complete result for most targets on libafl except for the 5 missing NaN.

tokatoka commented 2 months ago

although I said "terminated after 6 hours". but this is not really correct because in the experiment-folder https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-08-25-2039-bases-1/experiment-folders/curl_curl_fuzzer_http-afl/trial-3106096/corpus/

here it does have corpus for the last 23 hours. So in the end the fuzzer did run successfully!. but the problem is just about the measurement not working

DonggeLiu commented 2 months ago

Thanks for tracking the experiment progress, @tokatoka It's perfectly normal for the experiment report to pause for a few hours:

  1. The experiment trial VM might be preempted. Hence, that trial has to be re-run.
  2. The measurer might take a while to produce results.

As a result, we can expect a 1-day experiment report to take up to 3 days : )