Prescientfuzz testing

[DanBlackwell]

Hi, I have a new fuzzer based on LibAFL that I would like to integrate. I'd like to be able to run an experiment to compare it with the other fuzzers, but the documented approach (adding to https://github.com/google/fuzzbench/blob/master/service/experiment-requests.yaml) doesn't seem to be used lately - is there some automatic experiment that runs periodically?

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[DonggeLiu]

Hi, I have a new fuzzer based on LibAFL that I would like to integrate. I'd like to be able to run an experiment to compare it with the other fuzzers, but the documented approach (adding to https://github.com/google/fuzzbench/blob/master/service/experiment-requests.yaml) doesn't seem to be used lately - is there some automatic experiment that runs periodically?

Thanks for submitting a PR, @DanBlackwell! This makes our work a lot easier : ) Here is a guide on how to enable PR experiments: https://github.com/google/fuzzbench/pull/1967#issuecomment-2071143515, hope that helps!

Once it is ready, we can use the /gcbrun commands to run experiments and show results on this PR directly, without having to wait for another day before experiment-requests.yaml triggers a new experiment.

[DanBlackwell]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-10_prescientfuzz_init --fuzzers libafl aflplusplus prescientfuzz honggfuzz libfuzzer

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-10-prescientfuzz_init --fuzzers prescientfuzz

[DanBlackwell]

@DonggeLiu Has this failed to build? I can't see anything in that CI log

[DonggeLiu]

Experiment 2024-05-14-prescientfuzz-init data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[DonggeLiu]

@DonggeLiu Has this failed to build? I can't see anything in that CI log

Yes, I failed to notice that the experiment name does not match this pattern: "^[a-z0-9-]{0,30}$". Let me restart one named 2024-05-14-prescientfuzz-init now. The data & report will be available in the links above later.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-14-prescientfuzz-init --fuzzers prescientfuzz

[DanBlackwell]

Ok, still some dying from memory starvation. I think I have it fixed now; any chance you could rerun that exact setup for me @DonggeLiu ?

Oh, Is there any caching in the docker setup? I've only updated the fuzzer source repo, so if docker caches the build images it probably won't fetch the updated version.

[DonggeLiu]

Oh, Is there any caching in the docker setup? I've only updated the fuzzer source repo, so if docker caches the build images it probably won't fetch the updated version.

I vaguely recall that this has caused problems before. Could you please modify the dockerfile just in case? Thanks!

I am happy to re-run the experiment when you are ready, please feel free to ping me.

[DanBlackwell]

Ok, have manually specified the commit number which should trash the cache. All ready to go!

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-16-prescientfuzz-init --fuzzers prescientfuzz

[DonggeLiu]

Experiment 2024-05-16-prescientfuzz-init data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[DanBlackwell]

I forgot that it needs git fetch before checking out sorry... Any chance you can restart that @DonggeLiu ?

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-17-prescientfuzz-init --fuzzers prescientfuzz

[DonggeLiu]

Experiment 2024-05-17-prescientfuzz-init data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[DanBlackwell]

Hi @DonggeLiu ; any chance you can restart it? Just patched another bug sorry.

[DonggeLiu]

Hi @DonggeLiu ; any chance you can restart it? Just patched another bug sorry.

Sure! I've terminated all instances of the previous experiment and approved the CIs. Before we start another experiment, would you mind checking if there is any CI error? I will start the experiment if they behave as expected : )

[DanBlackwell]

The CI looks ok to me, and I ran one of the previously failing benchmarks through the debug-builder earlier. I'm hoping this run should have everything working finally; I appreciate your patience! (I'm trying to build a global CFG without the LTO pass - which has been tricky for me)

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-17-prescientfuzz-ini --fuzzers prescientfuzz

[DonggeLiu]

The experiment CI says failed, but the experiment instance and the data directory has been created, so I reckon we are safe.

Experiment 2024-05-17-prescientfuzz-ini data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[DanBlackwell]

Hi @DonggeLiu , finally I have everything building and running; am I allowed to run say 5 instances to test different parameter setups? I'm thinking to add each setup as a different 'fuzzer' (in ./fuzzers) and then they can all run in one experiment. Do let me know if there's a better approach.

[DanBlackwell]

Also, I wanted to generate that report just for PrescientFuzz vs LibAFL (as the graphs are hard to read with so many fuzzers); I tried doing the following but got an error:

(.venv) ➜  fuzzbench git:(prescientfuzz_initial_integration) ✗ PYTHONPATH=. python3 analysis/generate_report.py PrescientFuzz --report-dir PrescientFuzzReport --fuzzers prescientfuzz libafl --from-cached-data
INFO:root:Reading experiment data from PrescientFuzzReport/data.csv.gz.
/home/dan/Documents/fuzzbench/analysis/generate_report.py:139: DtypeWarning: Columns (1) have mixed types. Specify dtype option on import or set low_memory=False.
  experiment_df = pd.read_csv(data_path)
INFO:root:Done reading data from PrescientFuzzReport/data.csv.gz.
WARNING:root:Filtered out invalid benchmarks: set().
INFO:root:Rendering HTML report.
/home/dan/Documents/fuzzbench/analysis/plotting.py:485: OrangeDeprecationWarning: compute_CD is deprecated and will be removed in Orange 3.34.
  critical_difference = Orange.evaluation.compute_CD(
/home/dan/Documents/fuzzbench/analysis/plotting.py:488: OrangeDeprecationWarning: graph_ranks is deprecated and will be removed in Orange 3.34.
  Orange.evaluation.graph_ranks(average_ranks.values, average_ranks.index,
/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/jinja2/runtime.py:298: FutureWarning: this method is deprecated in favour of `Styler.to_html()`
  return __obj(*args, **kwargs)
Traceback (most recent call last):
  File "/home/dan/Documents/fuzzbench/analysis/generate_report.py", line 293, in <module>
    sys.exit(main())
  File "/home/dan/Documents/fuzzbench/analysis/generate_report.py", line 277, in main
    generate_report(experiment_names=args.experiments,
  File "/home/dan/Documents/fuzzbench/analysis/generate_report.py", line 261, in generate_report
    detailed_report = rendering.render_report(experiment_ctx, template,
  File "/home/dan/Documents/fuzzbench/analysis/rendering.py", line 46, in render_report
    return template.render(experiment=experiment_results,
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/home/dan/Documents/fuzzbench/analysis/report_templates/default.html", line 143, in top-level template code
    {{ experiment.relative_code_summary_table.render() }}
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/io/formats/style.py", line 344, in render
    return self._render_html(sparse_index, sparse_columns, **kwargs)
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/io/formats/style_render.py", line 162, in _render_html
    self._compute()
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/io/formats/style_render.py", line 205, in _compute
    r = func(self)(*args, **kwargs)
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/io/formats/style.py", line 1444, in _apply
    result = data.T.apply(func, axis=0, **kwargs).T  # see GH 42005
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/core/frame.py", line 8848, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/core/apply.py", line 733, in apply
    return self.apply_standard()
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/core/apply.py", line 857, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/core/apply.py", line 873, in apply_series_generator
    results[i] = self.f(v)
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/core/apply.py", line 138, in f
    return func(x, *args, **kwargs)
  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/io/formats/style.py", line 3560, in _background_gradient
    rgbas = plt.cm.get_cmap(cmap)(norm(gmap))
AttributeError: module 'matplotlib.cm' has no attribute 'get_cmap'

I've tried searching, but I'm a bit stumped as to how it's possible for this to happen; although I am not particular experienced with pip / python so maybe matplotlib is just not installed properly?

[DonggeLiu]

am I allowed to run say 5 instances to test different parameter setups?

Yep sure, this requires changing this value to 5.

I'm thinking to add each setup as a different 'fuzzer' (in ./fuzzers) and then they can all run in one experiment. Do let me know if there's a better approach.

Yep this is the simplest way. Unfortunately there is no better approach for now.

[DonggeLiu]

AttributeError: module 'matplotlib.cm' has no attribute 'get_cmap'

I reckon this is likely due to a mismatch version of matplotlib, which does not have get_cmap. I did a quick experiment and found at least this version works:

Unfortunately we did not document the exact version used in FuzzBench.

[DanBlackwell]

Hopefully running the following should get all 4 up together:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-20-prescientfuzz-tuning --fuzzers prescientfuzz_no_backoff prescientfuzz_0_999_backoff prescientfuzz_0_9999_backoff prescientfuzz_0_99999_backoff

I'm guessing you might have to tweak something so that it doesn't merge with the other experiments and leave the graphs too messy?

[DanBlackwell]

AttributeError: module 'matplotlib.cm' has no attribute 'get_cmap'

I reckon this is likely due to a mismatch version of matplotlib, which does not have get_cmap. I did a quick experiment and found at least this version works:

Unfortunately we did not document the exact version used in FuzzBench.

I fixed it locally; get_cmap is still in matplotlib v3, it just seems that pandas was namespacing incorrectly. Here's my fix in case anyone else comes across the same thing through search:

Replace the bad line at the bottom of the callstack, here style.py:3560:

  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/io/formats/style.py", line 3560, in _background_gradient
    rgbas = plt.cm.get_cmap(cmap)(norm(gmap))

Remove the .cm, so:

-     rgbas = plt.cm.get_cmap(cmap)(norm(gmap))
+     rgbas = plt.get_cmap(cmap)(norm(gmap))

[DonggeLiu]

Hopefully running the following should get all 4 up together: I'm guessing you might have to tweak something so that it doesn't merge with the other experiments and leave the graphs too messy?

Yep, if you want to compare these 4 only (i.e., no other fuzzers in the report), please set this value to false.

Do you still want to run 5 instances for each fuzzer/setup? I am happy either way : )

am I allowed to run say 5 instances to test different parameter setups?

Yep sure, this requires changing this value to 5.

[DonggeLiu]

Replace the bad line at the bottom of the callstack, here style.py:3560:

  File "/home/dan/Documents/fuzzbench/.venv/lib/python3.10/site-packages/pandas/io/formats/style.py", line 3560, in _background_gradient
    rgbas = plt.cm.get_cmap(cmap)(norm(gmap))

Thanks, @DanBlackwell! Let me add your solution to the issue.

[DanBlackwell]

Ok, I've set it to not merge; and 10 trials per fuzzer. I think the command should be: /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-20-prescientfuzz-tuning --fuzzers prescientfuzz_no_filter prescientfuzz_no_backoff prescientfuzz_0_999_backoff prescientfuzz_0_9999_backoff prescientfuzz_0_99999_backoff

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-prescientfuzz-tune --fuzzers prescientfuzz_no_filter prescientfuzz_no_backoff prescientfuzz_0_999_backoff prescientfuzz_0_9999_backoff prescientfuzz_0_99999_backoff

[DonggeLiu]

Experiment 2024-05-22-prescientfuzz-tune data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

I shortened the name because experiment names must be within 30 chars.

[DanBlackwell]

Hi @DonggeLiu ; good news is I think that I finally managed to find a combination of llvm flags that has got LTO working! I also have some other setups to trial; this time the command should be:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-prescientfuzz-weit --fuzzers prescientfuzz prescientfuzz_nn prescientfuzz_depth_squared prescientfuzz_depth_const_1

Do I need to wait for the previous trial to finish before I can run? (Note that I still want that one to run to the end, not be cancelled)

[DonggeLiu]

Do I need to wait for the previous trial to finish before I can run? (Note that I still want that one to run to the end, not be cancelled)

Nope, we can run them in parallel.

BTW, would you please use triple-backtick (```) to quote the command next time? I always worry that I might miss something when manually copying text across multiple lines. With triple-backtick, I can use the copy bottom : ) Thanks!

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-22-prescientfuzz-weit --fuzzers prescientfuzz prescientfuzz_nn prescientfuzz_depth_squared prescientfuzz_depth_const_1

[DonggeLiu]

Experiment 2024-05-22-prescientfuzz-weit data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[DanBlackwell]

Hi @DonggeLiu , any chance you could run this new setup for me? Same command again:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-23-prescientfuzz-weit --fuzzers prescientfuzz prescientfuzz_nn prescientfuzz_depth_squared prescientfuzz_depth_const_1

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-23-prescientfuzz-weit --fuzzers prescientfuzz prescientfuzz_nn prescientfuzz_depth_squared prescientfuzz_depth_const_1

[DanBlackwell]

Managed to change git checkout X into git Y when updating the commit hash (missing the checkout) there; hence it failed to build one of the setups. Have corrected that, but the original one can run being as it's just the control setup that failed to build (and I already have results for that).

[DanBlackwell]

Hi @DonggeLiu , did it fail to start because the CI build failed?

[DonggeLiu]

Hi @DonggeLiu , did it fail to start because the CI build failed?

Hi @DanBlackwell , the experiment started successfully. Experiment 2024-05-23-prescientfuzz-weit data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

I was distracted by other tasks and did not paste this. To avoid blocking you in case I forgot again: The links always share the same structure: Data: https://storage.googleapis.com/fuzzbench-data/index.html?prefix=<experiment-name>/ Report: https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/<experiment-name>/index.html

[DanBlackwell]

Ah, thanks for that! I thought it was an auto-generated message, didn't realise you were posting it manually sorry

[DanBlackwell]

Hi @DonggeLiu , I realised that libafl made a decent amount of progress compared to the version that PrescientFuzz was built off of, so I've updated it and want to see how they all compare. Any chance you can run the following when you're next working?

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-24-prescientfuzz --fuzzers libafl prescientfuzz libafl_19f5081 prescientfuzz_19f5081

PS I've set it back to merge with the rest of the fuzzer results, hopefully that's ok

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-27-prescientfuzz --fuzzers libafl prescientfuzz libafl_19f5081 prescientfuzz_19f5081

[DonggeLiu]

Experiment 2024-05-27-prescientfuzz data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[DanBlackwell]

Thanks for that! It seems that the updates to libafl behave a little differently, any chance you could run the following so that I can check some stuff @DonggeLiu :

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-03-prescientfuzz --fuzzers prescientfuzz_direct_neighbours prescientfuzz_reachable prescientfuzz_reachable_rarity prescientfuzz_reachable_rarity_depth prescientfuzz_reachable_rarity_depth_backoff prescientfuzz_all_no_mopt

[DonggeLiu]

Experiment 2024-06-03-prescientfuzz data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-03-prescientfuzz --fuzzers prescientfuzz_direct_neighbours prescientfuzz_reachable prescientfuzz_reachable_rarity prescientfuzz_reachable_rarity_depth prescientfuzz_reachable_rarity_depth_backoff prescientfuzz_all_no_mopt

[DanBlackwell]

Hi @DonggeLiu , just figured out that maybe my technique doesn't play well with MOpt (rather than random mutations); can you run the following for me?

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-06-prescientfuzz --fuzzers prescientfuzz_all prescientfuzz_all_no_mopt prescientfuzz_direct_neighbours prescientfuzz_direct_neighbours_no_mopt prescientfuzz_direct_neighbours_rarity prescientfuzz_direct_neighbours_rarity_no_mopt

[DonggeLiu]

Experiment 2024-06-06-prescientfuzz data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).