search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.09k stars 266 forks source link

Adding gfuzz and requesting an experiment #1980

Closed vwrewsge closed 4 months ago

vwrewsge commented 4 months ago

Hi, we want to test gfuzz, a variant of AFL++. We have tested all the target benchmarks, and all of them ran successfully. We'd like to run only 2 iterations first to see if our strategy is effective (already set in service/experiment-config.yaml), to avoid wasting resources. Could you please help me run the experiment? @DonggeLiu Thanks a lot! :)

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-08-aflpp --fuzzers aflplusplus gfuzz libafl --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpcap_fuzz_both libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

The default value of trials in service/experiment-config.yaml is 20. Could you please set it to 2?

DonggeLiu commented 4 months ago

Hi @vwrewsge,

Here is a brief guide on how to enable PR experiments. Could you please make the change in step 1 and revert the change in service/experiment-config.yaml?

The default value of trials in service/experiment-config.yaml is 20. Could you please set it to 2?

That can be done by modifying the experiment-config.yaml, which corresponds to --experiment-config /opt/fuzzbench/service/experiment-config.yaml) used in your command.

Could you please help me run the experiment? @DonggeLiu Thanks a lot! :)

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-08-aflpp --fuzzers aflplusplus gfuzz libafl --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpcap_fuzz_both libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

Sure, thanks for providing the command. We can run this once you enable PR experiments and update the number of instances. If there happen to be any CI failures later, please consider fixing them, too.

BTW, you can omit the --benchmark flag if you'd like to run your fuzzer on all coverage-based benchmarks, they are the default value : ) Similarly, the fuzzer command can be --fuzzers gfuzz, because FuzzBench will merge the previous results of core-fuzzers into your report as baselines, and we don't have to re-run aflplusplus and libafl in all experiments.

Also, please correct me if I am wrong, but I presume that changing this file is unnecessary: analysis/test_data/pairwise_unique_coverage_heatmap-failed-diff.png

vwrewsge commented 4 months ago

Thank you very much for your response!

I have made a modification to service/gcbrun_experiment.py and reverted the change in service/experiment-config.yaml. If I want to run only 2 trials, how should I set it? It seems that service/experiment-config.yaml does not allow us to set this.

Regarding the --benchmark flag, our approach only applies to certain programs, so I have excluded those that are not applicable. Let me know if there is anything else I need to do.

Thanks!

DonggeLiu commented 4 months ago

I have made a modification to service/gcbrun_experiment.py and reverted the change in service/experiment-config.yaml. If I want to run only 2 trials, how should I set it? It seems that service/experiment-config.yaml does not allow us to set this.

Here: https://github.com/google/fuzzbench/blob/master/service/experiment-config.yaml#L5 Change this 20 to 2 will run 2 trials only. Thanks!

vwrewsge commented 4 months ago

I have made a modification to service/gcbrun_experiment.py and reverted the change in service/experiment-config.yaml. If I want to run only 2 trials, how should I set it? It seems that service/experiment-config.yaml does not allow us to set this.

Here: https://github.com/google/fuzzbench/blob/master/service/experiment-config.yaml#L5 Change this 20 to 2 will run 2 trials only. Thanks!

Thanks a lot!!

vwrewsge commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-10-aflpp --fuzzers gfuzz --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpcap_fuzz_both libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

DonggeLiu commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-10-aflpp --fuzzers gfuzz --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpcap_fuzz_both libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

DonggeLiu commented 4 months ago

Experiment 2024-05-10-aflpp data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

DonggeLiu commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-10-aflpp --fuzzers gfuzz --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpcap_fuzz_both libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

vwrewsge commented 4 months ago

Hi, @DonggeLiu

May I ask if the container built by Fuzzbench can connect to the internet? Our method may automatically pip installs some libraries during fuzzing.

Thanks a lot!

DonggeLiu commented 4 months ago

May I ask if the container built by Fuzzbench can connect to the internet? Our method may automatically pip installs some libraries during fuzzing.

I don't think they do:

image

@jonathanmetzman Please correct me if I am wrong. https://pantheon.corp.google.com/compute/instancesDetail/zones/us-central1-c/instances/r-2024-05-10-aflpp-2881365?project=fuzzbench

vwrewsge commented 4 months ago

I don't think they do: image

@jonathanmetzman Please correct me if I am wrong. https://pantheon.corp.google.com/compute/instancesDetail/zones/us-central1-c/instances/r-2024-05-10-aflpp-2881365?project=fuzzbench

May I ask if there are plans to allow containers to access the network in the future? Without network connectivity, the latest tools combining LLM with Fuzzing won't be able to undergo testing on Fuzzbench.

vwrewsge commented 4 months ago

Hi, @DonggeLiu

I've updated my tools. Could you please help me run the experiment?

vwrewsge commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-12-aflpp --fuzzers gfuzz_1 gfuzz_2 gfuzz_3 --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

DonggeLiu commented 4 months ago

Experiment 2024-05-12-aflpp data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

DonggeLiu commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-12-aflpp --fuzzers gfuzz_1 gfuzz_2 gfuzz_3 --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

vwrewsge commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-12-aflpp --fuzzers gfuzz_1 gfuzz_2 gfuzz_3 --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

Hi, @DonggeLiu

It seems the experiment didn't run successfully. Could you help me restart it?

vwrewsge commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-12-aflpp --fuzzers gfuzz_1 gfuzz_2 gfuzz_3 --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

DonggeLiu commented 4 months ago

Weird, for some reason, it says Experiment not requested.. I will restart it below.

DonggeLiu commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-14-aflpp --fuzzers gfuzz_1 gfuzz_2 gfuzz_3 --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

vwrewsge commented 4 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-14-aflpp --fuzzers gfuzz_1 gfuzz_2 gfuzz_3 --benchmarks bloaty_fuzz_target freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpng_libpng_read_fuzzer openssl_x509 vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

It seems like it still hasn't successfully run. Do I need to open a new pull request to see if it will work?

DonggeLiu commented 4 months ago

This time the error message is different: image

Instead of opening a new PR, could you please re-check if this works locally? E.g., a tiny 30 minute local experiment with your fuzzer and some benchmarks.

It appears to be a dependency error.

vwrewsge commented 4 months ago

This time the error message is different: image

Instead of opening a new PR, could you please re-check if this works locally? E.g., a tiny 30 minute local experiment with your fuzzer and some benchmarks.

It appears to be a dependency error.

thanks!!