search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.09k stars 266 forks source link

Disable incompatible benchmarks for cloud experiments #2030

Closed DonggeLiu closed 1 month ago

DonggeLiu commented 1 month ago

Temporarily disable benchmark stb_stbi_read_fuzzer and openh264_decoder_fuzzerfrom cloud experiments, becaue they are proven to be incompatible in cloud build/run environment.

@addisoncrump kindly confirmed that they work in local experiments.

DonggeLiu commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-16-2030-1 --fuzzers libfuzzer

DonggeLiu commented 1 month ago

Experiment 2024-08-16-2030-1 data and results will be available later at: The experiment data. The experiment report. The experiment report(experimental).

jonathanmetzman commented 1 month ago

Temporarily disable benchmark stb_stbi_read_fuzzer and openh264_decoder_fuzzerfrom cloud experiments, becaue they are proven to be incompatible in cloud build/run environment.

@addisoncrump kindly confirmed that they work in local experiments.

COuld you explain why they don't work?

DonggeLiu commented 1 month ago

COuld you explain why they don't work?

I did not have a chance to verify this, but it's likely due to llvm-cov failed to generate coverage summary json at here. Maybe outdated llvm in cloud VMs?

jonathanmetzman commented 1 month ago

COuld you explain why they don't work?

I did not have a chance to verify this, but it's likely due to llvm-cov failed to generate coverage summary json at here. Maybe outdated llvm in cloud VMs?

Eh then this should probably be fixed instead.

jonathanmetzman commented 1 month ago

I don't object if you don't have the time though

DonggeLiu commented 1 month ago

I don't object if you don't have the time though

I am a bit inclined to merge this temporary mitigation for now and fix it after we settle down the agent work, if you don't mind.

BTW, do we plan to update anything in FuzzBench recently?

oliverchang commented 1 month ago

merge

+1, let's mitigate this first, and track this as a bug for when we can get to it.

DonggeLiu commented 1 month ago

+1, let's mitigate this first, and track this as a bug for when we can get to it.

OK I've created an issue for this and will merge the PR once the experiment finishes without any error.