search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 269 forks source link

Corpus Pruning Algorithm Experiment #2002

Closed tokatoka closed 3 months ago

tokatoka commented 3 months ago

This PR tries new idea from https://mschloegel.me/paper/schiller2023fuzzerrestarts.pdf

I implemented a fuzzer that periodically reset the scorpus every 30/120 minutes after novelty was not found/enough time has passed.

tokatoka commented 3 months ago

@DonggeLiu Could you run the CI?

tokatoka commented 3 months ago

@DonggeLiu Ping

DonggeLiu commented 3 months ago

Done! I was on leave last week.

tokatoka commented 3 months ago

it looks like everytime i update it needs additional approval 😅 can you run it again?

DonggeLiu commented 3 months ago

it looks like everytime i update it needs additional approval 😅 can you run it again?

Do you happen to know any way to allow certain users (like you) to always be able to run CIs?

tokatoka commented 3 months ago

I think you can make me "Collaborator".

DonggeLiu commented 3 months ago

I think you can make me "Collaborator".

Oh we will have to discuss this with other owners of this repo. Is there a more lightweight alternative?

tokatoka commented 3 months ago

https://docs.github.com/en/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization#configuring-required-approval-for-workflows-from-public-forks

I think all the options are written here but it looks like there's no functionality to allow specific users to run CI

tokatoka commented 3 months ago

but it's strange because previously you didn't have to manually run it for me right?

DonggeLiu commented 3 months ago

but it's strange because previously you didn't have to manually run it for me right?

I am not sure, maybe I did.

tokatoka commented 3 months ago

i'm still debuggin it :)

tokatoka commented 3 months ago

i think i resolved the problem, could you run again?

jonathanmetzman commented 3 months ago

/gcbrun

jonathanmetzman commented 3 months ago

I've changed things so we shouldn't need to approve every time actions wants to run

tokatoka commented 3 months ago

thank you!

tokatoka commented 3 months ago

@DonggeLiu The CI looks good can we run the experiment? The command is 

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-14-libafl-pruner --fuzzers libafl libafl_latest libafl_r120_force_10 libafl_r120_force_50 libafl_r120_last_10 libafl_r120_last_50 libafl_r30_force_10 libafl_r30_force_50 libafl_r30_last_10 libafl_r30_last_50
DonggeLiu commented 3 months ago

Sure! We are still resolving the bottleneck in measurement so we cannot run too many fuzzers in one experiment. Ideally let's keep ~5 fuzzers in each. How would you like to group them?

tokatoka commented 3 months ago

ok

This is group A.

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-14-libafl-pruner --fuzzers libafl libafl_latest libafl_r120_force_10 libafl_r120_force_50 libafl_r120_last_10

This is group B

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-14-libafl-pruner --fuzzers libafl_r120_last_50 libafl_r30_force_10 libafl_r30_force_50 libafl_r30_last_10 libafl_r30_last_50
DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-02-libafl-pruner --fuzzers libafl libafl_latest libafl_r120_force_10 libafl_r120_force_50 libafl_r120_last_10

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-02-libafl-pruner --fuzzers libafl_r120_last_50 libafl_r30_force_10 libafl_r30_force_50 libafl_r30_last_10 libafl_r30_last_50

DonggeLiu commented 3 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-02-libafl-pruner-1 --fuzzers libafl_r120_last_50 libafl_r30_force_10 libafl_r30_force_50 libafl_r30_last_10 libafl_r30_last_50

tokatoka commented 3 months ago

it looks like it didn't run unfortunately

DanBlackwell commented 3 months ago

HI @tokatoka , not my PR so sorry to intrude; it looks like your experiment did start, as the experiment data was created and the logs indicate it's running here. I've had the same thing happen on the last 2 runs of my PR here; the coverage sub-directory in the data never gets created, even though the fuzzer is running.

I wonder if there's anything obvious in the logs? (I guess one of the FB team can see these?)

tokatoka commented 3 months ago

thanks for the info! it looks like all the experiment that began today is affected..

DonggeLiu commented 3 months ago

This is likely due to no space on device:

image

@gustavogaldinoo could you please look into this? Thanks! I've removed all running experiments since none of them produced any results.

DonggeLiu commented 3 months ago

Also noticed many Profdata files merging failed. and https://github.com/google/fuzzbench/pull/2011#issuecomment-2270197163 in the cloud log, which may block the experiment report generation. Related: https://github.com/google/fuzzbench/pull/2011#issuecomment-2270197163.

BTW, will this PR generate a large corpus? This may explain the tons of no space left on device errors.

tokatoka commented 3 months ago

BTW, will this PR generate a large corpus? This may explain the tons of no space left on device errors.

yes. i'm thinking about the fix for it now..

DanBlackwell commented 2 months ago

Any chance you ran this somewhere in the end? It would be interesting to see the results even if it's only a subset of the available benchmarks that don't use much storage (e.g. open_h264 looks bad for storage, as does proj4 and woff2)

tokatoka commented 2 months ago

no i didn't run this in the end