search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.09k stars 266 forks source link

[DO NOT MERGE] Adding code for experiments using pubsub queues #2005

Open gustavogaldinoo opened 2 months ago

gustavogaldinoo commented 2 months ago

This PR adds code to use pub sub queues on the measurer when running cloud experiments, instead of python in-memory queues.

This PR is still a draft, I still want to do some improvements on top of it, but I appreciate any feedback.

Will try to trigger a cloud experiment before adding reviewers to it.

gustavogaldinoo commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name measurer-pub-sub-test-experiment --fuzzers afl libfuzzer

gustavogaldinoo commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pub-sub-test-experiment --fuzzers afl libfuzzer

gustavogaldinoo commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pub-sub-test-experiment --fuzzers afl libfuzzer

gustavogaldinoo commented 2 months ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pub-sub-test-experiment-2 --fuzzers afl libfuzzer

jonathanmetzman commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name metzman-2024 --fuzzers afl libfuzzer

jonathanmetzman commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name metzman-2024-2 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-1 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-2 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-3 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-4 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-5 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-6 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-7 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-8 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-10 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-11 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-12 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-14 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-15 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-16 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-17 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-18 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-19 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks sqlite3_ossfuzz bloaty_fuzz_target --experiment-name pubsub-measurer-20 --fuzzers afl libfuzzer

gustavogaldinoo commented 1 month ago

This PR is still a WIP.

In the experiments I've tried to run, it seems that we are having a problem in starting the measurer workers processes.

I couldn't debug to know the reason why this is happening.

As today is my last day at Google, am I afraid that I won't be able to debug any further and merge this PR so I am commenting to let you know that this is the current state of it.