google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 270 forks source link

Only trigger experiments when a comment starts with `/gcbrun` #1897

Closed DonggeLiu closed 1 year ago

DonggeLiu commented 1 year ago

Currently, a GCB experiment will be triggered if our comment mentions /gcbrun.

This is problematic because experiments will be launched unintendedly (e.g., when we quote previous /gcb commands, or even when we mention /gcb in any form).

Root cause

  1. GCB trigger simply checks for /gcb without any constraints.
  2. Our gcbrun_experiment.py iterates all previous comments to find the latest valid /gcb command, which might have already been run.

Proofs

  1. This example quotes an old command and modifies its experiment name. However, it triggers an experiment with the old name, indicating we launch the latest valid experiment command.
  2. This example includes a random /gcb command in the middle of the sentence. It also triggers the same old experiment, confirming the same conclusion.

Propose fix

Change https://github.com/google/fuzzbench/blob/56caa83e81bc59a1389367c6bd29d46fd35d03e6/service/gcbrun_experiment.py#L56-L57 to

 if not body.startswith(RUN_EXPERIMENT_COMMAND_STR): 
     return None 

Justification

  1. Based on our setting on Google Cloud, all /gcb commands will immediately trigger an experiment, there is no need to search back in history for an old command.
  2. If we need to run an old command, we can always add a new comment for it.

@jonathanmetzman: Did I miss anything? If it looks good to you, I will implement the fix proposed above.

jonathanmetzman commented 1 year ago

you mean [/]gcbrun run right?

jonathanmetzman commented 1 year ago

I think this is a good idea.

DonggeLiu commented 1 year ago

you mean [/]gcbrun run right?

Yep, fixed.