Currently, a GCB experiment will be triggered if our comment mentions/gcbrun.
This is problematic because experiments will be launched unintendedly (e.g., when we quote previous /gcb commands, or even when we mention /gcb in any form).
Our gcbrun_experiment.pyiterates all previous comments to find the latest valid /gcb command, which might have already been run.
Proofs
This example quotes an old command and modifies its experiment name. However, it triggers an experiment with the old name, indicating we launch the latest valid experiment command.
This example includes a random /gcb command in the middle of the sentence. It also triggers the same old experiment, confirming the same conclusion.
if not body.startswith(RUN_EXPERIMENT_COMMAND_STR):
return None
Justification
Based on our setting on Google Cloud, all /gcb commands will immediately trigger an experiment, there is no need to search back in history for an old command.
If we need to run an old command, we can always add a new comment for it.
@jonathanmetzman: Did I miss anything?
If it looks good to you, I will implement the fix proposed above.
Currently, a
GCB
experiment will be triggered if our comment mentions/gcbrun
.This is problematic because experiments will be launched unintendedly (e.g., when we quote previous
/gcb
commands, or even when we mention/gcb
in any form).Root cause
GCB
trigger simply checks for/gcb
without any constraints.gcbrun_experiment.py
iterates all previous comments to find the latest valid/gcb
command, which might have already been run.Proofs
/gcb
command in the middle of the sentence. It also triggers the same old experiment, confirming the same conclusion.Propose fix
Change https://github.com/google/fuzzbench/blob/56caa83e81bc59a1389367c6bd29d46fd35d03e6/service/gcbrun_experiment.py#L56-L57 to
Justification
/gcb
commands will immediately trigger an experiment, there is no need to search back in history for an old command.@jonathanmetzman: Did I miss anything? If it looks good to you, I will implement the fix proposed above.