search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.09k stars 266 forks source link

Fix broken test for corpus experiment requests #1995

Closed ndrewh closed 3 months ago

ndrewh commented 3 months ago

Follow up to #1993

ndrewh commented 3 months ago

@DonggeLiu

DonggeLiu commented 3 months ago

Thanks!

DonggeLiu commented 3 months ago

In addition to the testcases, did you happen to have a chance to check if there are other inconsistent usages elsewhere? E.g., would these be related by any chance? https://github.com/google/fuzzbench/blob/0f85b4a1843e022f323f8af0db7443b8d79544d0/experiment/scheduler.py#L777 https://github.com/google/fuzzbench/blob/0f85b4a1843e022f323f8af0db7443b8d79544d0/experiment/test_data/experiment-config.yaml#L34

Thanks!

ndrewh commented 3 months ago

I think everything else is "fine". experiment_requests.yml is not consistent with the experiment config itself but that is not a "problem" necessarily but is certainly a future source of confusion.

If you prefer: instead of #1993 we can switch experiment_requests.yml to use the underscore instead, which would be consistent with the experiment configs. We would revert #1993 and instead just fix docs and probably re-write the old experiment requests to use the underscore name (to avoid errors where someone copy/pastes them). But as was pointed out in #1993, I don't think experiment_requests.yml is actively used anymore, so it may be a moot point.

DonggeLiu commented 3 months ago

Thanks @ndrewh! It would be amazing if you could help us make the name consistent to avoid future confusion. We don't hold a strong opinion on using - or _, the one that's the most convenient for you is the best for us : )

There is no need to rewrite the old experiment requests YAML: it will be not used/parsed/revived anytime soon.

Thanks!

ndrewh commented 3 months ago

@DonggeLiu #1997