Archive and release private experiments

[vanhauser-thc]

Many academic papers user fuzzbench to evaluate their contributions vs other fuzzers. Rarely (as in "I have never seen this done but it might have happened without me having noticed") however the testing setup and the Dockerfile of the new fuzzer is made available. This makes assessing and reassessing papers impossible.

E.g. getting symqemu running is so far failed by Jonathan and me individually. In other experiments - did e.g. afl++ in the experiment have cmplog enabled and which version was it tested against? Impossible to know.

I therefore propose to archive private experiments and make these available 6 months later - unless the requestor actively announces this should not be done.

[inferno-chromium]

Good idea. Thoughts @lszekeres @jonathanmetzman @mboehme @laurentsimon ?

[jonathanmetzman]

Many academic papers user fuzzbench to evaluate their contributions vs other fuzzers. Rarely (as in "I have never seen this done but it might have happened without me having noticed") however the testing setup and the Dockerfile of the new fuzzer is made available. This makes assessing and reassessing papers impossible.

E.g. getting symqemu running is so far failed by Jonathan and me individually.

I'm not sure failed is the right word for this. I just had other things to do and forgot. But I'll try to finish #1062 soon.

In other experiments - did e.g. afl++ in the experiment have cmplog enabled and which version was it tested against? Impossible to know.

I therefore propose to archive private experiments and make these available 6 months later - unless the requestor actively announces this should not be done.

Given the glacial pace of academia, I'm not sure 6 months is long enough. Maybe instead of having a hard timeline we can just make them public after the paper is published. What do you think of this? I'll start by asking symqemu folks if it's OK that I do this with their's

[vanhauser-thc]

Many academic papers user fuzzbench to evaluate their contributions vs other fuzzers. Rarely (as in "I have never seen this done but it might have happened without me having noticed") however the testing setup and the Dockerfile of the new fuzzer is made available. This makes assessing and reassessing papers impossible. E.g. getting symqemu running is so far failed by Jonathan and me individually.

I'm not sure failed is the right word for this. I just had other things to do and forgot. But I'll try to finish #1062 soon.

yeah same for me. I was exaggerating a bit :)

In other experiments - did e.g. afl++ in the experiment have cmplog enabled and which version was it tested against? Impossible to know. I therefore propose to archive private experiments and make these available 6 months later - unless the requestor actively announces this should not be done.

Given the glacial pace of academia, I'm not sure 6 months is long enough. Maybe instead of having a hard timeline we can just make them public after the paper is published. What do you think of this? I'll start by asking symqemu folks if it's OK that I do this with their's

I thought more of an automated process that does not need hands-on unless the requestor says no (or sets a yaml "dont_disclose: true"). following up each experiment with which paper that is in is hard I think.

[lszekeres]

Thanks for bringing this up @vanhauser-thc! I agree, private experiments should be made public once the paper is accepted/published with the authors' approval.

I thought more of an automated process that does not need hands-on unless the requestor says no (or sets a yaml "dont_disclose: true"). following up each experiment with which paper that is in is hard I think.

This would be hard to automate as it should happen once the authors confirm that the paper is accepted for publication, and not after a fixed period of time. Authors would most likely prefer a system where they can say when to publish, as opposed to "data gets auto-published after some time unless authors remember to prevent it".

[mboehme]

I agree. It would be good if private experiments, specifically those that are presented in the paper are ultimately published after the paper is accepted.

• Not all experiments need publishing. Some experiments never appear in a paper. Some experiments are just part of the prototyping or debugging. For a peer-reviewed and published paper, the experiments with key results should be sufficient. The authors should be able to choose that experiment.
• Such a policy and its benefits should be well documented. Authors should know that they are going to consent to publishing one of the experiments if it results in a peer-reviewed publication. They should also know how this will benefit the community and reproducibility.
• At the end of the day, the authors should be able to approve and choose that experiment which is going to be published.
• Authors are welcome to link the paper, DOI, and other artifacts in the final report.

[sebastianpoeplau]

I'll start by asking symqemu folks if it's OK that I do this with their's

Absolutely yes (cc @aurelf)! I've been meaning to clean up the integration scripts and create a pull request anyway, but so far I haven't found the time. @jonathanmetzman do you have an easy way to publish the final configuration that you ran the experiments with? Otherwise I can piece it together from our email exchange at the time.