Tool instead of Prediction Challenge

[msrchallenge]

I talked to Tom Zimmermann and he was a of the opinion that the prediction challenge is getting a little bit old and its time for something new.

I was thinking of having something of a tool challenge where people submit tools that implements a given set of functionality. We would test them on correctness and compare them by speed.

I was thinking of two features (1) extract a CVS/SVN archive into both a PostGreSQL and MySQL database (2) extract failure introducing and fixing changes by searching for a given set of words that we provide

Do you think that makes sense?

[emadshihab]

After reading this post, I thought about this new spin. One thing that I could not answer is: What is the purpose of doing this? What is the outcome we would like to see? 1) Do we want people to make tools that others can later use or 2) do we just want people to write the tools so them become familiar with the process and have some fun.

If the purpose is 1) then I think we need to be very specific as to what we want and make sure the tools conform, are easily adaptable, etc... We probably need to give people more flexibility on what the tool does and how it will work.

If we are looking for 2) then maybe we should just design a class project-type thing and ask people to make tools that solve the issue we want.

Overall, I think the idea makes sense and could generate a lot of interest in the challenge.

[schadr]

It is kind of a (1) and (2) but the focus is definitely on (2). The idea is to get more students interested in MSR and extend it even to undergrads that are not that familiar with research but might know how to write cool tools.

The idea was to define the input and the output and then simply measure the time it took to generate the correct output. I want to avoid to much flexibility because then it becomes more of a major mining challenge in its own right rather than something "small" as the previous prediction challenge.

[ghost]

I like the idea. The main issue I see is to pick a problem/challenge that is interesting enough but has the right size/effort needed as the prediction challenges. Since several groups already worked on extracting CVS/SVN archives, why not focus on tools/analyses based on that data, leaving the initial history extraction to an already existing tool?

[schadr]

I think before we decide what tool challenge we actually come up with, we should decide on for whom this challenge is. To be honest, although I like Daniel German (winner of the last prediction challenge) I don't think the challenge is something faculty should really participate in. I had kind of the idea that we should make a shift towards students that are not really already part of the MSR community and give faculty the chance to include this tool challenge into their courses to bring undergraduates and graduates into the fold.

So something like a student version of this http://fimi.cs.helsinki.fi/

[ghost]

@schadr: I totally agree on shifting the challenge towards students. I also was thinking that in this case, the challenge would perfectly fit into an undergrad/grad course.