I don't get the quality aspects

[tsdh]

Sorry, but I don't get the quality criteria for which solution authors and reviewers should provide a score in the range of 1-5.

• Comprehensibility: Ok, I mostly get this although I disagree with the statement about the "high priests of some cult". Sometimes, a new mechanism might look incomprehensible and non-obvious at first because it conflicts with long-standing habits but still proves to be beneficial in the longer run.
• Readability: Aha, "outer appearance of the tool..." What if there is no "tool" but just a language, or in my case, just a library in some other language where you are free to use whatever editing environment suits you best?
• Communication with the user: The description says this is about the "...quality, informativeness and level of detailedness of the actual messages given to the user while implementing a solution". How should a reviewer decide about that? He'd need to understand a solution well enough to modify it in such a way that it errors because of a reason he explicitly had in mind when doing the modification. And even then its just scratching on the surface.
• Robustness: I guess most of us don't write their own Java parser but reuse JaMoPP, Modisco, JDT or something else. And then malformed input is handled by the reused tool. And with respect to out-of-scope keywords, there are two choices: either a solution treats a static method as a normal method or it doesn't consider it at all (that's what I do, i.e., static methods don't make it into the program graph at all). But those are design decisions which have nothing to do with the language/tool/solution's quality. And what is an "erroneous environment" here?
• Extensibility: Well, adding more refactoring rules is probably easy with any solution/tool. But adding support for additional Java constructs, say, visibility modifiers, requires changes to the program graph metamodel and thus to the Java-to-PG transformation and the refactoring rules. Again, that hasn't much to do with concrete solutions and tools.
• Debugging: Ok, this one is pretty clear. :+1:

To make a long story short, these criteria are far too imprecise. As a solution author/tool developer, I can write several pages on each of them discussing every possible aspect in every possible scenario. And as a reviewer, I'd pretty much depend on such an in-depth discussion in order to make my scores more meaningful than a score picked from plain gut feeling. So I'd vote for making the Reviewer opinion a simple "how much do you like the solution? 1 (bad) - 15 (excellent)" question and let solution authors off the penalty of having to write something clever about these aspects.

[gkulcsar]

OK, this part might have been over-engineered with a lot of vagueness. I myself not particularly insist on keeping these very aspects. We can start agreeing on a lightweight alternative. @SvenPeldszus your opinion? I think it can be a good idea to simplify the reviewer opinion score (and it is definitely too much effort to verify a solution against each of the aspects). Still, wouldn't it facilitate the discussion if the developers would shortly touch on the soft aspects of their solution?

[tsdh]

Still, wouldn't it facilitate the discussion if the developers would shortly
touch on the soft aspects of their solution?

Maybe, but naturally each author touches the aspects that seem most relevant to her and her tool. And then different solution descriptions discuss different aspects which might facilitate discussion but won't help with comparing solutions.

In the past, we've usually asked authors to highlight the strong and the weak spots of their solutions/tools without predefining concrete aspects. For example, I'd write for my FunnyQT solution that it's very concise and fast. On the negative side, using FunnyQT requires some Clojure/functional programming knowledge, and Clojure (and thus FunnyQT) is not statically type-safe. The same is also contained now in my soft aspects paragraph where I tried to place these statements at the right aspects.

[gkulcsar]

That's absolutely true that the authors naturally touch the relevant aspects. I would also follow your experience from the past and give the authors this idea about strong and weak spots.

On the other side, even if we omit the forced soft self-evaluation (of which you've convinced me by now), I think the aspects in the paper (maybe presented as guidelines in the next version of the case study) can help comparing the solutions in a non-objective manner.

Summarizing: are you OK with the following solution? (1) We simplify the reviewer score to a single value of 1 to 15, (2) I add this sentence about strong and weak spots, (3) the mentioned soft aspects remain in place but with the slogan "guidelines" or "possible directives" for soft evaluation without the need of scoring them.

A connected general comment: for the future editions of TTC, I would consider preparing something like a "Case Authors' Guide" which would help the unexperienced (cf. us :) structure their case study and prevent them from reinventing each part of the case study including those where a best practice may exist long ago. Of course we don't have to discuss the issue right here, right now :)

[tsdh]

@gkulcsar Yes, that seems good to me.

About the "Case Authors' Guide", that would be a good idea indeed.