./alexorso/something-wrong-with-the-universe.md

[timm]

After review, relabel to 'reviewTwo'. After second review, relabel to 'EditorsComment'.

[cabird]

Title of chapter

Software Analytics Under the Lamppost, or What To Do When Your Data Is Telling You There Is Something Wrong with the Universe

URL to the chapter

https://github.com/ds4se/chapters/blob/master/alexorso/something-wrong-with-the-universe.md

Message?

What is the chapter's clear and approachable take away message?

Research in data science ranges from the obvious, easy, and frankly not useful to the difficult, challenging, and game-changing; while a lot of work falls into the former, you should strive for the latter.

Accessible?

Is the chapter written for a generalist audience (no excessive use of technical terminology) with a minimum of diagrams and references? How can it be made more accessible to generalist?

The chapter is completely accessible. I can't imagine anyone associated with software engineering would have an issue reading and understanding this.

Size?

Is the chapter the right length? At first glance, it feels odd that the prologue is half of the chapter. However, the message is short and sweet and each of the "bins" is described well enough that I was able to determine what was meant by it.

Should anything missing be added? After reading the whole chapter, it all makes sense, but when during my first read through, I wasn't quite sure what the authors were trying to say after the first paragraph of "Learning from Data". The authors DO say that they are describing their view of the landscape of research in data science for SE, but that can mean a lot of things, and the authors present a taxonomy of conclusion validity and utility. I wonder if a bit more detail or concrete text can be added to the end of the prologue or beginning of the second section to explain a bit more how the landscape is going to be described.

Can anything superfluous be removed (e.g. by deleting some section that does not work so well or by using less jargon, less formulae, lees diagrams, less references).? Nope. I think everything in there is useful as is.

What are the aspects of the chapter that authors SHOULD change? It's not entirely clear what the difference is between point 3 and 5, especially by their title ("useless" contrasted with "not readily usable". My guess is that point 3 is more about easy research that are useless and point 5 is about more challenging research that may feel "deep" and "cool" to the research and even others, but still isn't really actionable in any way. Probably changing the names of these points could make a stronger distinction.

Gotta Mantra?

We encouraged (but did not require) the chapter title to be a mantra or something cute/catchy, i.e., some slogan reflecting best practice for data science for SE? If you have suggestion for a better title, please put them here.

The mantra makes sense after having read the chapter, but for someone new to the book looking at the title in the table of contents, I don't know if they'll have any idea what the chapter is about. To be fair, I think this happens a lot of times when trying to pick a mantra as a chapter or paper title. If not for the reference from SNG, I would say you could cut the second part of the title and just go with "Software Analytics Under the Lamppost". I don't have much in the way of explicit suggestions.

Best Points

What are the best points of the chapter that the authors should NOT change?

The categories and the accompanying graph of the distribution of research is great. Other than making the distinction between 3 and 5 (as mentioned above), I don't think this needs any major surgery.

Perhaps the best sentence I've read this year!: In other words, if a paper "practically writes itself", let it and move on to something else.

[lauriew]

@alexorso We're still waiting for the second review, though it is promised for tomorrow. Please keep in mind that we are hoping for new versions of the chapter by January 13.

In addition to the one review, I'll make the following observations: "the actual phenomena in which we are interested have not been recorded or are altogether unobservable" ... that sounds pretty intractable. It may disrupt the flow of the paragraph but can you explain when this may be the case and we continue on with our work (e.g. what is a good second order event)?

The list of seven was a bit confusing to me since the title "Learning from Data" is pretty general -- you don't say what the seven items are a list of. It's obvious that the first (incorrect conclusions) was a bad thing and the seventh (correct, non-obvious game changers) is a good thing. But where the list flipped from bad to good in the spectrum is not clear from the first to the last. Some of the words in "which bin is mine?" could be moved up to the prior section.

Can you add some more words of wisdom on how to avoid the bad categories?

[lauriew]

@alexorso Your second review.

General Comments I did struggle with this paper. The section on learning from data you have a list of categories which is out of context and very generalized. I assume this is your list but it is unclear if these are picked from random or just your gut feel or used by other people. So why this list and why not any other list. Also I am sorry but in an empirical book you should never have a graph which is purely theoretical, this is simply 87.5% of statistics are made up. This figure should be removed. Also to claim that data does not lie but people do is rather insulting to any researcher who has wrongly misinterpreted biased or incorrect data.

Other comments. Prologue I have no idea what "science of the artificial" means, which laws of physics are you refering to, existance of dark matter? You claim "we (consciously..." this is a gross generalization of the issue. Again you claim that "we deliberately..." from what basis can you claim this?

Learning from data The section has no context it has no discussion as to how the categories have been chosen and their descriptions.

Which Bin is Mine Placing a graph into a paper which is purely based on gut feel is the classic "how to lie with statistics" and should be removed. In fact creating a set of categories and then placing all of the worlds of SE research into the categories is so wrong

[alexorso]

We thank the reviewers for the feedback. We revised the chapter based on the reviewers' comments and also made a few additional changes. Below, we provide details on how we addressed the different comments.

REVIEW 1:

• After reading the whole chapter, it all makes sense, but when during my first read through, I wasn't quite sure what the authors were trying to say after the first paragraph of "Learning from Data". The authors DO say that they are describing their view of the landscape of research in data science for SE, but that can mean a lot of things, and the authors present a taxonomy of conclusion validity and utility. I wonder if a bit more detail or concrete text can be added to the end of the prologue or beginning of the second section to explain a bit more how the landscape is going to be described.

To address this comment, we reorganized and slightly expanded the text at the end of the first section and at the beginning of the second one. We hope that this makes the discussion clearer.

• It's not entirely clear what the difference is between point 3 and 5, especially by their title ("useless" contrasted with "not readily usable". My guess is that point 3 is more about easy research that are useless and point 5 is about more challenging research that may feel "deep" and "cool" to the research and even others, but still isn't really actionable in any way. Probably changing the names of these points could make a stronger distinction.

We changed the names of Category 5 and tried to explain the difference between the two.

• The mantra makes sense after having read the chapter, but for someone new to the book looking at the title in the table of contents, I don't know if they'll have any idea what the chapter is about. To be fair, I think this happens a lot of times when trying to pick a mantra as a chapter or paper title. If not for the reference from SNG, I would say you could cut the second part of the title and just go with "Software Analytics Under the Lamppost". I don't have much in the way of explicit suggestions.

We changed the second part of the title, hoping to make it more informative while keeping its spirit. We agree with the reviewer that picking a mantra as a chapter title can make the title less intuitive, unfortunately.

• Perhaps the best sentence I've read this year!: In other words, if a paper "practically writes itself", let it and move on to something else.

Thanks! We like it too :-)

REVIEW 2:

• The section on learning from data you have a list of categories which is out of context and very generalized. I assume this is your list but it is unclear if these are picked from random or just your gut feel or used by other people. So why this list and why not any other list.

We believe that the context of the categories is fairly clear from the discussion. The categories are indeed general, in the sense that they can be applied in other contexts (as we say before introducing them), but they are nevertheless appropriate for the context considered. We also clearly state that they represent our view of this area (i.e., they are our list).

• Also I am sorry but in an empirical book you should never have a graph which is purely theoretical, this is simply 87.5% of statistics are made up. This figure should be removed.

In our understanding, this is a book about data science for software engineering in general, and not an empirical book. And our chapter is clearly not a technical piece, so we do not expect it to be interpreted as one. In our chapter, we in fact simply express our opinion in a mostly narrative manner, which we believe makes it perfectly fine for us to use a qualitative diagram that is meant to express a concept, has no quantities associated with it, and does not claim to be something else.

• Also to claim that data does not lie but people do is rather insulting to any researcher who has wrongly misinterpreted biased or incorrect data.

We were not trying to make a general statement of that sort, which by the way does not really reflect our opinion. Misinterpreting accidentally and lying are not the same thing, and in fact, we do mention that conclusions may be wrong because of incorrect/noisy data and bias. We nevertheless removed that sentence, in case other readers may find it offensive.

• I have no idea what "science of the artificial" means, which laws of physics are you refering to, existance of dark matter?

Here we are simply conveying the fact that, unlike in physics experiments, when studying software-related phenomena we cannot rely on a fundamental and well understood set of laws. In fact, we may even make up such "laws" as we go. We note that this is not a concept we are introducing. We are merely restating Herb Simon’s widely referenced (and respected) view.

• You claim "we (consciously..." this is a gross generalization of the issue.

We softened the language to make it less definitive. We now say that this can happen, which we believe is true.

• Again you claim that "we deliberately..." from what basis can you claim this?

We revised this sentence to make it clear that we are referring to our (and, as it turns out, many other colleagues') impressions.

• Learning from data. The section has no context it has no discussion as to how the categories have been chosen and their descriptions.

See response to analogous question above.

• Which Bin is Mine. Placing a graph into a paper which is purely based on gut feel is the classic "how to lie with statistics" and should be removed. In fact creating a set of categories and then placing all of the worlds of SE research into the categories is so wrong

Also in this case, we responded to an analogous question earlier: the graph is clearly presented as a "hypothetical diagram", and we never claim it represents real data. We therefore really do not believe that we can be accused of lying with statistics. Also, we are discussing a very specific research (sub)area within SE, so we are far from trying to categorize "all of the worlds of SE research".

Overall, we respectfully disagree with several of the points raised by the reviewer. We believe that this disagreement mainly stems from a mismatch between the reviewer's interpretation of the chapter's spirit and the chapter's actual nature, as we discussed above.

ADDITIONAL REVIEW/COMMENTS BY THE EDITOR:

• "the actual phenomena in which we are interested have not been recorded or are altogether unobservable" ... that sounds pretty intractable. It may disrupt the flow of the paragraph but can you explain when this may be the case and we continue on with our work (e.g. what is a good second order event)?

We added an example of possible "second order event".

• The list of seven was a bit confusing to me since the title "Learning from Data" is pretty general -- you don't say what the seven items are a list of. It's obvious that the first (incorrect conclusions) was a bad thing and the seventh (correct, non-obvious game changers) is a good thing. But where the list flipped from bad to good in the spectrum is not clear from the first to the last. Some of the words in "which bin is mine?" could be moved up to the prior section.

We added some discussion to clarify the categories and the goodness of the corresponding research.

• Can you add some more words of wisdom on how to avoid the bad categories?

We tried to reorganize the final part of the chapter to make our general recommendation clearer. Paraphrasing a bit: "researchers in the area of data science for software engineering should carefully assess where their work falls in terms of usefulness and the actionability of their findings and act accordingly". The categories we propose (or similar ones) could be used to that end.