./budgen/eb-decision.md

[timm]

After review, relabel to 'reviewTwo'. After second review, relabel to 'EditorsComment'.

[baishakhir]

Title of chapter

Evidence-Based Decision Making

URL to the chapter

https://github.com/ds4se/chapters/blob/master/budgen/eb-decision.md

Message?

Even if the initial data indicates the presence of some phenomena, authors argue that it is important to gather other evidence through repeated experiments, unbiased questionings, etc. before reaching a conclusion.

Accessible?

Is the chapters written for a generalist audience (no excessive use of technical terminology) with a minimum of diagrams and references? How can it be made more accessible to generalist?

The overall message is accessible to the general audience. However, authors use several technical terms and examples that might confuse software practitioners who are not much familiarized with empirical studies.

The first half of the chapter is on collecting evidence. Authors propose five steps of the evidence-based process; first three are about Systematic Reviewing while the bottom two are about knowledge translation (KT). In the second half, authors suddenly shift focus on Systematic Reviewing. They did not talk much about KT while talking in length on Systematic Reviewing. I think these two portions need to be blended well.

Size?

Is the chapter the right length? Should anything missing be added? Can anything superfluous be removed (e.g. by deleting some section that does not work so well or by using less jargon, fewer formulae, lees diagrams, less references).? What are the aspects of the chapter that authors SHOULD change?

I think compared to the other chapters, this chapter is a bit lengthy. In the first section (What’s Evidence), authors introduce three examples: pizza, walking in the hills, salesman’s promise, none of these are from software engineering domain. Just reading the first four paragraph, it was hard to understand what might be its relationship with SE.

The last section can be concise as well. At this point, authors already motivated about Systematic Review. Thus, they can directly come to the point and explain how it can be useful in SE. In fact, an addition of one concrete example from SE to explain “clear-cut results.”

Other points:

• Figure 1 is missing
• SideBox 1: I do not understand the naming of this section
• Example of a Systematic review: The text is unclear, it needs some rewording. Authors talked about RQ5 without mentioning any four RQs.
• There are two Reference sections.

Best Points

What are the best points of the chapter that the authors should NOT change?

The five steps authors presented to implement evidence-based process.

[cabird]

Title of chapter

Evidence-Based Decision Making

URL to the chapter

https://github.com/ds4se/chapters/blob/master/budgen/eb-decision.md

Message?

What is the chapter's clear and approachable take away message? Decisions should be made on evidence and systematic reviews are a way of gathering such evidence.

Accessible?

Is the chapters written for a generalist audience (no excessive use of technical terminology) with a minimum of diagrams and references?

The chapter assumes that the reader is familiar with research papers and scientific studies. I think few software engineers would know where to go to find research papers. I think a little more "hand holding" is needed for a generalist, non-academic, audience.

How can it be made more accessible to generalist?

Give some more details. For instance, the first three items in the list is indeed a systematic review, but they are described at a fairly high level. For instance, tracking down the best evidence in a systematic way. I'm not sure that people would know that this means selected a set of papers. This is made more clear later in the chapter, but I would advocate more concrete details throughout the chapter.

Size?

Is the chapter the right length? It's fine.

Should anything missing be added?

Just more details about where to go to find the evidence. For instance, the authors talk about a search string, but they don't mention what search engine. I assume this wasn't google or bing that they used.

Can anything superfluous be removed (e.g. by deleting some section that does not work so well or by using less jargon, less formulae, lees diagrams, less references).?

Nope. I didn't see anything superfluous.

What are the aspects of the chapter that authors SHOULD change?

As mentioned above, more details and concrete steps.

Gotta Mantra?

We encouraged (but did not require) the chapter title to be a mantra or something cute/catchy, i.e., some slogan reflecting best practice for data science for SE? If you have suggestion for a better title, please put them here.

From the title and intro, I thought this chapter was about collecting evidence, creating evidence, and basically a general empirical approach to decision making. About half way through, I realized it was really about doing systematic literature reviews to collect evidence to make decisions. I think the title can make this more clear. I'm not good with titles, but here are some attempts:

"Using families of studies to make decisions" "Research is more trustworthy than you gut: How to find and use prior studies to make decisions"

Best Points

What are the best points of the chapter that the authors should NOT change?

Sidebar is really nice. Keep it.

[lauriew]

You now have two reviews. As was previously sent out, we are hoping for new versions of the chapter by January 13. I realize this is soon -- but the chapter is not too long.

In addition to the comments by the reviewers, I made the following notes:

• From a writing style perspective -- could you consider each phrase/sentence in parenthesis and see if you can simplify the parsing needed by the reader by removing the parenthesis?
• A surprise for me "Manual searching involved a search of over 30 journals and conferences containing 20,169 publications". Manually they came up with 20,169? That's a tremendously huge number. How was this manual work done?