./natalia-juristo/Reproducible

[timm]

After review, relabel to 'reviewTwo'. After second review, relabel to 'EditorsComment'.

[timm]

I am a fake review of Natalia's paper.

I am reviewing this paper on the criteria in the CFP:

[timm]

I am a fake review of Natalia's paper.

I am reviewing this paper on the criteria in the CFP (work in progress here...)

(If you have confidential comments about this chapter, please email them to one of the book editors.)

How would you categories the chapter: is it

1. SE-specific lessons (e.g. some comments on issues that are specific to data science for software engineering)
2. Some general lessons about data analytics 3, Other (specify)

??

On a scale of 1 to 7, nofun to fun, how fun is the chapter to read?

??

What is the core insight of the chapter?

written in one line or less

What is the chapter's clear and approachable take away message?

??

Is the chapters written for a generalist audience (no excessive use of technical terminology) with a minimum of diagrams and references?

??

Should the chapter be pruned (make it shorter)? If yes, how (e.g. (1) use less jargon, formula, diagrams, references; or (2) delete some section that does not work so well)

??

Can the chapter be longer? If yes, what else is needed?

??

If the chapter title a mantra; i.e. . some slogan reflecting best practice for data science for SE? If not, suggest a better title.

??

What are the best points of the chapter that the authors should NOT _ change?

??

What are the weaker aspects of the chapter that authors SHOULD change?

??

[gousiosg]

Title of chapter

Importance of Repeatable Results: Once is not enough

URL to the chapter

https://github.com/ds4se/chapters/blob/master/natalia-juristo/Reproducible%20results.md

Message?

What is the chapter's clear and approachable take away message?

If we want reliable research results, we must replicate case studies/experiments.

Accessible?

Is the chapters written for a generalist audience (no excessive use of technical terminology) with a minimum of diagrams and references? How can it be made more accessible to generalist?

Yes, I found it very accessible

Size?

Is the chapter the right length? Should anything missing be added? Can anything superfluous be removed (e.g. by deleting some section that does not work so well or by using less jargon, less formulae, lees diagrams, less references).? What are the aspects of the chapter that authors SHOULD change?

I found the introduction/motivating example too extensive. I think that the author makes the case for the need of replication early on, so, in my opinion, it is not strictly necessary to include all sub-cases where replication may be required in the motivating example.

Moreover, I think that the chapter could use some structure in the form of headers.

Gotta Mantra?

We encouraged (but did not require) the chapter title to be a mantra or something cute/catchy, i.e., some slogan reflecting best practice for data science for SE? If you have suggestion for a better title, please put them here.

Yes, but it is not very well reflected in the title. A punchier title, e.g. "Once is not enough" could attract the reader's interest better

Best Points

What are the best points of the chapter that the authors should NOT change?

The three concluding bullets capture the essence of why replications are important

[maleknaz]

Title of chapter

Importance of Repeatable Results: Once is not enough

URL to the chapter

https://github.com/ds4se/chapters/blob/master/natalia-juristo/Reproducible%20results.md

Message?

To form a phenomenon from results of empirical studies we need to replicate the studies.

Accessible?

It is accessible.

Size?

The chapter is on the right size.

While I like the ‘planet’ example and it is nice but might be good to summarize it.

Also putting more emphasis on some key ideas or adding some sub-titles would be great.

The three points at the end contain important message. Adding more description to them and summarizing and adding some guidelines would be good.

Gotta Mantra?

Yes it is good, I like the part “Once is not enough”. You might want to start the title with this.

Best Points

Chapter has an important message and Natalia wrapped it up within a nice example.

[timm]

Hey Natalia, I understand why you went to such a high level (approachability! great goal!) but I'm thinking you went a little too high level (but full marks for trying not to write the standard dry academic paper!).

The chapter need a little more "crisping up", more specifics. Perhaps some structural features might help:

• list of key takeaways
• list of traps (rookie mistakes, how NOT to do this)

[timm]

:+1: Good to go