llayman
commented
8 years ago This issue was addressed in the first Revision to EMSE
Closed timm closed 8 years ago
llayman
commented
8 years ago This issue was addressed in the first Revision to EMSE
DEAR AUTHORS-- THIS TEXT NOW MOVED TO PAPER
This new draft now maps our work into the SE literature on theory building.
This new draft now maps not contains extensive notes on the origin and characteristics of our data.
All our results are now augmented with statistical significance tests as well as effect size tests. Those tests greatly strength the overall message of this paper.
[ ] @fshull need words on this
Fig 10: Black and red is not a good pair for grayscale printouts. Please use black and gray instead.
Some figures are actually tables, thus their captions should be replaced accordingly.
Some figures should be resized to better match the page width.
Page 3:14 - "The above argument" Which argument? Could be precisely specified.
Page 3:22 - Please provide the full link to the dataset.
Page 4:8 - "More difficult" but "harder" is used in claim3. Why this inconsistency?
Page 18:26 - First sentence: "Unexpected results such as this one". Ambiguous reference, please be specific.
TYPOS ETC.
Sec 1, §1: [3] and [30] have been swapped?
Page 14:46 - Spell out IV&V the first (and only) time it appears. Page 3:24 - missing verb (is) Page 16:43 - "Did this study failed" Page 18:44 - Appache Page 19:19 - Al Ref 26 - oo --> OO
Reviewer #2: The topic of this paper is the empirical evaluation of a largely admitted claim in software engineering, i.e. the exponential cost of correcting errors according to the phase in which the errors are discovered. The claim is first confirmed by surveying and interviewing practitioners and experts. Next, the authors use a large set of data from projects in SEI database. The results show that, strictly speaking, the claim does not hold (although some effects of delayed corrections can be noticed).
Globally, the paper is well written and the authors develop a convincing demonstration. The topic of the paper is highly relevant for both practitioners and people from academia. Among many, I personally believed in this claim and had regularly taught it in my courses. Thus, reading this paper offers a refreshing perspective on our understandings of software engineering background and theoretical knowledge. I also like the suggested posture that insists on the necessary skepticism that we should have against such prevailing claims.
Beyond this positive global impression, I have some concerns about the study reporting and analysis: - It seems to me that more descriptive statistics about the sample would contribute to a better understanding of the scope of the study.
I think in particular about the size (lines of codes and/or number of software components); a duration histogram would also be relevant. Accordingly, the formula for calculating variable "Total effort" in Fig. 7 could be explained, and Figure 9 made more readable (bigger size).
If certain attributes for the issues and errors are available in the data (e.g. severity, priority, etc.), they should also be brought to the readers knowledge.
Figure 10 is central to the paper's demonstration; its expressiveness could be enhanced: i) the reader tends to think that the "right hand side bars" are different from the BLACK and RED bars, ii) the formula for calculating the 50th and 95 ratio could be provided, iii) the unit of column "Percentile" could be mentioned.
I like the discussion section, in particular, the idea of software architectures that could be a contributing factor for enhancing software evolution and reducing the cost of issues and errors fixing. Is there any available knowledge in the data set concerning any architectural design choices made in each project? If so, would it make sense to seek any correlation between these architectural choices and how expensive it was to solve issues?
In the same trend of ideas, I think what has fundamentally changed since early days of SE is the development of requirements engineering. In actual SE practices, problem and the solution spaces are systematically explored thanks to RE techniques; together with architecture, this could also explain why things have changed since the 70'. It could be, for example, that people tend to make less severe errors in early software project phases (thanks to RE techniques); a similar phenomenon was observed when SE practices become more mature (see Harter et al. 2012).
References Harter D. E., Kemerer C. F., Slaughter S. (2012). Does Software Process Improvement Reduce the Severity of Defects? A Longitudinal Field Study. IEEE Transactions on Software Engineering, vol. 38, n° 4, p. 810-827.
In the conclusion section, I feel uncomfortable with the assertion "That data held no trace of the delayed issued effect" (line 17). As mentioned elsewhere in the paper, the delayed issue is not absent; it is much less significant and systematic than what is usually claimed. Moreover, this reduction has been demonstrated for medium sized projects; we cannot generalize to larger projects.
Minor remarks: - p.9, line 29 : " … reported by Shull [52], found that the cost to find certain non-critical classes of defects …" => is it the "cost to find" or the "cost to FIX"?
Reviewer #3: I like the idea of the paper and I certainly would like to see more empirical studies that periodically check if our beliefs about software engineering practices still hold (or even if they have ever held). The authors address one of the beliefs that are more entrenched in the population of researchers and practitioners, and a very important one too.
At the same time, I think that the paper needs to be improved before it can be published.
The empirical analysis (Section 6) is a bit of a let down. I would like to see some sort of more robust and rigorous statistical analysis, based on statistical tests. Instead, the paper only provides qualitative comparisons between the results obtained at the 50-th and 95-percentiles. This is not to say that the results or the discussions are incorrect, but only that they need to be better supported. The lack of this kind of statistical analysis should have least be mentioned in the Threats to Validity. I understand that you say that your "claims" (including Claim 3) should not be considered "hypotheses" in the statistical sense and I agree with you on Claim 1 and Claim 2. However, Claim 3 lends itself to a statistical analysis, without which your results become a bit more anecdotal, and this is what you rightly criticize in the previous claims that DIE actually exists. In addition, you may want to analyze some additional and somewhat unexpected results that can be derived from your data (see Detail Comments below).
The paper seems to be written in a somewhat casual style, which makes the paper more pleasant to read, but less clear in some parts (see Detail Comments below).
DETAIL COMMENTS
Section 2
I'm not really sure what the authors mean by "We say that a measure collected in phase 1, ., i, .. j is very much more when that measure at phase j is larger than the sum of that measure in earlier phases 1 <= i < j." You are defining a property of a measure, but, the way this definition is written, it seems as if you are defining the property of a measure of "being much more," without any further qualifications. I can guess that you mean that a measure $m$ (collected in phases, so you can denote by $m(i)$ the value of $m$ in phase $i$) has this property in phase $j$ (so, it's a property m has in a specific phase and not a general property of the measure) if $\sum{1 <= i < i} m(i) < m(j)$. Then, it's up to you to make this a property of the measure, for example by using an existential "policy" ($m$ has this property if there exists one phase $j$ where $\sum{1 <= i < i} m(i) < m(j)$) or a universal one ($m$ has this property if for all phases $1<j$, $\sum_{1 <= i < i} m(i) < m(j)$).
The real problem with this definition, however, is that it doesn't seem to be used anywhere in the paper. The only point where it might be used is in the discussion of Figure 1 (which precedes it anyway). So, you may want to remove this property altogether.
Your definition of "difficult issue" is "Issues are more difficult when their resolution takes more time or costs more (e.g. needs expensive debugging tools or the skills of expensive developers)." That's not a very precise definition, though, because it has two different interpretations, one in terms of time and the other in terms of effort. The reasons why you introduce this definition become clearer only in Section 7.2, in the Threats to Validity, and that's too late. You should move some of the discussion of Section 7.2 here.
Why would the term "delayed issue effect" be a generalization of the rule "requirements errors are hardest to fix"? "Delayed" seems to be quite specific as it appears to refer to time, while "hardest to fix" may refer to other variables, like effort or cost.
In Claim 3, "very much more harder" sounds like a little bit too much ... ("more harder"?)
Section 4
A very minor issue: why is Figure 2 a ... figure, instead of a table? Same for all other tables ...
Section 5.1
This is not a complete sentence "All the literature described above that reports the onset of DIE prior to delivery."
Section 6.3
You should rephrase your definition "a defect is any change," since a defect is what existed before the change was made and not the change itself.
You do not explain that "QualTest" in Fig. 8 is.
The definition of "time per defect - The total # of defects found in a plan item during a removal phase divided by the total time spent on that plan item in that phase." is not correct as it is, as this would basically be the number of defects per unit time, instead. The roles of time and defects should be reversed.
Section 6.4
Besides adding a more rigorous statistical analysis, you need improve your explanations.
Figure 9. The caption says "Distribution of defects found and fixed by phase." Is that the distribution of defects found in some phase and fixed in that phase? The text says "The distribution of defects found and fixed per phase in our data is shown in Figure 9. A high percentage of defects (44%) were found and fixed in the early phases, i.e., requirements, high level design, and design reviews and inspections." which doesn't add much. If that's the distribution of defects introduced in a phase and fixed immediately in that phase, it is not clear why you introduce it, though.
Figure 10. The explanation of the data and histograms in the figure is, at best, confusing. The caption says "50th and 95th percentiles of issue resolution times" and the opening sentence of Section 6.4 says "Figure 10 shows the 50th and 95th percentile of the time spent resolving issues ..." However, these are not "times," because you write "expressed as ratios of resolution time in the phase where they were first injected" a couple of paragraphs below. This seems to be the right interpretation, but right after that you write "The BLACK and RED bars show the increases for the 50th (median) and 95th percentile values, respectively," which would be a third interpretation of the data in Figure 10.
You should provide the value of the mean, in addition to the 50-th percentile (i.e., the median).
You need to at least mention (or, better, discuss) some of your results, because they appear to be "counterintuitive," since they show that, for example, median resolution times can even decrease if issues are not fixed immediately, but in later phases. For example, looking at the Reqts section, most percentages are way below 1, which seems to indicate that some issues are actually much easier to fix at later stages (maybe because more or better information about the software system is available only later). That would be a very interesting result by itself, especially if you can provide some statistical support for it.
Section 7.1
Maybe you could analyze the projects developed in a "traditional" way and check if there is some sort of DIE there. For example, if you look at the projects developed with the waterfall life cycle, maybe you will find larger DIE than for the other life cycles, which might partially justify the claims of previous researchers about the existence of DIE. This would also show that newer kinds of life cycles (like Agile ones) help solve DIE. You do provide hints about this in Section 8 ("We also note that other development practices have changed in ways that could mitigate the delayed issued effect") and Section 9, but that's clearly not enough.
Section 8
I'm not sure what you are really trying to discuss in this section, because you are not discussing the results of the paper. It sounds like you are saying that maybe it's the newer software development approaches that made DIE disappear?