swan17-contributor: Analyzing Source Code for Automated Design Pattern Recommendation

[timm]

https://github.com/researchart/swan17/blob/master/pdf/swan17-contributor.pdf

ABSTRACT

"Mastery of the idiosyncrasies of object-oriented programming languages is undoubtedly challenging to achieve. In order to improve this situation, various design patterns aim on supporting software designers and developers in overcoming recurring design challenges. However, given that dozens if not hundreds of patterns have emerged so far, it can be assumed that their mastery has become a serious challenge in its own right. In this paper, we propose a recommendation system that aims on detecting those “design smells” in object-oriented systems that can be mitigated by the application of well-known design patterns. We describe a proof of concept for which we mined a significant collection of source code in order to derive a detection rule for identifying a smell that can be rectified by the Strategy pattern. Moreover, we present results from analyzing 25 open source systems with this rule, indicating that developers indeed can benefit from tool support in recognizing design smells, as more than 200 candidates were detected. Finally, we briefly sketch how we are currently extending this work to other patterns."

CCS CONCEPTS

• Software and its engineering -> Software notations and tools -> General programming languages -> Language features -> Patterns

KEYWORDS

"Design Patterns, Recommendation System, Code Analysis"

[timm]

AUTHORS: Important. Do NOT reply till all three reviews are here (until then, we will delete your comments)_.

---

Reviewer1

Insert reviewer github id here ==> swan17-reviewer4

Recommendation (select one)

• [ ] Accept
• [X] Accept if (see below)
• [ ] Reject

Summary (1 para)

The paper presents the initial steps towards an approach for detecting opportunities for the application of design patterns in Object Oriented and recommending such applications to developers. Specifically, the authors have developed an approach for detecting opportunities for the application of the Strategy pattern. This is done using static analysis, by using a detection rule on the AST of java programs. The paper also describes a preliminary evaluation of the approach based on several open source projects as well as a small user study.

Advocacy (accept since, reject since, 1 para)

The paper attempts to tackle an important problem and the preliminary results are very promising. The approach is well researched and related work is reasonably researched. The degree of evaluation is appropriate for a workshop paper. Overall, I think that this would be an interesting addition to the workshop. However, I have some reservations, which I describe at the bottom of the review.

List of "Pros"

• Important problem
• Interesting approach based on static analysis
• Reasonable preliminary evaluation

List of "Cons"

• Does not discuss the extremely relevant work of Nikolaos Tsantalis (e.g. [a][b]). This makes it hard to establish the novelty of the proposed approach.
• Makes some over-generalizing claims in Section 5.2: First, it is not inconceivable that opportunities for the Adapter, Interpreter, etc patterns cannot be identified by tools that rely on techniques other than static analysis (e.g. analysis of requirements?), so saying that they just cannot be detecting is over-generalizing. Second, the authors over-emphasize the prevalence of Agile techniques. There exist domains (e.g. automotive software) in which model-based techniques do allow the maintenance of design artifacts along the code.

[a] N. Tsantalis, A. Chatzigeorgiou, G. Stephanides and S. T. Halkidis, "Design Pattern Detection Using Similarity Scoring," in IEEE Transactions on Software Engineering, vol. 32, no. 11, pp. 896-909, Nov. 2006. doi: 10.1109/TSE.2006.112 [b] N. Tsantalis and A. Chatzigeorgiou, "Identification of Move Method Refactoring Opportunities," in IEEE Transactions on Software Engineering, vol. 35, no. 3, pp. 347-367, May-June 2009. doi: 10.1109/TSE.2009.1

Changes needed before I can recommend accept (if any)

• The authors should establish the novelty of their approach especially with respect to the work of Nikolaos Tsantalis.

[timm]

_AUTHORS: Important. Do NOT reply till all three reviews are here (until then, we will delete your comments)_.

---

Reviewer2

Insert reviewer github id here ==>

Recommendation (select one)

• [ ] Accept
• [X ] Accept if (see below)
• [ ] Reject

Summary (1 para)

This paper attempts to find a detection rule for design smells that should be mitigated by refactoring using the Strategy pattern.

Advocacy (accept since, reject since, 1 para)

I started off really liking this paper, but as I read on, some sizable flaws emerged.

List of "Pros"

• The methodology of using the PMD tool to build an AST which is then inspected for use of Strategy pattern is a fun idea.

List of "Cons"

• The biggest weakness in this paper - and where it really started to fall apart for me in a big way - was sections 4.1 & 4.2. These were extremely problematic. The ideas are not bad, but trying to read these sections was frustrating. The biggest issue for me was the unclear use of the words 'strategy', 'Strategy', 'state' and 'State'. Many of the words seem to be used interchangeably, even though the authors had taken care to distinguish the State and Strategy patterns earlier in Section 3.1. I feel like this distinction is supposed to be important to them, yet throughout Section 4 the terms are used interchangeably and in a very unclear, jumbled fashion. It feels like jargon, and misapplied. For example, the axis labels in Figure 1 refer to Strategies, and the text says that Table 2 refers to figure 1. But then table 2 refers to 'state' in its caption, and 'Strategies' in the column header. I just can't understand what is going on here. The words seem to be used sometimes as generic nouns, sometimes as proper nouns, sometimes singular, sometimes plural. I'm NOT trying to make a grammar argument here, rather, I am simply quite confused about what is going on. There are other similar confusing word choices. Do you mean 'useful' or 'suggestive' for the second threshold? Both words are used.

• The paper seems rushed and confused, especially beginning in Section 4. The tone throughout is odd; sweeping statements are made without references, and the paper is replete with assumptions.

Changes needed before I can recommend accept (if any)

• Section 4 needs to be completely re-written to use words carefully and with precision.

• Clarify and clearly include references to assumptions. Example: "Admittedly, identifying design smells..." on page one. This seems major. Too important for "due to various pragmatic reasons" and the extremely confusing "such as the better availability of open-source[sic] compared to "open designs")" This entire exchange feels like you are referring to some settled argument that seems obvious to you. However this "fact" - whatever it is - is not obvious. You again make the "open-source" argument at the bottom of section 5.1. I feel like there is a lot being left unsaid here, or assumed. What is going on?

• Section 2 is unsatisfying. In the first three paragraphs, there are two anonymized self-references. The rest of the references are old or are just links to software (PMD, Refactoring). This is the "State of the Art" in refactoring design smells? I am disappointed. Why is there a subsection 2.1 but no 2.2? I think this entire review of lit section needs to be expanded both in quantity of references and the organization of what you are reviewing.

• In section 4, and again in section 4.2, you make the statement that there is overhead involved in implementing a pattern. Doesn't it seem like this should be explicated further in the section about design patterns, 2.1?

• Table 4 should not have items from the survey left out, as it is misleading about what went on. The table should be as "standalone" as possible, without having to read through the text. Easy solution if you are running out of room: Re-orient the table so that the columns and rows are flipped.

• Clarify how you chose "two of these results (found in ArgoUML and Apache Lucene)" that you mention in section 5.

• "It should be obvious that..." (section 4.1) means that either you shouldn't need to write this, or it wasn't obvious to someone along the way.

• By the time you define "design smells" in Section 2.1, you have already used this phrase 4 times in the paper. Do you mean to move this definition to earlier in the work?

• Typos: "On note" -> one note, "relative large number" -> relatively.

[timm]

_AUTHORS: Important. Do NOT reply till all three reviews are here (until then, we will delete your comments)_.

---

Reviewer3

Insert reviewer github id here ==> anonymousReviewerX

Recommendation (select one)

• [ ] Accept
• [x] Accept if (see below)
• [ ] Reject

Summary (1 para)

The paper describes an approach to detect design smells where applying a design pattern may mitigate the smell. An example of the Strategy design pattern is analyzed in this paper; however, the concepts may generalize to other design patterns.

Advocacy (accept since, reject since, 1 para)

In general, I like the idea of extending PMD to detect deeper design flaws.

List of "Pros"

• Extends upon popular code analysis infrastructure
• Preliminary analysis includes feedback from potential users

List of "Cons"

• A key concern that I don't believe has been addressed is whether the design smells that are being detected are worse than an over-engineered solution. Applying too many design patterns can introduce unnecessary complexity that may clutter up a codebase. How will the proposed approach avoid suggesting over-engineered solutions?

• I am skeptical of the claim that "at the time being, there is only one tool available (called "Pattern4" [23]) that can be used for identifying existing patterns in arbitrary source code." For example, the Pattern4 paper [23] cites several papers that propose automated design pattern detection approaches. Is there really no other tool available?

Changes needed before I can recommend accept (if any)

• If the concept of software quality includes fault and change proneness, the work of Jaafar et al. [1] should also be discussed in Section 2.1.

[1] Jaafar et al., Evaluating the Impact of Design Pattern and Anti-pattern Dependencies on Faults and Changes, Empirical Software Engineering (EMSE), 2015.

• Citations or data are needed to support the claim that: "Rules based on these abstract descriptions alone would not be sufficient to produce practically usable pattern recommendations, since patterns usually come at the price of introducing overhead into the code."

Typos and Other Minor Concerns

p1:

"aim on supporting" -> "aim to support" "aims on detecting" -> "aims to detect" "tool that it integrates" -> "tool that integrates" "these the opportunity" -> "these opportunities" "mining a large source code collections" -> "mining large source code collections"

p2:

"(admittedly not too many)" <- Please be specific. How many is "not too many"?

p5:

"mini survey" <- What is that and how does it differ from a survey? "On note on" -> "A note on" (?) "distilled their code so that they were as easy to grasp as possible" <- How was "easy to grasp" operationalized?

[swan17-reviewer4]

Edited Reviewer1 comment above.

[ghost]

First of all, thanks to all reviewers for their valuable comments on how the paper can be improved.

We have already started to incorporate the feedback of reviewer 1 as follows:

• The work of Tsantalis et al. is certainly important and worth mentioning and hence we discuss it in our state of the art section. Thanks again for pointing us to it. However, as far as we can tell, there is no direct overlap with our approach: the first paper we were pointed to is tackling the detection of existing patterns in code and not the lack of a pattern as we aim to do. Their second paper is interesting in the sense that it also aims on detecting refactoring opportunities, although they aim on the relatively simple moving of methods from one class to another and not on higher level design patterns.
• It already said in section 5.2 that the discussed limitiations are for pattern recommendation approaches based on static code analysis techniques, only. Clearly, this leaves room for other, ptentially better approaches. We have nevertheless clarified in the paper that this is our current state of knowledge.
• As we are running out of space due to the suggested improvements, we have removed the discussion of agile approaches in section 5.2 as it is not too important here.

Comments on the feedback of the other reviewers will follow as soon as possible.

[ghost]

Please find our feedback to the comments of reviewer 3 in the following:

• work of Jafaar et al.: we discuss their work in section 2.

• Quantification of rules: the idea of coming up with threshold values is exactly to avoid "over-engineering" of systems by applying patterns all over the place. Based on these values it becomes possible to better judge whether a pattern should be introduced into the code, which after all still remains a human decision. We have rephrased various sentences throughout the paper to make that clearer.

• tools for pattern recognition: true, there are others as well (e.g. MARPLE), hence we rephrased that sentence so that Pattern4 is presented as one example.

• Typos etc: p1: -> thanks, we have fixed them all.

p2: "(admittedly not too many)" <- Please be specific. How many is "not too many"? -> around 70 refactoring examples, we have added this.

p5: "mini survey" <- What is that and how does it differ from a survey? -> we wanted to underline its small size, but since we mentioned the number of participants, we have removed "mini"

"On note on" -> "A note on" (?) -> Yes, I think we wanted to write "one" initially.

"distilled their code so that they were as easy to grasp as possible" <- How was "easy to grasp" operationalized? -> we have added a sentece explaining that we extracted the relevant switch statements.

[ghost]

Please find our reply to the remarks of reviewer 2 in the following:

• typos fixed, thanks for pointing them out
• moved definition of the term "design smell" to the introduction
• It should be obvious that..." (section 4.1) -> rephrased as it clearly important to be mentioned
• overhead of patterns in comparison to their benefit -> a difficult topic that clearly deserves to be discussed more prominently so that we extended section 2.1 as suggested (and added a pointer to the work of Jafaar et al. as suggested by reviewer 3)
• state vs. strategy pattern: at the time of writing it seemed like an important aspect to work out the difference between the two in patterns in terms of recognizing opportunities for them (and it still is an interesting challenge), however due to the limited space, we have removed it and plan to discuss it at another occassion in more detail.
• section 4 has been thoroughly revised so that it should be easier grasp now, there were indeed still some remainders from earlier version of the paper; thanks for pointing this out.
• "Admittedly, identifying design smells... & due to various pragmatic reasons" -> it is certainly an important aspect that problems in software become more expensive to fix the later they are detected (as e.g. described by Barry Boehm). However, this probably also needs to be discussed in more detail so that we removed these statements from the current paper due to the limited space available.
• open source in section 5.1 -> we are not sure whether we fully understand your concern? What we wanted to say here was that we have only applied the recommendation tool to open source systems (as opposed to closed source industrial systems that might be developed in a different way) and that we cannot generalize our results to the latter. We have revised this part of this section so that it now should become clearer
• state of the art -> we have thoroughly revised this section as well and added a number of additional and more recent references (especially a recent survey on recommendation approaches for refactoring opportunities)

[ghost]

We have uploaded a revised version of the paper to our initial repository here: https://github.com/swan17-contributor/paper/blob/master/SWAN17.pdf and created a pull request for the copy here.

Thanks again for all the helpful hints that helped significantly to make the paper more concise and convincing.