Lab 2: Method to your madness?

[ycoady]

Take any 2 questions identified in last week in Lab 1 and: a) refine them b) come up with a methodology (including tools, source code, data, steps, etc…) to answer the question c) demonstrate that the proposed methodology should be able to answer the refined questions on a given project (or projects!) by manual inspection and vigorous hand waiving...

[chrisjcook]

1 Original Question: "How is growth affected by coupling? Is it possible to analyze/visualize coupling in a software project?" (A) Refined Question: "Does the level of coupling in a project over the life of a software project influence the way it evolves?" (B) Methodology: Write a script that would iteratively move through the history of the master branch of a software project, and then use Snakefood to capture a frame of the dependency graph at that point in time. Compile all the frames into a video and run it concurrently beside a gource visualization. (C) Justification: By seeing the growth of a project next to the evolving dependency graph, relationships between between periods of high or low coupling, as well as periods of fast or slow growth could be discovered and analysed.

2 Original Question: "How does different team sizes and/or corporate backing (ie: structure) affect the growth and evolution of a software project?" (A) Refined Question: "Does the relative clustering of software developers in a software project on different areas of the code affect the health and/or long-term evolution of the project?" (B) Methodology: Analyse multiple Git repositories by looking at the commits by each developer, and classify the project on a scale of tightly-clustered vs. widely-spread clustering. Further analyse each project in terms of patches of legacy code (Gource), coupling (Snakefood), project activity (Gource), downloads (Github), and whether or not the project was abandoned (Github). (C) Justification: By comparing all of these metrics, conclusions could be drawn about the relationship between clustering of developers and the health of the project.

Chris Cook, Richard Claus, Sarah Nicholson

[paulmoon]

Paul Moon & Jian Guan & Jonathan Lam

What kind of errors can be detected via Static Analysis?

a) Refined question: Can the following errors be detected via static analysis tools?

• Syntax/type errors
• Unused code
• Concurrency bugs
• Security vulnerabilities
• Code style check (e.g. PEP8 for Python)

b) Method We can use an IDE to check syntax and type errors, unused code, and code style checks. Other static analysis tools such as ThreadSafe for concurrency bugs, and Cigital SecureAssist for security vulnerabilities can be used.

c) Loading a project into the IDE and running code analysis should be able to detect the aforementioned errors (example: IntelliJIDEA). Similarly, ThreadSafe and Cigital SecureAssist advertise that their software can respectively detect concurrency bugs and security bugs, so they should work.

Which parts of the system have had the most bugs? This could be related to number of commits that seem to not change the size of the repository.

a) Refined question: Can we analyze and visualize the number of bug fix commits for each component of the system?

b) Method Analysis: JIRA can be used to analyze the number of bugs for each component (assuming JIRA issues were documented well).

Visualization: If we can filter out the git log for only bug fixes, or somehow change the colour of bug-fix commits, then Gource can be used to visualize which components had the most bug-fix commits.

c) Using JIRA's advanced search and tagging system, one could search for bugs in certain components and the system as a whole. According to Gource documentation, colours can be assigned to each log message. If the .git log is edited such that bug fixes are assigned a specific colour, then Gource should be able to show the visualization that we want.

[Hoverbear]

Me (Andrew Hobden) and @fraserd (Fraser DeLisle)

On the average, do scripting languages have a different rate of errors/bugs compared to compiled languages?

Many scripting languages press errors into the runtime, while many compiled languages (particularly new ones) kick their errors at compile time.

Method: Pull a number of scripted/compiled projects with established histories and well-used issue trackers. Use Github's API to compare the number of issues against the number of LOC or other metrics. Preferably projects would be comparable in function (Eg a OS has more bugs than a web service usually)

Justification: We assume that when errors are found an issue will be created to reflect this. Also of interest are the time-until-fix (closed), and the amount of discussion created by such changes. A long-shot point of interest are how a fix to one error might propagate through the rest of the project.

Does having a geographically diverse team (as opposed to a local team) effect how the software develops?

Method and Justification: Use Github's API to look over projects and identify "Tightly" and "Loosely" coupled geographical development teams. Are there more comments on a "loose" team's issues, what is the sentiment on average? What is the average time till issue is closed? What is the overall development pace?

[eburdon]

1

orig: "Do developers repsond to user requests/feedback quickly?" a) Can we trace the time & content of a bug report/feature request (comment from user) to when it was implemented (responded to by developer(s))? b) We need of find out where the project's big report location is (some use GitHub-issues; Other may use Google, etc.). From there, parse the reports for time and relevant issue & file. To find it's implementation, we can look at both commit messages (e.g., "Fixed bug 29") and/or find the relevant file and any other files changed at the same time. You could also look at keywords in both issue report and commit messages (e.g., "Fixed blocky shadow") c) Go to project's issue page and find corresponding ticket number. Look for that reference number in commit messages. hand waving

2

orig: "Which sections of the code are modified most frequently" a) Can we track the number of times a specific section of one file has changed/been updated? b) Create a script to count the number of times the file itself have changed. Then run "diff" to find out where the code changed. From this, we can pursue statistics such as 1) percentage of commits to files modified [e.g., if a file is committed (with changes) 100% of the time, that file is unstable and must be constantly changed] 2) Type of change - major restructure of syntax. 3) Where changed - globals are consistent? Only specific functions need updating? c) On GitHub, we can view the history of a specific file. We can use this to verify the results of our script. We know it should work because every commit will have a time stamp associated with it and, even if a message is (foolishly!) not included, we can which files were modified and how.

[RyanMcDonald]

Question 1

Original Question: How can you analyze the code quality? This could be things such as code structure and adhering to standards, as well as good programming practices.

Refined Question: Can we find the number of common programming errors in the source code?

Method: Use the PMD tool.

Justification: PMD finds common programming flaws like unused variables, empty catch blocks, unnecessary object creation, and so forth. After running PMD, you should have a comprehensive list of all of the flaws in your source code that you can look at.

Question 2

Original Question: How can we determine the percentage of code coverage that our test cases cover? So as the software evolves, we can be confident that all of the code is being tested.

Refined Question: What is the percentage of code that our test cases cover?

Method: There are many tools that analyze code coverage for tests. EMMA is a free Java code coverage tool with numerous features.

Justification: These are sample reports that EMMA has generated. You can see all of the code coverage stats per class, method, line, or basic block.

• Ryan McDonald and Colin Knowles

[BrodyHolden]

Question 1

a) Are there certain components of a project which receive more developer attention than others? What causes this?

b) Pick a set of projects within the same problem domain hosted on github. Use the github web interface and other git analysis tools such as Gource to inspect which components of each project receive the most developer attention. Based on commit messages and issue tracking, extrapolate the cause of the distribution of developer attention. Compare and contrast the popular components between the set of projects.

c) This technique should be able to answer the first portion of the question since there will be a count of commits and issues related to each part. The causes could be shown from commit messages and discussion in issue tracking.

Question 2

a) Are there a certain types of components of a project which become abandoned? What causes this?

b) Pick a set of projects within the same problem domain hosted on github. Use the github web interface and other git analysis tools such as Gource to inspect which components of each project receive little or no developer attention. Based on commit messages and issue tracking, extrapolate why this occurs. Compare and constrast the abandoned components between the set of projects.

c) This technique should be able to answer the first portion of the question since there will be a count of commits and issues related to each part. The causes could be shown from commit messages and discussion in issue tracking.

Brody Holden & Kathryn Madrick

[Brayden-Arthur]

Question 1

a) How does the rapid expansion of a userbase contribute to the evolution of a software system? b) We can take any open source software that has documentation of their userbase, Then we can use a code visualization tool like Gource or Gitstats and try to show where these rapid expansions took place. We could also look at the amount of fork requests over time with our code visualization software to see if they correlate. c) The technique we have established would be able to fully answer the question as long as we can find data on the projects userbase.

Question 2

a) What languages are the most frequently altered? b) We can take a bunch of open source software projects and use the same visualization software as above to compare and contrast what kinds of files are altered or added most frequently. We would need to have a large variety of projects. We can then use R to see if the changes to one particular type are statistically significant. c) This should answer our question from a mathematical point of view since having statistic significance makes it so.

Brayden Arthur - Brandon Leech - Jorin Weatherston

[devinc13]

Question 1:

a) Refined question: How do the following attributes affect the number of bugs/errors in a software system - addition/modification of automated tests, size of the project, number of contributors to the project and the use of alphas/betas to find bugs before official releases.

b) Data will come from issue trackers in open source repositories (eg. JIRA, TestRail, Trac), inspection of commit history (seeing if automated tests are added/expanded), and the details from the open source repositories (number of contributors, size of project). We want to use this data to determine how and to what extent the variables affect the amount of bugs in the system. Finding the number of reported bugs could be measured by studying repositories with open JIRA instances (https://confluence.atlassian.com/display/JIRADL/Examples+of+Public+JIRA+Instances), and doing an advanced search for the time when bug tickets were opened (issuetype = Bug AND created < 2013-06-01 AND created > 2013-05-01). Or if the project uses issues on github, they can be searched using "bug created:2012-11-01..2012-12-01" to specify the dates of tickets created with the word bug in them, or "label:bug created:2012-11-01..2012-12-01" if that project uses a label to represent bugs.

c) Graphing the data collected with respect to time will be able to help draw conclusions as to what techniques help decrease the number of bugs, ideal numbers of contributors to certain projects, etc...

Question 2:

a) Refined question: Can we visualize the running/timing of multi-threaded code to help track down race conditions and thread issues?

b) We would need to use a tool similar to gource that would record data from a program as it's running, and create a visualization of the threads and their activities. Logging may need to be added to the code to generate the required data.

c) Looking at the visualization after running the program could help developers determine if events in different threads occur in the expected order, and it could help detect thread locks.

Devin Corrigall, Tyler Potter, Andrew Hansen

[gregnr]

Rabjot Aujla, Greg Richardson, Jordan Heemskerk

Questions:

How does a project’s development change after switching from one VCS to another?

a) How does the size and frequency of commits change (or doesn't) after switching from one VCS to another?

b)

1. Find projects that switch from one VCS to another (ie. Subversion to Git)
2. Identify date that switch occurred
3. Run a statistical analysis (custom scripts) that iterate through the commits of each VCS and measure the average commit size and frequency.

c) This would answer our question by giving quantitative values for both of our parameters (commit size and frequency) for each VCS and allow us to compare them.

How do existing maven dependencies hold up when implementing new features?

a) Do project dependencies stay the same, upgrade, get replaced, or get deleted when implementing new features?

b)

1. Pick a project to investigate that has sufficient dependencies.
2. Manually, or automatically detect when new features are committed, by using VCS logs or issue tracking system. Observe which dependencies were or were not modified. (ie. for maven, investigate changes in .pom files)
3. Gather statistics on changes.

c) If we are able to accurately determine the merge points for new features, and can target the changes in the .pom files during those times, the differences in the files will answer the question of how the dependencies are changing.

[Jsyro]

Question 1) a) How does the groupings of file types vs groupings of functionality affect the ability for the project to evolve?

b) Anaylze a large number of of projects to determine patterns in directory formatting. Then look for the relationship between the pattern and the nature of the re-factors.

c) With enough samples, patterns will start to emerge about the directory structures and how effective they are.

Question 2)

a) What does the the intensity and frequency of code re-factor say about the evolution of the software, is it evolving, or are these major issues that are having to be fixed after production?

b) Analyze code base revisions, pin-point revisions involving lots of add's and delete's. Look at the recent activity of those modules, try to understand the motivation of the large update. using dimensions of number of developer's committing updates and the frequency of updates from those developers. I wonder if there is away to quantify the amount of evolution that occurred on that commit.

c) If a large percentage of the development team is working on a portion of the project, with low number of commit's that may indicate a need for future maintenance. However a small number of developers that are very active on a new portion leading up to a re-factor may show that

Jason Syrotuck, Evan Hildebrandt, Keith Rollans

[mitchellri]

Question 1: How do software development methodologies affect the development time of a software system. a) Refined Question: What is the relation between software development methodologies, and development time of a software system? b) Methodology: The user would have to manually input the project schedule, or if there's already a schedule in a different software then use that as the input. From the schedule we could either try and determine the methodology, or have the user input it as well. The user should also post their approximate due date for the elements in the schedule. Then over time, take the element completion time(end day-start day)/estimated completion time(planned end day-start day) ratios for each element in the project, and visualize it. c) Justification: The ratio shows how fast they completed a task with relation to their planned finishing date. A visualization of this shows that relation over time, and the user input changes to the plan would/should be when the ratios make the most change.

Question 2: How does the addition/modification of automated testing software affect the number of bugs/errors in a software system? a) Refined Question: How does automated testing software affect the number of bugs in a software system? b) Methodology: I would imagine that you could create a general repository standard for testing files to sit, then you measure the files in that repository. Then you would use something like IntelliJ IDEA, or Pylint to measure the bugs and visualise the relations between the two. c) Justification: The repository for the testing files makes it much easier to track the files that they use to test, and allows the use on evolving software. After using something like IntelliJ IDEA, or Pylint to track the bugs you have all the data you need, and all you need to do is visualize it. The visualization could clearly show the relation between the two.

Mitchell Rivett

[RobertLeahy]

Question 1

Original: When styles, paradigms, et cetera change, either because the original style/paradigm was always bad (think tightly coupled code, a lot of global state), or trends/best practices change (the historical shift from procedural to object oriented programming), how do long-lived code bases react/what mechanisms does this reaction use?

Do they stick with the old style/paradigm? Do they rewrite all their code? Do they create wrappers? Which of these (in the long run) results in the lowest "cost", where "cost" is defined as new code that has to be added, bugs created, et cetera, and how long does each reaction take to "pay off" (if they ever "pay off" at all)?

Refined: When a project adopts or accepts a new paradigm, does this lead to better or worse code?

Methodology/Justification: Find a point where a project adopted or accepted a new paradigm (e.g. GCC recently switched to C++) and working outwards from that point examine trends in:

1. Number of bugs reported
2. Number of bugs fixed
3. Number of LoC added

If fewer bugs are reported after the transition, or if they are fixed more rapidly, or if substantially fewer LoC are added going forward from the adoption/acceptance, it can be considered to lead to better code (the code is more modifiable (fewer changes necessary moving forward), maintainable (less time to fix bugs), and correct (fewer bugs)).

Question 2

Original Question: How does the maturity of a language effect the speed and reliability of code bases?

Refined: How does the maturity of a language effect the reliability of code bases?

Methodology/Justification: Examine two code bases which perform similar functions but are written in languages with differing levels of maturity. Compare the number of reported bugs for each. Assuming you control for the use/market share of each (more highly used products will tend to have bugs discovered more frequently) you should be able to extrapolate the answer from this: Reported bugs are flaws in the software, and are therefore a measure for reliability, accordingly a difference in bugs between the two code bases could reflect on the underlying languages and/or their maturity.

[davidsjohnson]

@gregnr It looks like the Go Language has moved from Mercurial to Git: https://groups.google.com/forum/#!topic/golang-dev/sckirqOWepg%5B1-25-false%5D