Measuring Software Quality

[rowlando]

There are many ways to measure software quality. A recent article on IT Revolution called Measuring Software Quality digs into this topic.

Software Development teams strive (or should be) many "ilities", such as reliability, maintainability, scalability, agility, serviceability, security, etc... and software quality (or lack of) helps us achieve these traits.

The white paper explores the 2 sets of leading indicators that help us with software quality (1) code (2) process.

In regards to code quality, particularly static analysis, I want to explore what tools are currently in use or we should be using to help us get clean and stay clean (maintainable code).

Clean as you code We have some teams using SonarQube to help them own quality in new code. SonarQube adds a time dimension to code quality, making it easy to see when quality is slipping because you're ignoring static analysis, e.g. code smells, linting violations, code complexity, etc...

This breed of tool alerts us when it's time to refactor code, a crucial job as a developer. And by having a metric to point to, it helps us articulate to non-developers when we might need to do a bit more maintenance than usual.

Old code We have lots of older code that might not have benefitted from good software maintenance activities. We don't want to necessarily clean all of our old code. This type of investment must make strategic sense.

---

❓ With this context in mind, here are a couple of questions to kick start a discussion: ❓

• What tools or practices are you currently using in your team to measure software quality?
• What tools would you recommend we adopt or trial?
• How would describe your team's approach to maintenance

[alanmaydwell]

I've had a hasty look and have a couple of quick points:

1 Software Quality vs Code Quality

The Measuring Software Quality linked article refers to software quality but the detail is more about code quality. These aren't really the same thing.

Is it worth making a distinction between these two?

OK, if you've bad code you've also likely got bad software but it's possible to create a product that has clean maintainable code that adheres to good practices but is nevertheless completely horrible to actually use.

2 Here's a post with some interesting points

This blog post has some interesting and cautionary points to say about measuring quality: https://www.satisfice.com/blog/archives/487091

[rowlando]

Totally agree we're talking about code quality here @alanmaydwell. The white paper separates out the leading indicators for software quality into two parts (1) code and (2) process. The process bit covers "testing for customer expectations", which I think for us manifests itself in agile delivery, e.g. ensuring we understand the problem we're trying to solve, building or prototyping the smallest vertical slice, then getting feedback from user, and iterate.

I'll update the issue description to make the distinction clearer between code quality and software (or product/service) quality.

[forceedge01]

We've recently introduced SonarQube in the CCMS PUI team's workflow.

For us, in addition to the above mentioned valid points - it is also about controlling software entropy and the effects it has on the team. How the team can begin to value and learn from tools that actually help them become better developers sensitive to these issues even when such a tool is not in play. One only solves a problem when one can see it - and SonarQube for us is primarily a tool that surfaces issues in real-time for us to react to.

Whilst tools like SonarQube can have a massive impact on the code quality, it probably would have an impact on the engineers engaging with a tool like this. It also reduces the conversations in pull requests and saves both parties a lot of time. It doesn't tell you just of a problem, but why it is a problem which leads to personal growth.

More to the point here of which tools we use:

Here is SonarQube in action with deeper integration in pull-requests

And here is a bit of custom implementation that fetches data from SonarQube and posts it on our pull requests.

The above stats are overwhelming and we're going to have a discussion whether we should surface all these statistics on a pull request.

What tools would you recommend we adopt or trial? I've seen SonarQube integration to the point where it can create pull requests with recommended fixes on a click of a button. I am not sure which version it was, but maybe worth looking into it to save additional time.

In addition to the tools - I would advocate for better integration - the implementation model should encourage information finding users rather than users finding information.

In addition to static analysis tools - automated tests have a part to play here. TDD is a good development process to get going with which hugely impacts code quality and design. Having a SonarQube quality gate/profile that ensures a good level of automated test coverage could be an idea to try out.

Another tool that may be worth a trial is Codacy https://www.codacy.com/.

How would describe your team's approach to maintenance Our technical debt stands very high at the moment. Our strategy is to clean as we code much like you've mentioned. We are not keen on refactoring a great deal on legacy applications due to the cost. However we are still working out as a team what a feasible and sensible approach would exactly look like.

On a personal capacity I've started to use SonarCloud which is free for public repositories and paid for private repositories.

Whilst SonarCloud is free like SonarQube community edition - it isn't the same offering. SonarCloud comes with a few more perks such as an improved dashboard, multiple branch integration and pull request decorations. Still discovering.

Apologies if I have rambled on about irrelevant subjects.

EDIT: The tool mentioned above (raising PRs for suggested changes) which can help with software quality is https://scrutinizer-ci.com/

[your]

Thanks for sharing the article, @rowlando. I have a few considerations first, then I'll answer the three questions.

• I agree with @alanmaydwell (really loved the article you shared too, thanks), and I'll refer mostly to code quality
• I believe that any quality measurement is going to be quite subjective to the team that owns the software or the wider team the developers belong too (i.e. language community). Code is more folklore than science, and it can only be analysed in the context of a community. Code is also generally written according to business requirements, rarely well gathered or resistant to change. Because rarely applications are built to "be maintained", but rather to "be delivered". Code acts like a sponge, and often times the place where all the focus goes, simply because it's the last step. This is probably already off-topic, guess what I'm saying is that code quality is another side of a bigger coin to be aware of
• I loved the "clean as you code" paragraph, and agree on the fact that having a metric is very helpful to communicate needs to a non-technical stakeholder. It remains a pseudo-objective measurement, so I guess that a strong leadership would also need to play a crucial role and create empowerment from within the development team to make all the required room for refactoring work (if part of a strategy)

What tools or practices are you currently using in your team to measure software quality?

None in particular, our codebases are all classified as legacy and as you say, it wouldn’t strategically make too much sense (at least for the moment) investing time in forcing something on codebases there were really written without any code standards in mind.

Although, we’d normally have a technical group discussion to agree on how to approach any new code change or addition based on what we’d find already in the codebase.

As an example, we’ve established that one of the major factors that impact code quality in one critical PL/SQL package is the presence of hardcoded values, which led to an extremely high cyclomatic complexity. This makes changes and testing a titanic task. Our approach would be to normally reduce this complexity by moving static values to tables that represent them as an entity and delegate pieces of complexity to different procedures. We cannot try to fix everything, so static analysis wouldn't necessarily help us.

What tools would you recommend we adopt or trial?

I can speak for the Ruby ecosystem only and have always found Rubocop+Brakeman pretty much enough for the average project, maybe with Danger. But I know some people disagree or prefer more sophisticated IDE integrations / GUIs, so I don't really have a recommendation here.

How would describe your team's approach to maintenance

Conservative, but we’d still try to discuss whenever it’s possible to improve something (especially if we keep on ending dealing with the same piece of code/changes). It probably needs to evolve in a more strategic and well-defined set of rules.

[mpw5]

What tools or practices are you currently using in your team to measure software quality?

Similar to Giuseppe's comment for Ruby ecosystems - on Apply/CFE we're using a combination of Rubocop for linting, Brakeman for security analysis, and Dependabot to prevent dependencies from going out of date, and generally we feel like that gives us a good enough level of code quality. But I'm aware that none of those provide metrics - it would be interesting to have an objective measurement of quality.

What tools would you recommend we adopt or trial?

We'd potentially be interested in trying something like Sonarqube/Sonarcloud to see what it gives us. One alternative we've discussed using within the team is Code Climate which is similar. There have been concerns that using these can lead to writing code to please the tool, rather than good code. There's also potential for conflict between a new tool and our existing tools (eg code climate and rubocop having different opinions on what is 'right'). But if there's the possibility of configuring it to fit with our existing workflow with minimal pain then I think we'd be happy to trial something.

How would describe your team's approach to maintenance

Our approach mentioned above is essentially 'clean as you code' - static analysis tools help us write good code, and we can't release anything that doesn't meet those standards. We also regularly include tech debt tickets in sprints to address other maintenance issues that arise. The rest of the team is supportive of the need to improve/refactor code even if it's not delivering user need.

@ministryofjustice/laa-apply-for-legal-aid - feel free to chime in if you have other views ☝️

[msahilan07]

What tools or practices are you currently using in your team to measure software quality?

Crime Apps has mainly 2 types of applications same as many of the other service areas within LAA. 

• Modern Services 
• Legacy services. 

As stated above, we have legacy code with poor tooling in terms of software quality measures & these will need to be migrated with our long-term strategy in mind. 

However, We use Sonarqube on Java-based applications & the dependant bot is enabled on the modern services. Ruby Applications are built with Rubocop for the linting.  

Sonar gives us various metrics to measure our code quality as acknowledged.  This tool also helps us to learn & develop good coding practices as well as resolves some of the debates during code review. 

The beauty of Sonarqube is that we can integrate various plugins to achieve what we want. 

Crime Apps not only wants to measure the quality of the code, but we want to understand the quality of our tests (not just with line coverages).

We have integrated a mutation testing framework called  PI Test (https://pitest.org/) which gives good visibility on how our tests are doing. 

Jacoco (https://github.com/jacoco/jacoco) has been plugged in for code coverage.

Sonar Qube is currently enabled during our CI/CD process & here is a screenshot of how one of the modern applications looks like. 

Here is the snapshot of our mutation tests. 

To put this in perspective, The mutation framework has created 269  mutations (modifications to the source code) &  251 mutations are caught & killed by the tests, which gives 93.3 % mutation coverage. 

Here is an example of how the framework mutates the source code. 

What tools would you recommend we adopt or trial?

We would prefer to migrate the existing sonar framework to the SAAS version which will allow us to integrate various frameworks such as security framework OWASP (https://owasp.org/) etc.

This will also allow us to move towards DevSecOps model.

However, having spoken to security folks @Umar Qureshi  & @SimonHeron during last year's security review, Vera Code (https://www.veracode.com/) will be another choice we would love to explore.

How would describe your team's approach to maintenance? 

We would describe our approach as 'Celan as we Code' & 'Leaving the things better than we found them'.

We understand the importance of software maintenance & advocates not only within the team, but we have conversations with the stakeholders to make sure they understand it as well. This means we allocate time & effort for the software maintenance regularly & compromise with them to make sure we do the right thing in the right way. 

We use static tools such as Sonarqube & Our code review process to make sure we operate within clean coding principles such as SOLID (https://en.wikipedia.org/wiki/SOLID)  & KISS (https://en.wikipedia.org/wiki/KISS_principle) etc.

[rowlando]

Let me try to summarise what tools seem to be in use so far.

Tool	Types of static analysis	Language / Framework
Sonarqube Community Edition	Code Quality SAST	multi-language
Brakeman	SAST	Ruby on Rails
Rubocop	Code Quality	Ruby

Code quality - linting, code formatting, code smells, complexity SAST - Static Application Security Testing

Some tools are much more fully featured, e.g. Sonarqube. They provide a user interface, charts showing how quality metrics have changed over time, integrate well with GitHub workflow, e.g. decorate pull requests, amongst other things.

GitHub Advanced Security We also have teams in LAA Digital and across D&T who are starting to make use of GitHub's Advanced Security features, some of which are free for public repositories. Apply for Legal Aid has set up Brakeman to run as a GitHub Action and reports its results in the Security tab of the GitHub UI.

GitHub has its own security scanning tool, called CodeQL Analysis. It currently supports C, C++, C#, Java, JavaScript, TypeScript, Python, and Go (Ruby coming later in the year. CodeQL can be set up for public repos in two clicks - see https://github.com/ministryofjustice/laa-apply-for-legal-aid/security/code-scanning/setup for an example.

You'll also notice other third party tools available that do the hard work to run a tool with GitHub Actions and publish results to the Security tab by uploading a SARIF file.

GitHub are really good at building developer-first tools and security is no exception. See this video for more info. There are tools like Veracode that provide deeper analysis but this might be at the cost of developer experience.

I'd be keen to get more people's thoughts. And do you think there are any specific pieces of guidance we can pull out of this discussion yet?

[calv-w]

One way of increasing quality is to make the code more readable, by adding comments to explain what is going on. I also make sure comments for code changes reference JIRA ticket numbers and explain why the change is being made if it is a bug fix.

My area is ETL (Extract, Transform, Load) code. Another thing I do to improve the quality of my code and to make debugging of data problems easier, is I ensure that I have batch ids (MI load batch numbes), warehouse created & updated dates and source created/updated dates on all rows of data - these are populated in the ETL code.