Evaluate current and other 3rd party quality tools

[brandonrapp]

Overview

• There are several opinionated SaaS tools that we can use to enforce code quality standards.
• Pull request reviews are time consuming and often slow down teams while providing little benefit

Goals

• Reduce the burden of code reviews on the platform group
• Improve code quality with explicit and automatically enforced standards
• Identify the recommend quality standards for the platform to measure
  • Security
  • Stability
  • Scalability
  • Maintainability

Potential tasks

• Review current and other third party offerings
• Determine constraints for using SaaS products for code review
• Create a plan for running a few options through a trial
• Recommend a 3rd party tool or tools if more than one is required (Github tasks for PII)

Code quality tools to evaluate

• code climate (currently in evaluation)
• sonarcloud.io: https://sonarcloud.io/about
• Codacy https://www.codacy.com/
• Crucible
• Codefactor.io
• Helixswarm
• Smart bear collaborator
• Code scene empear
• sonarqube
• Github Tasks

[jhonnyoddball]

CODE QUALITY TOOLS

The static code analysis is an effective tool to have a good overview of the project code quality and to be able to predict potentials issues that can arise. We can understand code quality as everything related to code consistency, readability, performance, test coverage, vulnerabilities.

The idea here is find a tool that will add another stage to our Continuous Integration process.

Basic requirements:

• Full support for Javascript since it is the main programming language in our projects.
• Integration with GitHub in order to add inline comments and code quality checks in the Pull Requests
• Good static code analysis with an extensive set of rules
• Cloud-hosted. We want to focus on software development and not spend time on maintaining a server/tool
• Define code quality goals and fine-tune thresholds for each code quality measure (i.e Test coverage over 80%, No more than one critical issue)
• Well documented

Candidates:

• Code Climate
• Codacy
• SonarCloud
• CodeBeat
• Datree
• SonarQube [Server]

[jhonnyoddball]

Code Climate

Out of the box, the code analysis is not too accurate when you run the analysis for the first time. By default, it doesn’t have more rules than the ones related to Complexity (method count, file length, cognitive complexity, etc…) and duplicated code.

The only two engines are ESLint and Node Security, and since ESLint is something we can easily integrate into our workflow and validate via CI, it’s not so useful.

The maintainability is graded from A to F according to various measures (mainly the number of code smells and code duplications) Test coverage is also graded from A to F based on the overall percentage.

Cons:

• No distinct types for total issues number.
• No detailed description of the issue, only a header with source code.
• No issue searching/filtering, just a paginated list with all of them.
• No custom rules.
• Uses only ESLint.

[jhonnyoddball]

Codacy

It has the best UI of all analysed tools with a very clean user interface

The code quality measures are grouped into 8 categories: code complexity, compatibility, error-prone, security, code style, documentation, performance and unused code.

It also allows you to define goals for your projects, either per file or per category

Regarding Pull Requests, you can configure multiple thresholds giving the most advanced configuration you can find among these tools. It has the best integration with your github

its static code analysis for Javascript is not as good as the others tools. This has to do with the fact that in order to check the code quality issues in Javascript, it uses ESLint, which is something we already included in our workflow.

Duplicated code analysis works pretty well but it’s not considered for grading

Pros:

• Great and intuitive UI.
• The possibility to define issue-based goals to improve the codebase.
• Checking lots of security issues (like assigning strange values to private APIs which may lead to unexpected app behavior).
• Huge flexibility thanks to disabling/enabling patterns or whole packages and ignoring certain patterns in the selected files.

Cons:

• Incomplete documentation in some parts (some images are hard to read and the amount of information is insufficient sometimes).
• No custom rules.
• Uses only ESLint.
• Reports tons of noisy, irrelevant "quality" metrics

[jhonnyoddball]

SonarCloud

SonarCloud created a lot of their algorithms from scratch. They cover a large number of languages but focused mostly on Java.

SonarQube by far the most powerful code quality tool with a lot of measures and filters but that leverages a more complicated UI and configuration. All these features get lost on it’s cloud version SonarCloud

SonarCloud is meant to be integrated with cloud solutions like GiHub.com or BitBucketCloud.

Pros:

• Good UI
• Uses SonarQube algorithms

Cons:

• Lack of documentation
• Can’t manage rules
• Can’t edit threshold (all or nothing)
• Seems extremely Java-focused, despite advertising support for many languages

From SonarCloud Team

For custom rules, this is unfortunately not possible on SonarCloud - yet. (and I don't know when this is available - this is not in our short-term list)

[jhonnyoddball]

Code Review Tools Comparison Summary

Supported Languages and Technologies: CodeClimate: Ruby/Rails, JavaScript, Python, PHP, Swift, SCSS/CSS, Go, CoffeeScript, Apex, Ember, ESLint, Haskell, Haxe, RubyMotion, Vim Script; Codebeat: Ruby, Javascript, Python, Java, Swift, Go, Typescript, Objective-C, Kotlin, Elixir; Codacy: Ruby, JavaScript, Python, PHP, Java, Swift, CSS, TypeScript, CoffeeScript, Scala, C/C++, Dockerfile, SASS, Shell Script; Sonarcloud: 23 languages: Java, JS, C#, C/C++, Objective-C, TypeScript, Python, ABAP, PLSQL, T-SQL and more.

Measuring Tools: CodeClimate: many existing, open-source tools like Rubocop, Brakeman, CSS/SCSS Lint, ESLint, Flog, etc. Codebeat: their own algorithms and implementation written from scratch. Codacy: many existing, open-source tools like Rubocop, Brakeman, CSS/SCSS Lint, ESLint, Flog etc. Sonarcloud: their own algorithms and implementation written from scratch.

Documentation: CodeClimate: very good and comprehensive; Codebeat: still some things missing; Codacy: not bad, not too much text and some images are not clickable so the readability is limited; Sonarcloud: Good enough but it needs more details

API: CodeClimate: yes, still in beta version; Codebeat: yes, simple but usable; Codacy: yes, not described perfectly in their docs; Sonarcloud: yes, very simple

	CodeClimate	Codebeat	Codacy	SonarCloud
Coverage report	yes	yes	yes	yes
Security analysis	yes	no	yes	no
Team per project	yes	yes	yes	yes
Github PR integration	yes	yes	yes	yes
Slack integration	yes	yes	yes	yes

[jhonnyoddball]

After evaluating all the 3rd party code review tools. We have found that all of them use ESLint in the backend for JavaScript checks and code coverage. This is something we are already doing. We are using a variety of plugins along with ESLing for code checks.

The only one that caught my attention is SonarQube since they implemented new advance rules from scratch. Fortunately, SonarQube shares their new JavaScript rules publicly. We will evaluate SonarJS for a possible addition to our current ESLint set up.

In conclusion, Since any of these 3rd party sites will not provide us with any extended benefits, I think it's best to switch all our efforts and try to improve and extend the ESLint coverage.