Like the road sign that is "Not In Use" most of the code being written never gets executed.
There are a few obvious issues with this:
Code coverage tells you when code you have written is being executed so you can decide if un-covered lines are superfluous (and can be removed) or require additional testing.
The rest of this page will focus on practical usage example, so if you are completely new to Code Coverage we recommend you read the wikipedia article: https://en.wikipedia.org/wiki/Code_coverage first.
Istanbul is a code coverage analysis script you run when executing your unit tests: https://github.com/gotwarlost/istanbul/ we like it because it's simple and prints out nice html reports (see below)
We prefer to install istanbul as a "devDependencies" in each of our projects:
npm install istanbul --save-dev
to check if the installation worked, (copy-paste and) run the following command in your terminal:
node_modules/.bin/istanbul help
For our first example create a file called test.js
.
vi test.js
type (or copy-paste) the following code in the test.js
file:
x = 42;
if(false)
x =-1;
Now run the istanbul command to generate a coverage report:
node ./node_modules/.bin/istanbul cover test.js
Alternatively you can insert the line
"coverage": "istanbul cover ./test.js"
into the scripts section of your package.json
and run the following command in your terminal:
npm run coverage
This will create a directory in your project called coverage
where you will find the generated coverage reports.
In our case:
learn-istanbul/coverage/lcov-report/learning-istanbul/test1.js.html
If you open the test1.js.html file in your web browser you will see a visual coverage report:
Istanbul gives us four code coverage metrics:
if/else
).
This metric tells you how many of your branches have been executed.when you click test.js
to view the coverage for the file you see:
Two things to note in the example above:
false
is always false!This may be a trivial example but it shows exactly where the useless code is.
Try executing the mischief.js
file by running npm test
:
What is wrong with the following picture?
There are plenty of developers/organisations that can only dream about getting 96% code coverage! and yet when we inspect the detail, there's something big slipping through the net!
We have 100% functional code coverage, but only 50% "Branch" Coverage. This means one or more conditional execution branches is not being executed.
Most of the time it will be something innocuous but what if a disgruntled person slipped in something like:
if(employee.status === 'terminated' && employee.left - today() > 90) {
selfDestruct();
}
The 97% Coverage is not looking so hot anymore ...
What if we add a Test that follows the branch containing the rogue code? We reach our mythical 100% Coverage:
And if we simply allow this code to be promoted without further checks, the rogue code will be in production and soon forgotten.
The solution here is to not rely (solely) on tools such as Istanbul to check code. Its essential would advocate a separation between the people writing the tests and the developers who write the code.
And there is still no substitute for Code Review!
Knowing the coverage locally is nice, but it's way more useful to a team to track coverage over time using an independent service. Thankfully, there are a few you can chose from:
We have used all the services and found that Codecov i s the clear winner for 3 reasons:
Sign-up to use Codecov (Free) using your GitHub account: https://codecov.io/
Select the repository you want to track coverage for from your list.
Add a few lines to your CI configuration file e.g:
before_install:
- pip install --user codecov
after_success:
- codecov --file coverage/lcov.info --disable search
This installs the Codecov reporter tool on your CI and sends the lcov.info
report
(which gets generated by Istanbul) to Codecov where it's kept safely.
Real-world example in
.travis.yml
file: https://github.com/dwyl/hapi-auth-jwt2/blob/master/.travis.yml#L7-L10
Now when you create a pull request your CI will send a coverage report to Codecov and Codecov will leave a comment on the PR:
Yes, it's "possible" to have a "large" project with 100% Test Coverage. (ALL DWYL projects do!)
So you can see at a glance if new code is being added without corresponding tests ...
When the coverage is lower the Pull Request "fails":
Note: if you want to prevent people (your team) from creating Pull Requests with less than an agreed level of coverage (e.g 100%!!), add a
coverage
checking script and correspondingpre-commit hook
. More on pre-commit hooks: https://github.com/dwyl/learn-pre-commit
Click on the Settings
tab for your chosen repo, click on Badge
and click
Copy
e.g:
Then paste the markdown into your Readme.
More about badges: https://github.com/dwyl/repo-badges
Note: DWYL is not financially affiliated with Codecov, but we :heart: @stevepeak and the product he has built. They are focussed on doing one thing really well and have innovated a lot in code coverage tracking making developers lives much better.
Istanbul (the JavaScript Code Coverage tool) https://github.com/gotwarlost/istanbul should not to be confused with istanbul the desktop screen recorder, they are totally diferent animals! Shame about the name collision... :-(