A test coverage library for Emacs Lisp.
A few important notes about undercover.el
:
edebug
and can have some issues with macro coverage;See the combined usage example and buttercup integration example samples for more information.
Add undercover.el
to your Cask file:
(source gnu)
(source melpa)
(package-file "awesome-package.el")
(development
(depends-on "undercover"))
Before invoking load
or require
with your package in your test runner (test/test-helper.el
/ features/support/env.el
/ etc),
call undercover
with wildcards that will match your package's source files:
(when (require 'undercover nil t)
(undercover "*.el" "awesome-extensions/*.el" (:exclude "awesome-examples.el")))
(require 'awesome-package)
Add your repository to a coverage reporting service, such as Coveralls or Codecov.
Activate undercover
plugin in your Eldev file:
(eldev-use-plugin 'undercover)
When running tests on CI server, make sure not to do it in packaged or byte-compiled mode. Or do it twice: once with your project loaded as source code, once as a package. The reason is that undercover.el
doesn't work with byte-compiled files.
Add your repository to a coverage reporting service, such as Coveralls or Codecov.
See relevant documentation on Eldev's own page for more information.
Eask is very similar to Cask; anything that applies to Cask will apply to Eask
Add undercover.el
to your Eask file:
(package-file "awesome-package.el")
(source "gnu")
(source "melpa")
(development
(depends-on "undercover"))
Before invoking load
or require
with your package in your test runner (test/test-helper.el
/ features/support/env.el
/ etc),
call undercover
with wildcards that will match your package's source files:
(when (require 'undercover nil t)
(undercover "*.el" "awesome-extensions/*.el" (:exclude "awesome-examples.el")))
(require 'awesome-package)
Add your repository to a coverage reporting service, such as Coveralls or Codecov.
See relevant documentation on Eask's own page for more information.
undercover
supports a wide range of combinations of code forges / CI providers / report formats / upload methods.
Some common combinations are documented below:
GitHub Actions |
Travis CI |
(other CI) | |
---|---|---|---|
Coveralls |
• With GitHub action • With built-in uploader |
• With built-in uploader |
• With built-in uploader |
Codecov |
• With GitHub action • With bash uploader |
• With bash uploader |
|
(other coverage service) |
• Manual configuration |
• Manual configuration |
Steps:
Add the Coveralls GitHub action to your GitHub Actions workflow YAML file, after your test invocation.
To support matrix builds, add a final job with parallel-finished: true
, as described in the action's documentation.
Invoke undercover
with (:report-format 'lcov) (:send-report nil)
.
A complete minimal example (using ert + Cask + ert-runner) can be found here.
You will need to export the GitHub Actions access token into the environment.
To enable Coveralls parallel builds, set COVERALLS_PARALLEL
in the shell environment,
and add a final job with if: always()
which pings the webhook.
Here is a complete example:
on: [ push, pull_request ]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
emacs_version:
- 25.3
- 26.3
- 27.1
- snapshot
steps:
- uses: purcell/setup-emacs@master
with:
version: ${{ matrix.emacs_version }}
- uses: conao3/setup-cask@master
- uses: actions/checkout@v2
- name: Test
env:
COVERALLS_FLAG_NAME: Emacs ${{ matrix.emacs_version }}
COVERALLS_PARALLEL: 1
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
cask install
cask exec ert-runner
finalize:
runs-on: ubuntu-latest
if: always()
needs: test
steps:
- run: curl "https://coveralls.io/webhook?repo_name=$GITHUB_REPOSITORY&repo_token=${{ secrets.GITHUB_TOKEN }}" -d "payload[build_num]=$GITHUB_RUN_NUMBER&payload[status]=done"
Alternatively to exporting GITHUB_TOKEN
, you may instead specify COVERALLS_REPO_TOKEN
, as with any other CI service.
No configuration necessary.
To enable Coveralls parallel builds, set COVERALLS_PARALLEL
in the shell environment,
and configure the web hook as described in the Coveralls documentation.
undercover.el
has basic support (for reading and parsing relevant environment variables, such as build ID) for the following CI services:
Detected values may be overridden by setting the following environment variables:
UNDERCOVER_CI_TYPE
UNDERCOVER_CI_NAME
UNDERCOVER_COMMIT
UNDERCOVER_REF
UNDERCOVER_PULL_REQUEST
UNDERCOVER_BUILD_ID
UNDERCOVER_BUILD_NUMBER
UNDERCOVER_JOB_ID
UNDERCOVER_JOB_NUMBER
UNDERCOVER_JOB_NAME
See the documentation of undercover--detect-ci
for a description of the semantics for these variables.
For CI services which are not "magically" supported by Coveralls,
you will need to set the COVERALLS_REPO_TOKEN
environment variable
before running tests, for example:
$ COVERALLS_REPO_TOKEN=<your-coveralls-repo-token> cask exec ert-runner
Consult the Coveralls documentation for details.
The token should not be made public, so it should be placed in the CI service's secrets store.
Fields in the submitted Coveralls report may be overridden using standard environment variables:
COVERALLS_SERVICE_NAME
COVERALLS_REPO_TOKEN
COVERALLS_SERVICE_NUMBER
COVERALLS_SERVICE_JOB_ID
COVERALLS_SERVICE_PULL_REQUEST
COVERALLS_PARALLEL
COVERALLS_FLAG_NAME
COVERALLS_RUN_AT
See the Coveralls API reference for a description of these fields.
Steps:
Enable the Codecov app for your account / organization / repository.
Add the Codecov GitHub action to your GitHub Actions workflow YAML file, after your test invocation.
Invoke undercover
with (:report-format 'codecov) (:send-report nil)
.
A complete minimal example (using ert + Cask + ert-runner) can be found here.
Codecov is supported in combination with their bash upload script.
In your test runner:
(undercover "*.el" (:report-format 'codecov)
(:send-report nil))
And in your pipeline (.travis.yml
or equivalent):
after_success:
# Upload coverage
- bash <(curl -s https://codecov.io/bash)
If the coverage service supports coverage reports in the LCOV, SimpleCov, or Coveralls file format,
it should be usable with undercover
as follows:
undercover
, set :report-format
to an appropriate compatible format:send-report nil
:report-file
to the location where the report file should be savedSet the report-file
option to change the report file location:
(undercover "*.el" (:report-file "/tmp/local-report.json"))
undercover.el
will try to merge new report with existing one.
Set the send-report
option to nil
to disable uploading the coverage report to an online service:
(undercover "*.el" (:report-file "/tmp/local-report.json")
(:send-report nil))
Set report-format
to use a different format for the report file:
(undercover "*.el" (:report-file "coverage/.resultset.json")
(:report-format 'simplecov)
(:send-report nil))
See the documentation of the undercover
function for more details.
Set the UNDERCOVER_FORCE
environment variable to calculate coverage even when not running on a CI:
$ UNDERCOVER_FORCE=true cask exec ert-runner
Alternatively, use (setq undercover-force-coverage t)
before calling (undercover)
.
Set the UNDERCOVER_CONFIG
variable to configure undercover.el
via the environment:
(when (require 'undercover nil t) (undercover))
$ UNDERCOVER_CONFIG='("*.el" (:exclude "awesome-examples.el"))' cask exec ert-runner
With Eldev generating local reports is very easy:
$ eldev test -U local-report.json
Option -U
is the short form of --undercover-report
and is only available if the plugin is activated (see above). Option -u
(--undercover
) lets you configure the library from command line. For more information see Eldev's documentation.
Selecting which exactly files you want undercover.el
to instrument is not possible from command line: these always default to all .el
files in main
fileset. However, you can adjust variable eldev-undercover-fileset
in file Eldev
if you need to change that for some reason.
You can generate a simple coverage summary report using the 'text
report format:
(require 'undercover)
(setq undercover-force-coverage t)
(undercover "*.el" (:report-file nil) ; or a text file to save the report to
(:report-format 'text))
Install coverage-mode
Create the coverage
directory in your project root
Configure undercover.el
as follows:
(require 'undercover)
(setq undercover-force-coverage t)
(undercover "*.el" (:report-format 'simplecov)
(:send-report nil))
Run your tests
Open a source file, and enable coverage-mode
.
You may need to teach edebug
how to instrument the affected macros.
See "Instrumenting Macro Calls" in the Emacs manual
and the documentation of def-edebug-spec
for more details.
The problem may be due to edebug failing to parse the file. Follow the instructions under the error message.
Try the following:
*.elc
) of your projectert-runner
does not load your project files (your project's .ert-runner
should use -L
instead of -l
for files you want to measure coverage of)