Visualize result

[DamienCassou]

While trying to improve code coverage, it would be useful to see what is covered and what is not covered by visualizing the report inside emacs with colors.

[Fuco1]

Hi @DamienCassou I just wrote this code exactly for this purpose.

It has some minor issues still but I think we can get some support for local execution into the trunk (along with this patch maybe).

First issue is that you have to specify undercover--send-report to be nil when you run tests, I do this like so:

cask exec buttercup -L tests -L site-lisp --eval "(setq undercover--send-report nil)" tests

Then read the docstring on my-display-coverage to see how to set undercover--report-file-path. When you run the tests also make sure the envvar TRAVIS is set.

(require 'undercover)

(require 'dash)
(require 'ov)

(bind-key "C-c C" 'my-toggle-coverage)

(defun my-toggle-coverage ()
  "Toggle display of undercover coverage report in current buffer.

If coverage is displayed, hide it, if not, show it."
  (interactive)
  (if (ov-in 'type 'undercover-report)
      (my-hide-coverage)
    (my-display-coverage)))

(defun my-display-coverage (&optional report-file)
  "Display undercover coverage report in current buffer.

Before you call this function you should have manually run the
test suite over your project.

It is a good idea to set the value of
`undercover--report-file-path' locally for your project.  Use
`add-dir-local-variable' to store the report in the project
directory so that it won't get overwritten by a report from
anohther project.  You must then set the same value to the Emacs
instance that runs the tests, for example with

  --eval \"(setq undercover--report-file-path \\\"$PWD\\\")\"

runtime option."
  (interactive)
  (-when-let* ((root (locate-dominating-file (buffer-file-name) "Cask"))
               (data (json-read-file (or report-file
                                         undercover--report-file-path)))
               ((&alist 'source_files source-files) data)
               (this-file (-first
                           (-lambda ((&alist 'name name))
                             (equal (expand-file-name
                                     (concat root "/" name))
                                    (buffer-file-name)))
                           (-concat source-files nil)))
               ((&alist 'coverage coverage) this-file)
               (line-number 1)
               (lines-covered 0)
               (lines-coverable 0))
    (ov-clear 'type 'undercover-report)
    (set-window-margins (selected-window) 0 4)
    (save-excursion
      (mapc
       (lambda (call-count)
         (when call-count
           (cl-incf lines-coverable)
           (when (< 0 call-count)
             (goto-char (point-min))
             (forward-line (1- line-number))
             (ov (line-beginning-position)
                 (line-end-position)
                 'type 'undercover-report
                 'face '(:background "#4e9a06")
                 'before-string (propertize
                                 " "
                                 'display `((margin right-margin)
                                            ,(propertize
                                              (format "%dx" call-count)
                                              'face 'default))))
             (cl-incf lines-covered)))
         (cl-incf line-number))
       coverage))
    (message "%d from %d lines covered (%.2f%%)"
             lines-covered
             lines-coverable
             (/ (* 100.0 lines-covered) lines-coverable))))

(defun my-hide-coverage ()
  "Hide undercover coverage report in current buffer."
  (interactive)
  (set-window-margins (selected-window) 0 0)
  (ov-clear 'type 'undercover-report))

[Fuco1]

It looks like this in Emacs

[Fuco1]

@sviridov Hi. Would it be possible to add some environment variable that woud tell undercover it runs locally?

What it should do is:

• Toggle undercover--send-report to nil, we don't want to send the reports if not in CI.
• Set undercover--report-file-path to current Cask-root so the report is saved in the project directory (with a dot so it is hidden). We can then change the code above to just look there unconditionally, it would be much simpler.

[sviridov]

@Fuco1 sure, it's possible. Will try to find time to implement it.

[DamienCassou]

@Fuco1: great job @Fuco1! Thanks.

[sviridov]

@Fuco1 I just realize that you can do it with existing environment variables. You can write something like this:

$ UNDERCOVER_FORCE=true UNDERCOVER_CONFIG='((:send-report nil) (:report-file ".undercover.json"))' cask exec ert-runner

UNDERCOVER_CONFIG will patch setttings in elisp code!)

I just added UNDERCOVER_FORCE for cleaner API but it works just like TRAVIS=true.

[Fuco1]

@sviridov hehe, amazing :)

Are the keys documented somewhere?

[sviridov]

@Fuco1, yeah, here https://github.com/sviridov/undercover.el#configuration

[Fuco1]

Okey, I will try to find some time and turn this into a pull request if you think this should be included. Maybe turn it into a minor mode or something?... Ideas welcome :)

[sviridov]

Sure, feel free to do it.

Maybe turn it into a minor mode or something?... Ideas welcome :)

Minor mode sounds good.

Just FYI, while coveralls is the only supported report type right now, undercover is capable to handle more detailed coverage information based on expressions instead of lines. You can take a look at testcover.el as an example of expression based coverage.

I just think that if one wants to see coverage info inside Emacs such way of coverage visualisation might be better.

But anyway, support for coveralls format is a good start!

[Fuco1]

Hm, that seems like a built-in solution for what we want, but only works for elisp right? The code above would work with any language.

Maybe we could later add a specific report type for elisp that would use testcover in the background.

[sviridov]

but only works for elisp right

Right, but undercover.el is elisp only too. I made it because I wanted CI integration, support for ecukes, etc.

The code above would work with any language.

Now when you mentioned it... Maybe it makes more sense to put this code in individual package (with ability to select file with coverage information) and just mention this package in undercover.el README.

[xendk]

There's already https://github.com/trezona-lecomte/coverage and https://github.com/AdamNiederer/cov/ that might be extended?

[sviridov]

@xendk few thoughts about this:

1. Both mentioned projects use external files to get information about files coverage. That is perfectly fine.

2. undercover.el can generate such file.

3. So, in theory, one could configure undecover.el to generate a file with coverage information and some other project to read from this file.

The only issue is that file formats are probably not compatible between these projects. It is possible to extend undercover.el to generate reports in other file formats (via report-type option), but unfortunately, I'm not ready to spend time on implementing it. But PRs are always welcomed.

Hope it makes sense.

[xendk]

I've hacked up cov to work with undercover: https://github.com/xendk/cov/tree/undercover-support

I load it with use-package and straight.el:

(use-package cov
  :straight (:host github :repo "xendk/cov" :branch "undercover-support")
  :config
  (add-to-list 'cov-coverage-file-paths
               #'(lambda (file-dir file-name)
                   (let ((try (format "%s/coverage-final.json"
                                      file-dir)))
                     (and (file-exists-p try)
                          (cons (file-truename try) 'coveralls))))))

Currently it needs (:report-type :codecov), as that's the file it looks for. It can't really work with :coveralls as it is now, as that saves in /tmp/undercover_coveralls_report, and who knows if that file was generated from the current buffer.

[xendk]

@sviridov I was thinking, how about using coverage-final.json in the project dir as the default filename, and changing the logic to always generate the coverage file, and only upload if CI is detected? Then cov could work pretty much out of the box.

Alternatively, save the coverage in a more Emacs native format and move the conversion to the upload routine, and let cov load the more easily parsed file.

[xendk]

(sidenote)

The (add-to-list 'cov-coverage-file-paths ... configuration isn't needed anymore.

[sviridov]

I was thinking, how about using coverage-final.json in the project dir as the default filename

I'm afraid that some projects actually use the current default value, and it can be a breaking change.

Maybe it is possible to resolve this issue by adding new :report-type? It will be possible to run coverage locally only for this :report-type, change the filename and do other required stuff.

[xendk]

@sviridov I hear what you're saying, but with the current situation you can't really use both coveralls and cov without having to fiddle with overriding settings from the command line, while things work out of the box for codecov users. Cov is already supported by the :codecov :report-type.

Of course we could make a new meta :report-type that saves the file where cov can find it and upload it to coveralls, but that feels a bit iffy. On the other hand, the ones currently using it is most likely not using :report-type, so using coverage-final.json as default file only when :report-type is explicitly :coveralls could work.

[sviridov]

but with the current situation you can't really use both coveralls and cov without having to fiddle with overriding settings from the command line

I think it is fine. undercover has many configuration options and eventually will have many combinations of reports, so it's okay if it will require more tweaking.

[CyberShadow]

While trying to improve code coverage, it would be useful to see what is covered and what is not covered by visualizing the report inside emacs with colors.

Please see "Viewing coverage in Emacs" in the README.