Need a way to run code between config and test loading

[DarwinAwardWinner]

In order to use undercover.el, it must be loaded before the file(s) to be tested. However, if undercover is loaded in a separate file, there's no way to force that file to be loaded first. I can try naming it 001-setup-undercover.el, but that doesn't help if there's more than one test directory and the directory containing the undercover setup code isn't lexically first. This is exactly the case I'm running into with my package ido-completing-read+, where I have 2 test directories: essentially ./tests/ and ./tests-that-should-run-with-undercover. I actually run eldev test twice, once with only the former directory in eldev-test-fileset, and once with both directories in it. Obviously the undercover setup is in the 2nd directory, which means that it gets loaded after the code in ./tests, too late to instrument my package. This happens regardless of what order I specify the directories in eldev-test-fileset.

I tried setting up undercover in a local Eldev file, but that's too early because the dependencies aren't in load-path yet. So it seems that what I need is a way to run some setup code after the load-path is set up but before any test files are loaded. Is there a way to do this? One possible solution would be to respect the order in eldev-test-fileset. Then I could just specify the undercover setup file first in the list.

[DarwinAwardWinner]

For what it's worth, here's what I came up with: https://github.com/DarwinAwardWinner/ido-completing-read-plus/tree/e1a4edc40b1fbfeb444de691e390e81022b8a2b1. It's kind of unsatisfying, because it relies on the implicit behavior of test files loading in lexical order.

[doublep]

I plan to integrate undercover directly into Eldev. Presumably, it would be as simple to use as adding

(eldev-use-plugin 'undercover)

to Eldev (Eldev already knows how to find main and test files, so you wouldn't need to duplicate this information). I think it would be a good idea to add "plugin" concept where plugins could be both built-ins (likely in this case) and downloadable from Melpa, so that other people could extend Eldev independently from upstream.

I wanted to suggest eldev-test-buttercup-hook, but that is called after loading files. But you could probably achieve something with eldev-load-dependencies-hook instead: check if test additional set is loaded or check if (require 'undercover nil t) succeeds.

[DarwinAwardWinner]

Note that you probably only want to allow running undercover with eldev -s. Otherwise, you get this: https://coveralls.io/builds/28756391. Note the coverage info that was recorded on files inside .eldev/ from running with eldev -p. (Also, this seems to imply that eldev -p doesn't byte-compile the file? Is there a way to do eldev -p with byte-compilation?)

[doublep]

Also, this seems to imply that eldev -p doesn't byte-compile the file? Is there a way to do eldev -p with byte-compilation?

With -p it builds a package, lets Emacs install and activate it (i.e. with package-install, package-activate). The files do get byte-compiled, but not by Eldev, rather by Emacs itself when it installs the package:

~/logview$ eldev -s eval "(byte-code-function-p (symbol-function 'logview-mode))"
nil
~/logview$ eldev -c eval "(byte-code-function-p (symbol-function 'logview-mode))"
t
~/logview$ eldev -p eval "(byte-code-function-p (symbol-function 'logview-mode))"
t

I'm not sure about the output you get, it looks like a local dependency is loaded as packaged, but somehow undercover still instruments it. Maybe you should tell undercover to ignore everything under .eldev.

Without undercover at least everything looks fine. E.g. with (eldev-use-local-dependency "~/datetime" 'packaged) in Eldev-local I get this:

~/logview$ eldev eval "(byte-code-function-p (symbol-function 'datetime-list-locales))"
t

In comparison, with (eldev-use-local-dependency "~/datetime" 'source) the function is not byte-compiled:

~/logview$ eldev eval "(byte-code-function-p (symbol-function 'datetime-list-locales))"
nil

Anyway, as said, I will later integrate undercover so that projects using Eldev don't need to bother with any quirks.

[DarwinAwardWinner]

I've actually tried telling undercover to ignore everything in .eldev, but it apparently doesn't support recursive wildcards, so a pattern like .eldev/* will only ignore files 1 level deep. I could count the depth of subdirectories that results from eldev -p, but adding a pattern like .eldev/*/*/*/*/* seems kind of silly, not to mention relying on an implementation detail of eldev. So when you build in support, you'll probably want to generate the exact list of files for it to ignore at runtime.

[doublep]

Eldev 0.3 features undercover integration, which solves the mentioned issues. As for the title of this issue: you could misuse some of the existing hooks, but I won't add any additional features for it.