ERL-77: common test code coverage does not honour excl_dirs etc to exclude directories

[OTP-Maintainer]

Original reporter: linearregression Affected version: OTP-18.0 Component: common_test Migrated from: https://bugs.erlang.org/browse/ERL-77

---

Per http://www.erlang.org/doc/apps/common_test/cover_chapter.html
We should eb able to exclude directories, modules etc.
{code}%% Directories to exclude in cover.
{excl_dirs, Dirs},
{excl_dirs_r, Dirs}.{code}
But it seems like ot being honored notice in 18.2,18.1, 18.0.Have not tried other versions.

Output after runnjng code coverage below shows the list of test code to exclude:
content of coverpec
{code}{nodes, ['gen_rpc_master@127.0.0.1', 'gen_rpc_slave@127.0.0.1', 'nonode@nohost']}.
{export, "../_build/test/cover/ct.coverdata"}.
{level, details}.
{incl_dirs_r, ["../src"]}.
{excl_dirs_r, ["../test"]}.{code}
using excl_mods does not work either.
Basically all the exclude doe snot seem to work.
This is basic functionality of a test tool. 
Can some one assigned to fix?

Btu end results
{code}|----------------------------------|------------|
| module | coverage |
|----------------------------------|------------|
| local_functional_SUITE | 98% |
| multi_rpc_functional_SUITE | 87% |
| gen_rpc_test_helper | 79% |
| remote_functional_SUITE | 96%{code}

Originally reported here https://github.com/priestjim/gen_rpc/issues/42

[OTP-Maintainer]

siri said:

I haven't been able to re-create this problem. Could you please point me to a runnable example?

[OTP-Maintainer]

siri said:

I'd like to add that I got gen_rpc from github and ran 'rebar3 ct'. This did not cause any modules to be cover compiled at all. The reason is that no .beam files were found in the directory pointed to by the incl_dirs_r option in gen_rpc.coverspec. After rewriting the coverspec to point at "../_build/test/lib/gen_rpc/ebin" instead, it worked, and no suite files were included.

[OTP-Maintainer]

edwardttril said:

Hi I just tried using as what wrote in comment. I still see modules SUITE reported as covered 100%.
I am still seeing the test modules reported as part of the coverage despite excludlng them in coverspec.
Are those modules there just for reporting purposes or they are counted towards the final coverage 79%?

Possible you submit a pull request to gen_rpc or comment on it? The version you tried working so taht I can try on my own here? I am not sure what is the mismatch that causes the difference.

The coverspec under test/gen_rpc.coverspec  content below:

    {nodes, ['gen_rpc_master@127.0.0.1', 'gen_rpc_slave@127.0.0.1', 'nonode@nohost']}.
    {export, "../_build/test/cover/ct.coverdata"}.
    {level, details}.
    {incl_dirs_r, ["../_build/test/lib/gen_rpc/ebin"]}.
    {excl_dirs_r, ["../_build/test/lib/gen_rpc/test"]}.

    results get:

     |----------------------------------|------------|
      |                          module  |  coverage  |
      |----------------------------------|------------|
      |                  gen_rpc_client  |       75%  |
      |                gen_rpc_acceptor  |       64%  |
      |                  gen_rpc_server  |       56%  |
      |              gen_rpc_dispatcher  |       59%  |
      |            gen_rpc_acceptor_sup  |       75%  |
      |                         gen_rpc  |       80%  |
      |              gen_rpc_server_sup  |       78%  |
      |              gen_rpc_client_sup  |       83%  |
      |                     gen_rpc_sup  |      100%  |
      |                  gen_rpc_helper  |       77%  |
      |                     gen_rpc_app  |       50%  |
      |  Elixir.ExRPC.Supervisor.Server  |      100%  |
      |          local_functional_SUITE  |       97%  |   <------
      |      multi_rpc_functional_SUITE  |       87%  |   <-----
      |         remote_functional_SUITE  |       96%  |   <-------
      |             gen_rpc_test_helper  |       80%  |  <------
      |----------------------------------|------------|
      |                           total  |       79%  |
      |----------------------------------|------------|

    I Tried two wayss:
   0) make testclean or rm -rf _bild folder
    1 way) make test

    2 way) use rebar3 directly without makefile
    - rebar3 ct
    - rebar3 cover

Got the same old result I reported for both cases.

    Erlang :
    Erlang/OTP 18 [erts-7.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

    Eshell V7.2.1  (abort with ^G)
    1> 

[OTP-Maintainer]

siri said:

Ok, now I see. The cover result you are looking at is produced by rebar3, not common_test. rebar3 does not care about the content of the common_test coverspec file. I don't know if it is possible to instruct rebar3 which modules to cover compile and not, but I haven't found anything about it in the documentation. From the code I see that it finds the ebin and test directory from its internal state, and cover compiles all .beam files in these directories.

The cover result that I was looking at are the ones in the common_test HTML logs. You can find a link from the top page of each test run. To get it right, you could remove everything regarding cover from your rebar.config (except the 'cover' option to common_test, which points out the coverspec file). Then change the coverspec file to someting like this:

{code}{nodes, ['gen_rpc_master@127.0.0.1', 'gen_rpc_slave@127.0.0.1', 'nonode@nohost']}.
{export, "../_build/test/cover/ct.coverdata"}.
{level, details}.
{incl_dirs_r, ["../_build/test/lib/gen_rpc/ebin"]}.{code}

Note that you don't need to exclude anything here since common_test does not include anything by default - only what's specified in the coverspec file.

Also note that since the coverspec file specifies a coverdata file to export in <profiledir>/cover, you can actually use rebar3 to look at the result using 'rebar3 cover -v'... However, as I mentioned above, you do have the same results in the common_test HTML logs. (Actually, the results are slightly different. This might be because common_test uses round/1 while rebar3 uses trunc/1 when calculating the percentage - but this I haven't verified.)

[OTP-Maintainer]

siri said:

This is not a problem in common_test, it is a misunderstanding of the code coverage functionality in rebar3.