erlang / rebar3

Erlang build tool that makes it easy to compile and test Erlang applications and releases.
http://www.rebar3.org
Apache License 2.0
1.7k stars 518 forks source link

REBAR3: Common test code coverage does not honour excl_dirs etc to exclude directories #1057

Closed linearregression closed 8 years ago

linearregression commented 8 years ago

We are using rebar3, and run with coder coverage. It will add tests as part of coverage skeing coverage result!.

Our rebar.config https://github.com/priestjim/gen_rpc/blob/master/rebar.config this point to cover spec at: https://github.com/priestjim/gen_rpc/blob/master/test/gen_rpc.coverspec

Per http://www.erlang.org/doc/apps/common_test/cover_chapter.html We should eb able to exclude directories, modules etc. %% Directories to exclude in cover. {excl_dirs, Dirs}, {excl_dirs_r, Dirs}. But it seems like ot being honored notice in 18.2,18.1, 18.0.Have not tried other versions. Output after runnjng code coverage below shows the list of test code to exclude: content of coverpec {nodes, ['gen_rpc_master@127.0.0.1', 'gen_rpc_slave@127.0.0.1', 'nonode@nohost']}. {export, "../_build/test/cover/ct.coverdata"}. {level, details}. {incl_dirs_r, ["../src"]}. {excl_dirs_r, ["../test"]}. using excl_mods does not work either. Basically all the exclude doe snot seem to work. This is basic functionality of a test tool. Can some one assigned to fix? Btu end results ---------------------------------- ------------ module coverage
local_functional_SUITE 98%
multi_rpc_functional_SUITE 87%
gen_rpc_test_helper 79%
remote_functional_SUITE 96%

Originally reported here https://github.com/priestjim/gen_rpc/issues/42

linearregression commented 8 years ago

ping???

ferd commented 8 years ago

I'm a bit confused by the usage of relative paths here. How are you planning to run these? My suspicion, aside from cover being broken, is that the paths are wrong (rebar3 never cds anywhere and stays at the root dir) and therefore cannot be either excluded nor included.

linearregression commented 8 years ago

it used to work for old version of rebar and oldver version of erlang 17 (with rebar 3 older version a few months back last year, around Oct 2015 timeframe) Those relative path entries are from coverspec https://github.com/priestjim/gen_rpc/blob/master/test/gen_rpc.coverspec rebar.config is referencing the coverspec file as in https://github.com/priestjim/gen_rpc/blob/master/rebar.config

tsloughter commented 8 years ago

rebar3 is a whole new tool. Those relative paths are not going to work with rebar3 because it does not change directories while running like rebar did.

linearregression commented 8 years ago

I thought rebar3 just read the coverspec in rebar.config and give it to underlying coverage functionality of common test??

tsloughter commented 8 years ago

Yes, but that doesn't change the fact that ../ is different in rebar3 than rebar2 because it does not change directories anymore.

linearregression commented 8 years ago

So it means I should try putting the coverspec as root directory of project and see if it excludes? Actually I have another project but internal that does just that (not using relative path) and code coverage execution still ignores all the exclusion criteria.

talentdeficit commented 8 years ago

when using rebar3 and ct the relevant beam files that ct works with are in '_build/PROFILE/lib/*/ebin'. try excluding those directories

linearregression commented 8 years ago

Yes tried that and also using absolute path. No luck. Still including the test beams

talentdeficit commented 8 years ago

ok. i will take a closer look at this over the weekend

sirihansen commented 8 years ago

When setting {cover_enabled,true} in rebar.config, it seems rebar3 will perform its own cover compilation/analysis, independent of common_test. The coverspec file given to common_test is not read by rebar3 at all, so files included/excluded there are not honoured by rebar3's cover analysis. I don't know if it is possible to specify which modules rebar3 should cover compile and not, but by default it seems that it takes all beam files found in ebin and test.

If you want common_test to do the cover analysis, then set cover_enabled to false (default) in rebar.config, and only keep the cover option to common_test, pointing out the coverspec file.

The cover result can then be found in the HTML logs from common_test.

priestjim commented 8 years ago

@tsloughter thanks for updating the docs to explain rebar3's behavior when it comes to coverage. However, the question remains: why does rebar3 account for test code coverage INSIDE the test files? Can we do something about it?

talentdeficit commented 8 years ago

@priestjim are you trying to exclude coverage statistics from the rebar3 cover tool, or from the ct cover tool? they are two different tools.

rebar3 just passes through the ct cover spec to ct. you'll have to ensure that the directories you are excluding are the ones in the _build directory, as that is where rebar3 loads all modules from

if you are asking how to exclude coverage stats from the build in cover tool, you currently can't. it could be added as an option in the future. is that what you are asking for?

priestjim commented 8 years ago

I am trying to exclude coverage accounting and reporting for the test files (ct and eunit)

talentdeficit commented 8 years ago

i've made a pr that fixes the paths in your cover spec. let me know if this fixes your issue

talentdeficit commented 8 years ago

can you explain a little bit what you mean by proper coverage? i have zero plans to support ct coverspecs in the rebar3 cover tool (but note that it works fine if you use the ct cover tool instead) but i would like to add a way to only include modules that are part of the applications being tested (so modules that are listed in the mod key of all .app files). would that fix your problem?

priestjim commented 8 years ago

Yes it would. Essentially, when I use rebar3 to generate code coverage information, it takes into account ALL the modules available to the system, including the ones in the test/ directory, which is kind of pointless. Evaluating only the modules listed in the .app files would definitely fix the problem.

talentdeficit commented 8 years ago

ok. i probably won't get to it until later this week, but i will see what i can do

paulo-ferraz-oliveira commented 4 years ago

I was able to unwillingly generate coverage data for _SUITE.erl by using {src_dirs, ["test", "test/util"]} instead of {extra_src_dirs, ["test/util"]}. Removing _build and changing to {extra_src_dirs, _} got rid of that problem.

This is (somewhat?) related to #1179.

ferd commented 4 years ago

If you declare test files to be source files (by putting them in src_dirs) then we will generate coverage report for them since they're seen as source files. This would be a case of "this is not a bug, it's a feature."

paulo-ferraz-oliveira commented 4 years ago

If you declare test files to be source files (by putting them in src_dirs) then we will generate coverage report for them since they're seen as source files. This would be a case of "this is not a bug, it's a feature."

Just wrote it here in case somebody gets bit like me 😄