pytest.main with coverage arguments shows less coverage report compare to commandline py.test

[erwagasore]

I am using pytest.main function call to create a flask test command

@app.cli.command()
@click.option('-c', '--cov-report', type=click.Choice(['term', 'term-missing']), help="Turn on coverage reporting")
@click.option('-d', '--debug', is_flag=True, help="Enter pdb on test fail or error")
@click.option('-n', '--number-of-cpu', type=click.IntRange(2), help="Parallelize test on multiple cores")
@click.option('--flakes', is_flag=True, help="Run pyflakes on python files")
@click.option('--pep8', is_flag=True, help="Perform some PEP8 sanity checks on python files")
def tests(debug, cov_report, number_of_cpu, pep8, flakes):
    options = '--cov-config {}/.coveragerc'.format(app.config['PROJECT_DIR'])
    if number_of_cpu:
        options = '{} -n {}'.format(options, number_of_cpu)
    if cov_report:
        options = '{} --cov {} --cov-report {}'.format(options, app.config['PROJECT_DIR'], cov_report)
    if debug:
        options = '{} --pdb'.format(options)
    if pep8:
        options = '{} --pep8'.format(options)
    if flakes:
        options = '{} --flakes'.format(options)
    print options
    # os.system('py.test {}'.format(options))
    pytest.main(options)

Once pytest.main(options) the coverage is 32% and it seems to ignore all method definition lines into all tested files

app/users/tests.py .
--------------------------------- coverage: platform darwin, python 2.7.10-final-0 ----------------------------------
Name                                 Stmts   Miss  Cover   Missing
------------------------------------------------------------------
app/auth.py                             23     19    17%   1-7, 12, 14, 18-38
app/cli.py                              20     20     0%   1-35
app/commands.py                         70     70     0%   1-98
app/config.py                           56     55     2%   1-61, 65-89
app/decorators/caching.py               38     21    45%   1-10, 21-33, 61-62, 67-68, 72-75, 78
app/decorators/jsonify.py               20      7    65%   1-6, 16, 19, 28
app/decorators/paginate.py              23      9    61%   1-8, 25, 31, 44-45
app/errorhandlers.py                    17     11    35%   1-4, 10, 16-25
app/exceptions.py                        3      3     0%   1-5
app/extensions.py                        7      7     0%   1-9
app/factory.py                          43     23    47%   1-15, 53-68
app/main/views.py                        9      9     0%   1-14
app/models.py                          176     98    44%   1-24, 65, 97-109, 116-144, 147, 150, 153, 156, 159, 163, 172, 185, 188, 191, 195, 200-218, 227, 231-281
app/roles/views.py                      34     34     0%   1-54
app/tasks.py                             0      0   100%   
app/test_client.py                      30      0   100%   
app/users/views.py                      50     25    50%   1-12, 20-21, 26-27, 35-36, 43-45, 50-52, 57-59, 65-66
app/utils.py                            26     26     0%   1-39
app/validators/unique_validator.py       7      4    43%   1-7, 11
------------------------------------------------------------------
TOTAL                                  652    441    32%   

But when I choose to use os.system('py.test {}'.format(options)) the coverage become 75% which is the realistic coverage i was expecting

--------------------------------- coverage: platform darwin, python 2.7.10-final-0 ----------------------------------
Name                                 Stmts   Miss  Cover   Missing
------------------------------------------------------------------
app/auth.py                             23     12    48%   12, 14, 20-24, 29-31, 36-38
app/cli.py                              20     20     0%   1-35
app/commands.py                         70     46    34%   20-37, 47-59, 72-79, 86-89, 95-98
app/config.py                           56      0   100%   
app/decorators/caching.py               38      7    82%   61-62, 67-68, 72-75
app/decorators/jsonify.py               20      2    90%   16, 19
app/decorators/paginate.py              23      2    91%   25, 31
app/errorhandlers.py                    17      6    65%   17-19, 23-25
app/exceptions.py                        3      0   100%   
app/extensions.py                        7      0   100%   
app/factory.py                          43     11    74%   54-68
app/main/views.py                        9      1    89%   14
app/models.py                          176     16    91%   210, 213, 216, 232-233, 251-253, 257, 261, 265, 270, 274, 277-278, 281
app/roles/views.py                      34     15    56%   13-16, 22, 28-31, 37-39, 46, 53-54
app/tasks.py                             0      0   100%   
app/test_client.py                      30      0   100%   
app/users/views.py                      50      0   100%   
app/utils.py                            26     26     0%   1-39
app/validators/unique_validator.py       7      0   100%   
------------------------------------------------------------------
TOTAL                                  652    164    75%   

[RonnyPfannschmidt]

this is a known issue wrt coverage not covering imported modules, there is a plan to fix this by loading the coverage plugin before the other plugins/conftest

[erwagasore]

Thanks @RonnyPfannschmidt. Is there any kind of work around?

[abendebury]

Having the same issue. The weird thing is that using pytest.main() in a script by itself seems to fix the issue for me.

[erwagasore]

@RonnyPfannschmidt any progress so far with this issue? Just asking without the intention to be pressing. Thanks

[RonnyPfannschmidt]

no progress so far, personally i'm also working on other issues for quite a while longer

[webknjaz]

The workaround I've been having in my configs is -p pytest_cov.