Coverage results not being sent to codecov/coveralls

[dr-rodriguez]

I've been setting up a new package with this cookiecutter template and have most of the CI setup complete. However, the coverage information has been causing issues. Initially, I followed the instructions to set up for coveralls, but after seeing 0% coverage repeatedly I figured there was an issue and switched to codecov. Codecov is giving me a more interesting message and after some debugging I believe it may be due to the location of the .coverage file. See this travis log for example: https://travis-ci.org/github/dr-rodriguez/AstrodbKit2/jobs/703353853 But here is some of the relevant output:

$ pwd
/home/travis/build/dr-rodriguez/AstrodbKit2
$ ls -a
.             docs    MANIFEST.in       setup.cfg  tox.ini
..            .git    pyproject.toml    setup.py   .travis.yml
astrodbkit2       .gitignore  README.rst        .tmp
astrodbkit2.egg-info  licenses    .readthedocs.yml  .tox
$ ls -a .tmp/py38-test-cov
.  ..  .coverage
$ codecov
      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.1.7
==> Detecting CI provider
    Travis Detected
==> Preparing upload
==> Processing gcov (disable by -X gcov)
==> Collecting reports
Error: No coverage report found

The .coverage file my tests produce is located in .tmp/py38-test-cov My guess is that the default setup has codecov running at the top level directory and thus does not find this .coverage file. I suspect this is also why coveralls failed- no actual coverage data was being sent. Should codecov be running as a travis step or does it need to be switched to part of a tox build? Do I need some extra configuration to tell travis to navigate to that test environment? I'm surprised as my tox/travis/setup files are pretty much the template defaults and are all very similar to the astropy one.

[dr-rodriguez]

As a quick update, adding a cd .tmp/py38-test-cov to travis prior to calling codecov does get the .coverage file processed, but it's probably not ideal to have that hard-coded in, particularly if one is considering coverage for different environments.

[pllim]

Looks like the actual command on astropy is a little different: https://github.com/astropy/astropy/blob/7cc204fc187fdf0187ef7155ea79ecdf0fe53de5/.travis.yml#L260-L264

after_success:
    - if [[ $TOXENV == *-cov ]]; then
        pip install codecov;
        codecov --gcov-glob "*cextern*";
      fi

[dr-rodriguez]

True, but I think that addresses more #458. I ended up using:

after_success:
      - if [[ $TOXENV == *-cov ]]; then
          pip install codecov;
          cd .tmp/$TOXENV;
          codecov;
        fi

The underlying issue for me is that after the tox test runs travis navigates back to the top level directory and codecov can no longer access the .coverage file. I would recommend that a cd statement like what I have (now with the $TOXENV variable) be part of the default travis that gets installed, but I'm not sure how frequently that's needed as clearly astropy does not.

[pllim]

Yeah, I have no idea why it works for astropy but not others. Maybe @Cadair knows.

[saimn]

Astropy is specifying the config path in the test command : --cov-config={toxinidir}/setup.cfg https://github.com/astropy/astropy/blob/master/tox.ini#L93 This way the coverage report works also when running tox manually. I found this while working on the coverage for astroscrappy recently, but forgot to report, sorry !

[saimn]

Oh and probably this as well: https://github.com/astropy/astropy/blob/7cc204fc187fdf0187ef7155ea79ecdf0fe53de5/.travis.yml#L226-L228

edit: --cov-config is already in the template, I think what is missing is the piece above which will produce the xml coverage report in the travis build directory.

[dr-rodriguez]

Aha! Yes, it was the missing before install statements that @saimn points out. Those are missing from the default .travis.yml file but present in the astropy one. When I added them I got a coverage.xml file produced in the top-level directory that codecov was able to access.

[pllim]

Ah, it's set in before_install. I totally missed it. Thanks!

So, I guess the template needs fixing.