Automating tests with tox

[kamino]

From the project webpage:

Tox is a generic virtualenv management and test command line tool you can use for:

• checking your package installs correctly with different Python versions and interpreters
• running your tests in each of the environments, configuring your test tool of choice
• acting as a frontend to Continuous Integration servers, greatly reducing boilerplate and merging CI and shell-based testing.

Tox enable to automate tests. It can be used for unit tests, functional tests and code quality checks. Once installed with pip, one can run all default tests using only tox or a specific test with tox -e <test>

[pebenito]

Perhaps I don't fully understand tox, but were does the coverage report go when it runs on travis-ci?

[kamino]

tox can run tests either implicitly or explicitly.

With the implicit mode, runing tox command will run all the tests that are specified in tox.envlist (here, py27, py33, py34 for which commands are specified in the default testenv).

With the explicit mode, a test environment must be specified to tox using the -e option. To run explicitly the coverage environment for instance, one must pass tox -e coverage. By default, I have deactivated coverage on travis, as it is not enabled neither in the tests on master.

I should define a specific coverage environment for travis if you want it to be enabled for travis as generating the html report would be useless as it will not be browsable (travis shuts down the vm after tests). Would you like to add coverage to tests that are run on travis ?

[pebenito]

Ok, I understand. I think the config should support these functions:

• Default/implicit target is run unit tests only
• Generate a text coverage report (coverage report)
• Generate an HTML coverage report (coverage html)
• Run the pep8 tool

The Travis CI should run the coverage (which of course also runs the unit tests) and displays the text report afterwards. It should also run pep8 on everything.

I've pushed updates to the code base so the pep8 tool should pass, as long as the max line length is 100, and qpol.py is excluded. The command I used was: pep8 setools/*.py setools/policyrep/*.py tests/*.py seinfo seinfoflow sedta sesearch --exclude qpol.py --statistics --max-line-length=100

Note: I changed the NetworkX dependency on .travis.yml to be ==1.9, with the idea that the minimum version is tested.

Thanks for putting this together.

[kamino]

I am not fully sure that I made the changes as you excepted as changes were not clear to me. Can you review my PR again and get back to me if I misunderstood sth ? On travis, coverage, unit tests and pep8 are made separately as I wanted to keep things atomic.

[kamino]

typo: as you expected

[pebenito]

It's not exactly what I was expecting, but in some ways it's better than what I was asking for.

• I like having the pep8 on it's own Travis CI build.
• From the pep8 docs, it looks like you can put the pep8 configuration into the tox.ini, rather than putting a long command line.
• What I don't want is coverage to run on it's own build, as I'd like coverage reports for each version of Python. Currently 2.7 has less coverage than 3.3/3.4 because NodeconQuery's full functionality requires Python 3.3+. This may happen more in the future.

So I think to wrap this up, this is what needs to be done:

1. Add the command pep8 --version to [testenv:pep8], as the first command
2. See if you can get the pep8 configuration into the tox.ini.
3. Instead of having a TOX_ENV=coverage in .travis.yml, add an after_success than runs tox -e coverage for any build but pep8. Perhaps this will work: [ "$TOX_ENV" != pep8 ] && tox -e coverage

In the mean time, I'll update my local copy of pep8 since I didn't see the errors your Travis build got.

[kamino]

imho, we should only run pyXX-coverage instead of running pyXX,pyXX-coverage as the latter already run test commands.

[pebenito]

I agree. I can fix that.

[pebenito]

Thanks for all the work you put in to this!