Make tests runnable in empty environment

[jbednar]

Right now, if you install holoviews, install nose, and then try running the holoviews tests with nosetests --with-doctest -v, there are 18 failures (and pages of associated output) because of not having iris, flexx, plotly, and other optional libraries installed:

FAILED (SKIP=56, errors=407, failures=18)

This makes it very difficult to determine if the basic holoviews installation is working, particularly if you want to see if it works well without those optional libraries. Is it just a matter of needing to raise 18 more SkipTest exceptions?

[jbednar]

Or more likely 13 SkipTests, as I get 5 failures even after installing those optional libraries...

[philippjfr]

nosetests --with-doctest -v -a '!optional' should improve things somewhat but there are still some differences due to SciPy versions.

[jbednar]

Thanks. @ceball, sounds like we should probably define a "coretests" or similar target for your doit setup, which would supply that flag to nosetests (or pytests if that's what we end up with) so that the basic tests could be run on any system.

[ceball]

Yes, that's the plan.

But I'll take this chance to say more, at the risk of boring everyone, in the hope of getting feedback, or better ideas...

There's doit unit_tests, doit nb_flakes, doit slow_tests, etc etc, kind of thing. Those commands are as far as possible just a wrapper of pytest (or nose) commands (with plugins for stuff, e.g. nbsmoke plugin for basic notebook tests). And the pytest commands use pytest's features for selecting groups of tests to run, resulting in - say - just pytest to run unit tests, or pytest -m slow_tests for "slow tests", etc, or something like that).

That sounds like pointless duplication/shadowing, but it does mean things are all documented in one place (doit list will show them), and those commands can be consistent across projects, change of underlying tool, etc. At the same time, it will remain no problem for anyone to run the tests without doit. Plus, doit supports task dependencies etc, so if some tests require fetching data or other setup, they can depend on a task that does whatever's needed. (However, this 'shadowing' still feels a bit odd, so if anyone has a better idea, I'd love to hear it.)

I think that covers groups of tests. However, there's also the idea of testing in different environments (different pythons, different sets of dependencies, etc). In python land, tox provides the ability to test in different environments . E.g. param has a tox.ini that's a bit like this (*):

[tox]
envlist = py36,py35,py34,py33,py27,pypy

[testenv]
commands = nosetests --verbose --nologcapture --with-doctest

[testenv:with_numpy]
deps = numpy

[testenv:with_ipython]
deps = ipython

Running tox defaults to running testenv, testenv:with_numpy, and testenv:with_ipython across python 2.7, 3.3, 3.4, 3.5, and 3.6, plus pypy. That can be configured (e.g. to run just testenv with your current python). Or a command-line option can be supplied to run just one, or a subset, etc. tox also supports syntax to generate and combine environments, which could be helpful if there are many combinations.

By default, tox builds a package, creates a new virtual environment, installs the package into it, then runs the tests. But it can also test develop/editable install.

conda is not compatible with tox (or: tox only works with python's standard/built-in virtual environments). I'm not sure there's an equivalent for conda. conda-build gives you the build package/create environment/install package/run tests bit, but you'd need one meta.yaml for each set of dependencies etc. You could generate them, maybe.

(Part of the point of gathering all these things into doit+pyct is that it helps clarify what features of underlying tools might be desirable. doit+pyct can provide a workaround to implement something missing, until maybe the missing thing is implemented in an underlying tool. Even just collecting issues in one place for some things we can't do seems useful, which we can then use to create issues/implement things in underlying tools. Also, I guess doit+pyct is trying to support both "pip world" and "conda world" where necessary, with commands that work in either where possible, or having variants of a command for each case.)

---

(*) param's tox.ini has extraneous stuff I haven't removed yet. And specification of dependencies will probably move out of tox.ini and into setup.py or something (different tox commands would then just run different pip commands - pip install package[optional]etc). That's not really relevant for param, but is more important for packages with more complex dependencies (to make sure dependencies are specified in one place only).

[jbednar]

That all sounds reasonable, if somewhat painful. And there's no hope of making tox support conda?

[ceball]

Yes, it would be a bit painful in parts even if just considering conda or python alone, but supporting both worlds is tough.

making tox support conda

Presumably it could be done, but I don't know how much work it would be (more info: https://github.com/tox-dev/tox/issues/273).

[obestwalter]

@jbednar and @ceball - saw this referenced and came here to say that there is hope now. This will likely be a plugin called tox-conda, but as I don't use conda myself I would appreciate it, if we could find contributors/maintainers for that plugin then.

[jbednar]

Thanks! There is plenty of interest inside Anaconda in tox+conda, both in my own group and from the conda maintainers. My own group (PyViz) would be happy to test it out and provide fixes for any problems we can figure out, and I think the conda team will be able to help us with any deeper issues that come up...

[obestwalter]

For reference: progress will be tracked in the already above mentioned issue (https://github.com/tox-dev/tox/issues/273).