search

ipython / ipython_genutils

Vestigial IPython utilities: DO NOT USE
Other
11 stars 22 forks source link

onlyif_cmds_exist should get a kwarg to fail when on CI services #7

Closed jankatins closed 2 years ago

jankatins commented 8 years ago

In this case the an incomplete latex setup on travis lead to missing tests in nbconvert, which prevented the tests from registering a bug. See https://github.com/jupyter/nbconvert/pull/356 for the story...

IMO a testsuite should not skip any tests in a CI environment, so it would be nice if the decorator could raise a failure instead of skipping if it is run on a CI service (=change the default to fail if the command is not present on travis/appveyor/circleci).

CI like travis set some environment variables, so a simple test for such env variables could be used to change a skip to a fail.

takluyver commented 8 years ago

I think previously it was harder to install things like Tex on Travis, which was why we had some skipped tests.

I don't think I'd go for checking for specific CI services, but rather I'd allow setting some environment variable like FAIL_ON_TEST_SKIP=1. Maybe nose/py.test already support an option for something like this?

jankatins commented 8 years ago

I asked for pytest on twitter, but it seems they don't have such an switch. https://twitter.com/janschulz/status/763650601796395008

takluyver commented 8 years ago

It sounds like it should be doable as a py.test plugin rather than changing project code, though. Based on this gist in a response.

Nose has a --no-skip option which I imagine would do what we want.