start using pytest and tox

[schmir]

I asked for your opinion on py.test in some other issue, it's probably easier to see the benefits with some real code.

so, here it is.

[majek]

Okay. Thanks for bringing py.test to my attention, but no thanks, I don't see a convincing reason. Maybe I'll use it in new projects, but I don't see the benefit of changing current puka testing infrastructure.

0) What's the benefit of py.test vs nosetests?

1) Right now the idea is that you can type make on any machine with RabbitMQ and you should get all tests running. That includes three things:

• doctests from the puka subdirectory
• nose tests from the tests subdirectory
• and coverage report from that

In your patch when I type PYTHONPATH=. ./venv/bin/py.test I can see only the py.tests (updated nose tests) running.

2) With current nose tests you can run a single file with tests selectively:

 $ (cd tests && PYTHONPATH=.. ../venv/bin/nosetests test_bug3:TestBug3.test_bug3_wait)

Actually it is sometimes useful to rely on nosetests magic. I don't think I know how to run test selectively using py.test.

3) py.test and tox seem to be independent tools. If you want to focus on testing particular python revisions why not just use tox?

4) In your patch you rewrite the tests to be in py.test style. why? Example:

https://github.com/schmir/puka/blob/b567cf5b08a26d40c56f630de6ff9a646f92f5d7/tests/test_limits.py https://github.com/schmir/puka/blob/0b004acb727fe832508ee16fa678eafcbb0e192c/tests/test_limits.py

py.test docs say: "pytest can run many nose, unittest.py and doctest.py"

5) I like the way tox can be used to test multiple versions of python, but I'm uncomfortable with the prerequisite of having a lot of python versions installed. If we went for using tox, we'd need a simple make target that will build all required python releases from sources. Basically, in any linux release that can compile cpython I'd like to be able to type make test and see the tests running on all the required pythons. Can tox do that out of the box?

[schmir]

Okay. Thanks for bringing py.test to my attention, but no thanks, I don't see a convincing reason. Maybe I'll use it in new projects, but I don't see the benefit of changing current puka testing infrastructure.

0) What's the benefit of py.test vs nosetests?

I know nosetests just a bit, but AFAIK py.test has the following benefits:

• Asserting with the assert statement (http://pytest.org/latest/assert.html#assert-with-the-assert-statement) you don't have to use assertTrue, assertFalse, assertEqual and whatever they are called. just use assert and py.test will give you a meaningful report with the values from the assert statement
• the py.test funcargs/fixture system, which is quite powerful
• test parametrization e.g. you could easily test against multiple rabbitmq instances (2.8, 3.x) by making the amqp_url fixture "return multiple values". each test that uses the client fixture would then be run multiple times since the client fixture uses the amqp_url parameter.
• probably better error/traceback reporting.

1) Right now the idea is that you can type make on any machine with RabbitMQ and you should get all tests running. That includes three things:

• doctests from the puka subdirectory

pytest can also run doctests. that should be easy to add.

• nose tests from the tests subdirectory

it already does that.

• and coverage report from that

that's possible with

pip install pytest-cov py.test --cov=puka

In your patch when I type PYTHONPATH=. ./venv/bin/py.test I can see only the py.tests (updated nose tests) running.

2) With current nose tests you can run a single file with tests selectively:

 $ (cd tests && PYTHONPATH=.. ../venv/bin/nosetests test_bug3:TestBug3.test_bug3_wait)

that's sure possible: py.test -k bug3_wait

Actually it is sometimes useful to rely on nosetests magic. I don't think I know how to run test selectively using py.test.

3) py.test and tox seem to be independent tools. If you want to focus on testing particular python revisions why not just use tox?

right, you could also use tox with nosetests.

4) In your patch you rewrite the tests to be in py.test style. why? Example:

https://github.com/schmir/puka/blob/b567cf5b08a26d40c56f630de6ff9a646f92f5d7/tests/test_limits.py https://github.com/schmir/puka/blob/0b004acb727fe832508ee16fa678eafcbb0e192c/tests/test_limits.py

py.test docs say: "pytest can run many nose, unittest.py and doctest.py"

I like writing test functions much more than subclassing TestCase. In that particular example the test doesn't manually create the client and the msg since they are passed in as funcargs (btw. msg isn't used). It also does not have to determine the RabbitMQ server to use.

the fixture also makes sure that the client connection is being closed. makeid creates test specific names. I used that to get an idea which test functions do not clean up.

5) I like the way tox can be used to test multiple versions of python, but I'm uncomfortable with the prerequisite of having a lot of python versions installed. If we went for using tox, we'd need a simple make target that will build all required python releases from sources. Basically, in any linux release that can compile cpython I'd like to be able to type make test and see the tests running on all the required pythons. Can tox do that out of the box?

no, sorry, it won't build python on your platform. that's out of scope for tox.

[schmir]

ok. closing that one now. If you ever see the light :) or have more questions, please let me know :)