Support for better rendering of parameterized tests

[tknerr]

I'm currently using parameterized tests like here: http://docs.pytest.org/en/latest/parametrize.html

Example:

import pytest

@pytest.mark.parametrize("name,version", [
    ("ansible", "2.1.0.0"),
    ("git", "2.7.4"),
    ("python", "2.7.11"),
    ("python-pip", "8.1.1"),
    ("meld", "3.14.2"),
    ("vim", "7.4.1689"),
])
def test_package_is_intalled_at_version(Package, name, version):
    assert Package(name).is_installed
    assert version in Package(name).version

The resulting output with pytest-spec looks like this:

spec/test_tool_versions.py
    [PASS]  Package is intalled at version[local-ansible-2.1.0.0]
    [PASS]  Package is intalled at version[local-git-2.7.4]
    [PASS]  Package is intalled at version[local-python-2.7.11]
    [PASS]  Package is intalled at version[local-python-pip-8.1.1]
    [PASS]  Package is intalled at version[local-meld-3.14.2]
    [PASS]  Package is intalled at version[local-vim-7.4.1689]

Any ideas how we could render the parameters more nicely?

Eg. by formatting the test name to include the params:

def test_package_NAME_is_intalled_at_version_VERSION(Package, name, version):
    assert Package(name).is_installed
    assert version in Package(name).version

To produce something like:

spec/test_tool_versions.py
    [PASS]  Package "ansible" is intalled at version "2.1.0.0"[local]
    [PASS]  Package "git" is intalled at version "2.7.4"[local]
    [PASS]  Package "python" is intalled at version "2.7.11"[local]
    [PASS]  Package "pyhon-pip" is intalled at version "8.1.1"[local]
    [PASS]  Package "meld" is intalled at version "3.14.2"[local]
    [PASS]  Package "vim" is intalled at version "7.4.1689"[local]

What do you think?

[pchomik]

Your proposition looks better than original but I'm not sure if it is clean enough. I have to think a little bit about it. Never the less issue will be implemented.

[tknerr]

Hi @pchomik , just came across this again.

I still don't know whether it's a good idea or not (especially how to "template" the parameters in the test method name), but the output would looks much more readable with that.

Did you have some more thoughts about it, too?

[pchomik]

Hi @tknerr, Sorry for delay. I have a problem with time. Could you tell me how to execute such test because py.test raises error?

Br

[pchomik]

Ok. I read docs and I modify test to get same result. I'm looking into your idea.

[pchomik]

I have one more problem. My parametrize "suffix" doesn't have local. I have only e.g. ...[pip-9.0.6] Could share environment and complete test to reproduce your case?

Br

[pchomik]

In meantime I have a proposition based on one line example: [PASS] Package is intalled at version[local-ansible-2.1.0.0]

To enable support new parameter will be introduced: spec_test_parameters Every integer greater than 0 will activate support for such cases. Number is a max. expected parameters - in your case >3.

For configuration: spec_test_parameters = 4 result will be as follow:

p0 = local
p1 = ansible
p2 = 2.1.0.0
p3 = ''

All above variables will become available in spec_test_format like follow: spec_test_format = "[{result}] Package {p1} {name} {p2}"

Last step is to change test definition to:

def test_is_installed_at_version(Package, name, version):
    pass

Result will be: [Pass] Package ansible is installed at version 2.1.0.0

What do you think?

[tknerr]

@pchomik thanks for the prompt reply and all the ideas!

First of all, the above examples are using testinfra, which adds the [local-* suffix (because I ran them using the local backend). Also note that testinfra always adds that parameter (or at least I don't know how to suppress it). Sorry, pyhon newbie... ;-)

I like your proposal, just wondering on which scope you can define the spec_test_format. Is it only in the .ini file (i.e. globally), or can it be done on a per-module level, or even on a per-test method level (e.g. via annotations -- or do you call them decorators in python?)?

Being able to define that on a per-method level would be ideal from my point of view.

Also, just wondering: would we need spec_test_parameters at all? Or couldn't we simply trigger it based on the presence of any {p<N>} in spec_test_format?

Thanks for your help!!

[pchomik]

Good question about "spec_test_parameters" parameter. It should be easy to do that without this parameter. The first idea was to have global scope but I would like to have at least per module. I have to double check one more time what is available and where I have access.

[pchomik]

I looked into code and I will add support for format per module. I have to think about few more details but concept is almost ready.

[OddState]

Looking forward to this feature :) Parameterized test are a great tool but their output readability does leave space for improvement.