Make plugin fixtures work with parameterized tests

[xymaxim]

Closes #4.

While adding the feature, I considered the following things about the resulting pattern filenames:

1. Must contain only valid path characters
2. Must be unique, non-ambiguous for every test

Also, (3) the filename length must not exceed the limit, but it's not in this PR.

As a result, the plugin will store and read pattern files having the following filename format:

<test-function-name>[-<callspec>[-<number-suffix>]][-<system-suffix>].<ext>

Reasoning

1. Pytest escapes non-ASCII characters in test IDs. I lean towards POSIX-compliant characters and just to remove non-valid ones. The white spaces are replaced with underscore.

2. After removing the invalid characters, callspec IDs might be non-unique: e.g., [“2*2”, “2+2”] -> [“22”, ”22”]. The solution is to use a numbered suffix added to the callspec part: [“22”, “22-1”]. I’ve added a test for this case to ensure that expectation files are not overwritten during tests.

   (This is only observed for test IDs subject to non-valid characters removing. For a counter-case, e.g., [22, “22”] -> [“220”, “221”], see this issue.)

3. There’s no limitation on the length of callspec IDs. With user provided strings, it can easily hit the PC_NAME_MAX value. This is not part of this PR, because I think it’s more about the filename itself than parametrization. I’ll create a new one later. As a workaround for now, someone can use ids argument.

Mismatch between test IDs and pattern filenames

With the proposed approach, test IDs in the terminal output don’t match our filenames. However, the pattern filename is printed in mismatch reports for failed tests. As for reporting hooks, the filename can be accessed via plugin fixtures. So, it’s not an issue for me. Even if someone use a special hook, pytest_make_parametrize_id, the pattern filenames will be consistent.

[xymaxim]

As a result, the plugin will store and read pattern files having the following filename format:

<test-function-name>[-<callspec>[-<number-suffix>]][-<system-suffix>].<ext>

To be consistent, a hyphen is used as a separator between parts after the original test function name.

For example, given this Python test function and its test IDs:

@pytest.mark.parametrize(
    "x,y",
    [
        ("hyphen-and space separated", "123"),
        ("hyphen-and_space_separated", "123"),
    ]
)
def test_parametrized(x, y): ...

(1) test_parametrized[hyphen-and space separated-123]
(2) test_parametrized[hyphen-and_space_separated-123]

The following pattern files will be created:

(1) test_parametrized-hyphen-and_space_separated-123-linux.out
(2) test_parametrized-hyphen-and_space_separated-123-1-linux.out

The parametrization part is not very readable. Do you think it makes sense to use an underscore (with a double underscore after the test function name part) instead:

(1) test_parametrized__hyphen-and_space_separated-y0_linux.out
(2) test_parametrized__hyphen-and_space_separated-y0-1_linux.out

Or leave it as it is?

[zaufi]

Thank you very much for the PR! :pray:

What do you think if the urllib.parse.quote + replace / w/ _ can help to make the filename valid and keep its uniqueness yet be readable/predictable by end-users?

[zaufi]

For example, this code can produce filesystem-safe names and still IMHO quite readable:

    result /= (
        urllib.parse.quote(request.node.name, safe='[]')
      + ('-' + platform.system() if use_system_suffix else '')
      + ext
      )

Example:

…/parametrized_case_test0/test_parametrized[0-y].out
…/parametrized_case_test0/test_parametrized[1-some%20words].out
…/parametrized_case_test0/test_parametrized[2-~%2Fsome%2Fpath%2F].out

[xymaxim]

Pretty straightforward solution! It looks more than legit. And the filenames can be unquoted back:

$ ls *.out | xargs -I{} python -c 'from urllib.parse import unquote as u; print(u("{}"))'
…/parametrized_case_test0/test_parametrized[0-y].out
…/parametrized_case_test0/test_parametrized[1-some words].out
…/parametrized_case_test0/test_parametrized[2-~/some/path/].out

[xymaxim]

Should I rebase all commits in this PR and add new changes or we can close it and open a new one?

[zaufi]

Please, just edit current PR by force-pushing new commits.

On Wed, Jan 10, 2024, 20:42 Maxim Stolyarchuk @.***> wrote:

Should I rebase all commits in this PR and add new changes or we can close it and open a new one?

— Reply to this email directly, view it on GitHub https://github.com/zaufi/pytest-matcher/pull/10#issuecomment-1885211087, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEF626CRF46TRRMZYVU3R3YN3AGTAVCNFSM6AAAAABBUSV3ROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBVGIYTCMBYG4 . You are receiving this because you were assigned.Message ID: @.***>

[xymaxim]

Special thanks for the cool project!

I really like how the concept of Cram tests was embodied into pytest testing protocol with a nice usage of fixtures.