Make the fixture content editable

[xymaxim]

Let me extend #1 not only to accessing the content of fixtures, but also to editing it.

Use case

It could be useful to check non-reproducible outputs. Let me reference here to this (https://discuss.ocaml.org/t/cram-tests-on-short-notice/6256/9) discussion about coping with such tests in Dune (https://dune.build/) build system.

My current workflow of testing such cases is:

store -> modify a test function -> check

Here is an example of matching a random temporary directory:

def capture_some_output(value):
    return f"Temp directory: {value}"

# Run 1 (store)
def test_temp_directory(tmp_path, expected_out):
    actual = capture_some_output(tmp_path)
    assert expected_out == actual

# Run 2 (check)
def test_temp_directory(tmp_path, expected_out):
   actual = capture_some_output(tmp_path)
   expected_edited = re.sub(
       r"Temp directory:\s.+",
       f"Temp directory: {tmp_path}",
       expected_out._pattern_filename.read_text()
   )
   assert expected_edited == actual

The main problem with this approach is that each subsequent storing requires modifying the test function back to the previous version every time.

Proposed solution

After putting the content reading out into the fixture function, it could be possible to simplify the workflow to just two steps (as for regular tests):

store -> check (act, edit if needed, assert)

The example above can be rewritten as follows:

def test_editing(tmp_path, expected_out):
   actual = f"Temp directory: {tmp_path}"
   if not expected_out.store:
       expected_out.content = re.sub(
           "Temp directory: .+",
           f"Temp directory: {tmp_path}",
           expected_out.content
       )
    assert expected_out == actual

While the expected_out.content variable can be editable, we can keep the original content in expected_out._expected_file_content as-is.

This allows to create reproducible and backward compatible tests to check non-deterministic outputs.

What do you think about these changes? Do you maybe have other ideas regarding this issue?

[zaufi]

How about this:

def test_editing(tmp_path, expected_out):
    actual = f"Temp directory: {tmp_path}"
    assert expected_out.edit(
        lambda content: re.sub(
            "Temp directory: .+"
          , f"Temp directory: {tmp_path}"
          , content
          )
      ) == actual

The edit() method accepts Callable[[str], str] and assigns the result to internal member _edited: str| None used for actual comparison instead of _expected_file_content if set.

A bit more advanced idea is to use Jinja2 to substitute local variables into an output expectations template:

def test_substitute_variables(tmp_path, expected_out):
    actual = f"Temp directory: {tmp_path}"
    assert expected_out.substitute_vars() == actual

Where the expected pattern file is a Jinja2 template:

Temp directory: {{ tmp_path }}

[xymaxim]

How about this:

I like how the first example is quite explicit and that the edit() method can be called multiple times.

A bit more advanced idea is to use Jinja2 to substitute local variables into an output expectations template:

As I understand correctly, this won't handle storing such templates automatically, will it?

It’ll be very useful to have an ability to create such templates on the fly, without needs to manually convert some values into placeholders afterward. Anyway, personally, I would find a use for something like this.

This discussion somewhat reminds me of the correlating idea of substituting some common paths used here, but your second approach with a direct call could feel more transparent. It would be more flexible if additional context template variables can be passed to the function. For example, I have a fixture called app_temp_dir = tmp_path / <temp-subpath> and I want to use this particular template variable: {{ app_temp_dir }}:

def test_substitute_variables(tmp_path, app_temp_dir, expected_out):
    actual = f"Temp directory: {tmp_path}\nApp directory: {app_temp_dir}"
    assert expected_out.substitute_vars(
        extra_context={“app_temp_dir”: app_temp_dir}
    ) == actual

[zaufi]

Oh... I'm sorry. I misread your initial description. I thought you were trying to eliminate mutating parts before asserting ;)

However, the same approach can be used for storing captured patterns :)

Meanwhile, dunno if the idea w/ Jinja applies to store pattern files.

Anyway, it'll be nice to allow a user to "edit" captured output (somehow per test basis) before storing it. It'll be useful when the stored pattern is used as a regular expression for expected_out.match(), and it's always annoying to manually escape symbols that have special meanings for regexes (*, +, ., &etc). Having at least this "filter" (standard editing action) will help a lot...

[zaufi]

One more idea:

@pytest_matcher.on_store_out_replace(
    'Temp directory: .+'
  )
def test_editing(tmp_path, expected_out):
    actual = capture_some_output(tmp_path)
    assert expected_out == actual

The on_store_out_replace decorator accepts arbitrary strings to perform re.sub(line, line, content) to match the mutating parts and replace 'em with the same regex for future matches by expected_out.match().

[xymaxim]

Oh... I'm sorry. I misread your initial description. I thought you were trying to eliminate mutating parts before asserting ;)

However, the same approach can be used for storing captured patterns :)

Meanwhile, dunno if the idea w/ Jinja applies to store pattern files.

Anyway, it'll be nice to allow a user to "edit" captured output (somehow per test basis) before storing it. It'll be useful when the stored pattern is used as a regular expression for expected_out.match(), and it's always annoying to manually escape symbols that have special meanings for regexes (*, +, ., &etc). Having at least this "filter" (standard editing action) will help a lot...

Actually, as we can see, there are various different cases for pre- and post-editing, and we've already found quite a few very flexible ways to solve it. As I understand, there are no best practices here, and it all depends on the user needs and preferences.