Open xymaxim opened 9 months ago
How about this:
def test_editing(tmp_path, expected_out):
actual = f"Temp directory: {tmp_path}"
assert expected_out.edit(
lambda content: re.sub(
"Temp directory: .+"
, f"Temp directory: {tmp_path}"
, content
)
) == actual
The edit()
method accepts Callable[[str], str]
and assigns the result to internal member _edited: str| None
used for actual comparison instead of _expected_file_content
if set.
A bit more advanced idea is to use Jinja2 to substitute local variables into an output expectations template:
def test_substitute_variables(tmp_path, expected_out):
actual = f"Temp directory: {tmp_path}"
assert expected_out.substitute_vars() == actual
Where the expected pattern file is a Jinja2 template:
Temp directory: {{ tmp_path }}
How about this:
I like how the first example is quite explicit and that the edit()
method can be called multiple times.
A bit more advanced idea is to use Jinja2 to substitute local variables into an output expectations template:
As I understand correctly, this won't handle storing such templates automatically, will it?
It’ll be very useful to have an ability to create such templates on the fly, without needs to manually convert some values into placeholders afterward. Anyway, personally, I would find a use for something like this.
This discussion somewhat reminds me of the correlating idea of substituting some common paths used here, but your second approach with a direct call could feel more transparent. It would be more flexible if additional context template variables can be passed to the function. For example, I have a fixture called app_temp_dir = tmp_path / <temp-subpath>
and I want to use this particular template variable: {{ app_temp_dir }}
:
def test_substitute_variables(tmp_path, app_temp_dir, expected_out):
actual = f"Temp directory: {tmp_path}\nApp directory: {app_temp_dir}"
assert expected_out.substitute_vars(
extra_context={“app_temp_dir”: app_temp_dir}
) == actual
Oh... I'm sorry. I misread your initial description. I thought you were trying to eliminate mutating parts before asserting ;)
However, the same approach can be used for storing captured patterns :)
Meanwhile, dunno if the idea w/ Jinja applies to store pattern files.
Anyway, it'll be nice to allow a user to "edit" captured output (somehow per test basis) before storing it. It'll be useful when the stored pattern is used as a regular expression for expected_out.match()
, and it's always annoying to manually escape symbols that have special meanings for regexes (*
, +
, .
, &etc). Having at least this "filter" (standard editing action) will help a lot...
One more idea:
@pytest_matcher.on_store_out_replace(
'Temp directory: .+'
)
def test_editing(tmp_path, expected_out):
actual = capture_some_output(tmp_path)
assert expected_out == actual
The on_store_out_replace
decorator accepts arbitrary strings to perform re.sub(line, line, content)
to match the mutating parts and replace 'em with the same regex for future matches by expected_out.match()
.
Oh... I'm sorry. I misread your initial description. I thought you were trying to eliminate mutating parts before asserting ;)
However, the same approach can be used for storing captured patterns :)
Meanwhile, dunno if the idea w/ Jinja applies to store pattern files.
Anyway, it'll be nice to allow a user to "edit" captured output (somehow per test basis) before storing it. It'll be useful when the stored pattern is used as a regular expression for
expected_out.match()
, and it's always annoying to manually escape symbols that have special meanings for regexes (*
,+
,.
, &etc). Having at least this "filter" (standard editing action) will help a lot...
Actually, as we can see, there are various different cases for pre- and post-editing, and we've already found quite a few very flexible ways to solve it. As I understand, there are no best practices here, and it all depends on the user needs and preferences.
Let me extend #1 not only to accessing the content of fixtures, but also to editing it.
Use case
It could be useful to check non-reproducible outputs. Let me reference here to this (https://discuss.ocaml.org/t/cram-tests-on-short-notice/6256/9) discussion about coping with such tests in Dune (https://dune.build/) build system.
My current workflow of testing such cases is:
Here is an example of matching a random temporary directory:
The main problem with this approach is that each subsequent storing requires modifying the test function back to the previous version every time.
Proposed solution
After putting the content reading out into the fixture function, it could be possible to simplify the workflow to just two steps (as for regular tests):
The example above can be rewritten as follows:
While the
expected_out.content
variable can be editable, we can keep the original content inexpected_out._expected_file_content
as-is.This allows to create reproducible and backward compatible tests to check non-deterministic outputs.
What do you think about these changes? Do you maybe have other ideas regarding this issue?