kvas-it / pytest-console-scripts

Pytest plugin for testing console scripts
MIT License
78 stars 14 forks source link

print_result='on_fail' #49

Closed nick-youngblut closed 2 years ago

nick-youngblut commented 2 years ago

Unfortunately there's no easy way to print it only if the test fails because by the time a script run completes we don't yet know whether the test will fail or not

How about adding the option: print_result='on_fail', and in this case, the output is saved in memory until it is known whether the test has failed or not? If the test fails, then the output is printed, but otherwise it is discarded from memory.

kvas-it commented 2 years ago

Hi Nick.

Usually you only see the prints for the tests that fail anyway, because pytest capturing eats all the output from the tests that pass. I suppose you turn off the capturing and that's why the results printed in the successful tests get in the way. Am I right about that?

Cheers, Vasily

kvas-it commented 2 years ago

the output is saved in memory until it is known whether the test has failed or not?

After looking at the code... the problem here is that the result printout would be stored in RunResult, or perhaps it can be in the script_runner fixture, but neither of them knows if the test failed or not. RunResults might even be discarded by the time this is decided and pytest doesn't inform the fixtures about the outcome of the test (at least as far as I know).

However, you (the test writer) can know that the test has failed. What if I add a .print() method to RunResult, or perhaps a .print_results() method to the script_runner fixture? Then you could throw try/except around the body of the test and print the results in the exception handler.

nick-youngblut commented 2 years ago

However, you (the test writer) can know that the test has failed. What if I add a .print() method to RunResult, or perhaps a .print_results() method to the script_runner fixture? Then you could throw try/except around the body of the test and print the results in the exception handler.

I'm definitely willing to give it a try!

kvas-it commented 2 years ago

What do you think about #49 for starters? Here you can suppress automatic result printing with --hide-run-results and then do result.print() "manually" as necessary.

nick-youngblut commented 2 years ago

Thanks! I'll give it a try. It might be good to integrate that into the docs for anyone else who would like to use that feature... or they can just search for this issue.

kvas-it commented 2 years ago

Yeah, definitely should be in the docs. Let me know if this approach works for you, I'll add it to the docs and make a release.

kvas-it commented 2 years ago

Hi @nick-youngblut! So did #49 fix work for you in the end?

nick-youngblut commented 2 years ago

That seems to have worked. Thanks for adding the feature!

kvas-it commented 2 years ago

Released v.1.1.2 with manual result printing and added a note to the README. Thanks for the suggestion!