Question about test execution order

smarie / python-pytest-harvest

Store data created during your `pytest` tests execution, and retrieve it at the end of the session, e.g. for applicative benchmarking purposes.

https://smarie.github.io/python-pytest-harvest/

BSD 3-Clause "New" or "Revised" License

61 stars 8 forks source link

Question about test execution order #22

Closed Sup3rGeo closed 5 years ago

Sup3rGeo commented 5 years ago

Hi,

Is there anything that guarantees that test_synthesis runs after all other tests, and therefore the requested fixture will not miss information from any other test/fixture that might run after it?

Or it is up to the user to guarantee that test_synthesis runs after everything?

smarie commented 5 years ago

As explained in the documentation there are many places where you might wish to execute your synthesis code.

pytest guarantees that execution order is deterministic and follows the order of appearance in files, so you can even insert several test_synthesis_1/2/3.. functions at the position of your choice to get partial synthesis if that makes sense. I would usually create one at the end of each module where I need synthesis collection.

Today I deployed a new 1.7.0 together with a complete API reference in the docs page, and updated slightly the "advanced usage" page. You might want to go through it to better understand the philosophy.

Sup3rGeo commented 5 years ago

Thanks for pointing out the documentation, totally missed it.

You mentioned the sessionfinish hook, but I would like to suggest another hook-related approach: Using pytest_collect_modifyitems to find the test synthesis test and put it at the end of the list.

smarie commented 5 years ago

I have disagree with this proposal as

it only works for the particular case where the "synthesis" test is unique and session-wide. I have plenty of usage cases where I have one per module :)
and for this particular case (a single test synthesis to execute after all other tests) it is not better than positioning that test at the end of all tests. For example in a module named "test_z.py" or at the end of the last test module, whatever it is. Although in that case I would rather suggest to use the "session finish" hook - probably more readable than creating a dummy module wih a strange name.

Let me know if there is anything that I could improve in the doc to make it more straightforward - it was created in several "waves" and therefore there is probably a lot of room for improvement ! Cheers

Sup3rGeo commented 5 years ago

Agree with your first point, this would really be for the case where we have a single synthesis for the session. In fact it might be the case that for my use case I might also want to do it per module :)

The only inconvenient about using the session finish hook, if I am not mistaken, it runs after the whole test session is already completed - meaning that I cannot actually have a test_synthesis that is part of the same test run. Or that is not correct?

smarie commented 5 years ago

Perfectly True yes. That would not appear as a "test". If that's needed, positioning a test_synthesis as the last test makes sense again.

I let you close the topic if you feel that answered your question (we can still continue the conversation here of course)

smarie commented 5 years ago

Hi @Sup3rGeo , is this topic still valid ? otherwise we can maybe close it

Sup3rGeo commented 5 years ago

@smarie sure, doing it!

smarie commented 5 years ago

thanks for the feedback :)