Document test execution order

Xceptance / neodymium-library

A test automation library based on common other best practice open source libraries. It adds missing functionalities but does not reinvent the wheel. Just glues stuff together nicely and adds some sprinkles.

MIT License

80 stars 11 forks source link

Document test execution order #134

Closed h-arlt closed 4 years ago

h-arlt commented 4 years ago

The JUnit runner used by Neodymium executes the tests in a specific and fix order. This fact and what execution order is used in detail are not document anywhere.

Also, we need to determine the effect of annotating a test class with @FixMethodOrder as described here. In case there is no effect, fine. But that is something the user should be aware of. On the other hand, if the annotation has an effect on the test execution order, the user should be aware of that fact too.

occupant23 commented 4 years ago

Added a new documentation page for JUnit within the develop branch.

occupant23 commented 4 years ago

@h-arlt Please check if this suits your need.

occupant23 commented 4 years ago

@rkl-xc Please also recheck it.

h-arlt commented 4 years ago

Some remarks:

IMO page title is too specific; what about to use 'Test Execution' instead?
For Neodymium this results in a mixture of ordered and unordered.

An example would be nice, e.g. two methods (testA and testB) that are to be run with 3 data sets and two browsers.

Q1: Does Neodymium's test multiplication apply after JUnit's method ordering or before? Q2: When using one of JUnit's method sorters that work on the test method's name, has the method's data set (or browser) any impact on the execution order?

occupant23 commented 4 years ago

added more information and extended the example like recommended

h-arlt commented 4 years ago

Looks good to me.

rkl-xc commented 4 years ago

review done, added some suggestions, please have a look at it again