search

MichaelTamm / junit-toolbox

Useful classes for writing automated tests with JUnit 4
Other
140 stars 25 forks source link

Allow ordered declarations using WildcardPatternSuite #25

Open jgreffe opened 4 years ago

jgreffe commented 4 years ago

Hello,

when I declare a suite with this, classes are randomly executed:

@RunWith(WildcardPatternSuite.class)
@Suite.SuiteClasses({FirstTest.class, AnotherTest.class})
@SuiteClasses("?*/**/*Test.class")

Maybe using LinkedHashSet would resolve this?

MichaelTamm commented 3 years ago

Hi jgreffe,

I really appreciate that you not only created an issue, but provided a PR too!

But I really wonder, why do you care about the test order in the first place? I am a little bit hesitant to add more complexity to junit-toolbox without seeing a valid reason for doing so. Could you please tell me more, why you want that tests are executed in a certain order?

jgreffe commented 3 years ago

Hi jgreffe,

I really appreciate that you not only created an issue, but provided a PR too!

But I really wonder, why do you care about the test order in the first place? I am a little bit hesitant to add more complexity to junit-toolbox without seeing a valid reason for doing so. Could you please tell me more, why you want that tests are executed in a certain order?

Hello Michael,

this PR actually has two features:

This allows to have structured tests always executed in the same order so it's predictable. Sometimes you may need specific cases to be run before all others. And sometimes you may need test cases on level 1 to be run before the ones on level 2.

Not sure if it's clear enough?

MichaelTamm commented 3 years ago

Hi again,

depending on tests to be executed in a predictable order sounds like a code smell to me: Tests should be independent from each other.

The best execution order (in my opinion) is: random. (Or maybe: failed first, then random)

I'm still hesitant to merge your PR, because I'm not convinced that it is a good idea.

On the other hand side, predictability isn't such a bad thing. ;)

Would the following compromise be good enough for you: We just replace all HashSets with LinkedHashSets.

jgreffe commented 3 years ago

Hey Michael,

just pointing out this: http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#runOrder

failsafe allows Suites being ordered to give more predictability on Suite level.

Would also like having the same predictability when Tests are resolved from 

@RunWith(WildcardPatternSuite.class)
@Suite.SuiteClasses({FirstTest.class, AnotherTest.class})
@SuiteClasses("?*/**/*Test.class")

depending on their level in dir/package structure.

But it's up to you :)