search

gintool / gin

GI in No Time - a Simple Microframework for Genetic Improvement
MIT License
43 stars 20 forks source link

Test poisoning over multiple repetitions in ExternalTestRunner #53

Closed GiovaniGuizzo closed 3 years ago

GiovaniGuizzo commented 3 years ago

I came across a problem with test poisoning when executing multiple repetitions of a same test suite while using ExternalTestRunner. Basically, in the second execution of a test case (more precisely, this one), it fails. Apparently there are some interactions between tests. Probably a static variable is being changed by other tests and it is making this test fail.

I fixed the bug in my development branch by forcing the start of a new process after each repetition, i.e., after all test cases in the test set are executed, the next test set repetition is done in a clean process (see it here). This makes the test pass (intended behaviour).

However, I am not sure if this should be considered default behaviour. In any case, I am happy to fix this bug, I just need your input on how to properly do it. A few options:

  1. Add this feature as default;
  2. Add this feature as default, but also add an option to Sampler to disable it;
  3. Do not add the feature as default, but add an option to enable it in Sampler.

Cheers, Giovani.

GiovaniGuizzo commented 3 years ago

It is worth noting that this also affects the InternalTestRunner, but I don't know how to fix it in this case. Maybe one CacheClassLoader for each repetition?

sandybrownlee commented 3 years ago

Can this be fixed by running with the -J option? This sets ExternalTestRunner.inNewSubprocess true which should make a new process for each test, IIRC.

GiovaniGuizzo commented 3 years ago

Can this be fixed by running with the -J option? This sets ExternalTestRunner.inNewSubprocess true which should make a new process for each test, IIRC.

It does work with -J. However, the cost becomes prohibitive when we have thousands of test cases (processes = test cases X repetitions). With resets only after each repetition, we remove the number of test cases from the equation and the overhead becomes negligible in those cases.

sandybrownlee commented 3 years ago

-jj option added to address this.