Courgette does not finish test run after running all test with given tags

ThisComarero commented 1 year ago

Hello,

I am using courgette in my project from last year, and recently I have started facing new issue. From time to time my courgette test run runs twice in one jenkins build. Once courgette runs all the test be given tag, and then proces does not finish but runs again all the test in the repository no matter which tag there have (It's looks like some default courgette command is run). Sometimes it fails before it runs all the test (I set the time limit for jenkins build).I was able to reproduce it on jenkins with normal test run, locally and even with cucumber dry run. I have no idea what can cause this problem.

I have noticed during full debug run two exceptions before courgette starts second run

java.lang.IllegalThreadStateException: process hasn't exited
    at java.base/java.lang.ProcessImpl.exitValue(ProcessImpl.java:521)
    at courgette.runtime.CourgetteFeatureRunner.run(CourgetteFeatureRunner.java:42)
    at courgette.runtime.CourgetteRunner.runFeature(CourgetteRunner.java:244)
    at courgette.runtime.CourgetteRunner.lambda$run$3(CourgetteRunner.java:82)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

java.lang.InterruptedException
    at java.base/java.lang.Object.wait(Native Method)
    at java.base/java.lang.Object.wait(Object.java:328)
    at java.base/java.lang.ProcessImpl.waitFor(ProcessImpl.java:495)
    at courgette.runtime.CourgetteFeatureRunner.run(CourgetteFeatureRunner.java:34)
    at courgette.runtime.CourgetteRunner.runFeature(CourgetteRunner.java:244)
    at courgette.runtime.CourgetteRunner.lambda$run$3(CourgetteRunner.java:82)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

I have noticed that this issue affect runs where there is a lot of test. I was not saw this on my build with smaller ammout of test ~50. My runner which runs the most tests has more than 700. Test are run in 12 threads with 2 reruns.

Another thing I have noticed is in the test report. Result are spitted into two sections in the allure report (depends on path?) 2022-12-09_08h57_22

In my project I have over 2000 scenarios split to different modules and stages (by severity). I have around 1300 scenarios automated already, and other scenarios are in repository as a part of documentation.

In my project I use Courgette version: 6.8.0 Gradle version: 6.8.1

prashant-ramcharan commented 1 year ago

Hello,

It seems something is terminating the JVM before Courgette is able to complete. Do you use System.exit() in your project or have any dependencies that calls exit() on the JVM?

Does this also happen when you don't rerun failed scenarios?

I have projects that run large volume of tests without issues and I've not been able to reproduce this. Are you able to provide an example project to reproduce this?

ThisComarero commented 1 year ago

Hello,

Unfortunately I cannot share my project with you.

Today I have perform some testing without reruns and was not able to reproduce this issue in ~20 builds, after that I have changed back to reruns and issue appears again in 3rd run.

I failed run there were some reruns as it is described in Courgette Test Statistics 2022-12-14_15h33_10 Again all te test were run 2554 instead of 712 (limited by tags)

I do not use System.exit() anywhere as well

prashant-ramcharan commented 1 year ago

Hi, I've not been able to reproduce this. I've tried with small and large test runs (500+).

If you can provide any demo / example project for me to reproduce this I will definitely take a look.

MBogdanski commented 1 year ago

So, I tried to debug it a little bit, and maybe found a clue what can be the issue. First of all, the issue is, that Courgette is executing another build of tests, meanwhile existing build (it occur on the end of the build, or in the middle of the build) As you can see, there are 2 packages which were executed (src/test/resources/features is correct one, second features/bom is package from features, it will take anothers packages, but we have timeout set to our build, so other packages are not executed further). When you look on timeline, we can see, that another build started executing without parallel On this build, this occured twice. So we tried to find the issue, and on 99% we can confirm, that issue only occur when we have used retries failed scenarios. I started investigating it and found, that sometimes featureUri to rerun (which is path to Feature file) is empty, because rerun file was empty: So maybe on CourgetteFeatureRunner, when process is executed with empty String featureUri, Cucumber is taking all Feature Files which he can find by default and that's why he execute all of the tests once again? Here it will be builder.command.get(108) -> on this index featureUri is stored. Can you please check it? If you want I can participate on it, just let me know and we can discuss it more on some Hangouts or another communicator. @prashant-ramcharan