`NullPointerException` on `extractTestCase`

[marojor]

First of all, thanks for courgette-jvm, it's a great tool that has allowed us to boost the execution of our tests.

Running 5.12.0 I've seen following exception occasionally:

    java.lang.NullPointerException
        at courgette.runtime.CourgetteNdJsonCreator.extractTestCase(CourgetteNdJsonCreator.java:183)
        at courgette.runtime.CourgetteNdJsonCreator.addMessage(CourgetteNdJsonCreator.java:115)
        at courgette.runtime.CourgetteNdJsonCreator.lambda$null$1(CourgetteNdJsonCreator.java:78)
        at java.util.ArrayList.forEach(ArrayList.java:1259)
        at courgette.runtime.CourgetteNdJsonCreator.lambda$getScenarioMessages$2(CourgetteNdJsonCreator.java:78)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at courgette.runtime.CourgetteNdJsonCreator.getScenarioMessages(CourgetteNdJsonCreator.java:72)
        at courgette.runtime.CourgetteNdJsonCreator.createScenarioMessages(CourgetteNdJsonCreator.java:56)
        at courgette.runtime.CourgetteReporter.createMessages(CourgetteReporter.java:128)
        at courgette.runtime.CourgetteReporter.<init>(CourgetteReporter.java:33)

I've not being able to reproduce the problem consistently yet 😞 so can't provide additional details.

[prashant-ramcharan]

Hi @marojor

I've not been able to reproduce this myself.

Can you please advise on the following:

1. Approximately, how many tests are you running?
2. How many threads are you using?
3. Are you using CourgetteRunLevel.SCENARIO or CourgetteRunLevel.FEATURE ?
4. Does this error usually happen when running locally or in CI ?

[prashant-ramcharan]

I will be upgrading to Cucumber v7 soon and as a result this class will change so this error may fix itself 🙂

[marojor]

I've been trying to reproduce the issue to get additional info but with no much success for now.

Regarding your questions:

1. ~150 tests
2. 3 threads
3. CourgetteRunLevel.FEATURE with 4 feature files.
4. CI. No luck to reproduce it manually yet

[marojor]

Ok, forget what I was saying before, I'm able to reproduce the problem now locally, and it seems to happen while doing retries.

In the scenario I'm working on, I keep on running the same two tests (which are meant to fail) over an over, and eventually I get the exception because List<Messages.Envelope> envelopes is null 🤷

[marojor]

OK, a bit more of info.

The only way I've been able to reproduce it consistently now is to set a higher number of threads than tests.

When that's the case optimizedThreadCount does as expected and set the number of threads to runnerInfoList.size().

Still, with that setup, occasionally I get the NullPointerException reported above.

Looking at the code, there's something that makes me scratch my head:

        try {
            runtimePublisher.publish(CourgetteEvent.TEST_RUN_STARTED);
            executor.invokeAll(runners);
        } catch (InterruptedException e) {
            printExceptionStackTrace(e);
        } finally {
            runtimePublisher.publish(CourgetteEvent.TEST_RUN_FINISHED);
            executor.shutdownNow();
        }

I might be missing something obvious here probably, but looking at the docs of the executor, it's supposed to return a list of Futures with the result of the tasks, but I don't see where in the code it ensured that the tasks are done by the time shutdownNow() is called.

My current naive guess now is that when the number of threads in the executor is the same as the number of tasks to be executed, sometimes the tasks are not fast enough to finish on time, generating a null envelope which triggers an exception when generating the reports.

[prashant-ramcharan]

Thanks for the additional info.

1. In addition to the 2 test scenarios that are meant to fail, how many other scenarios are in the feature file?
2. Does this only happen when rerunFailedScenarios is set true?
3. Does it also happen when running using CourgetteRunLevel.SCENARIO ?

I tried it with the example project and set the threads to 100 but didn't get the error.

If you can, please provide a project that I can reproduce this error?

It may be worth waiting for me to upgrade Courgette to version 6.0.0 as I need to make changes in the CourgetteNdJsonCreator class as a result of the changes in Cucumber 7.0.0

[marojor]

1. No other scenario, just those two
2. rerunFailedScenarios is not set. We use a retries system outside courgette, so as far as courgette knows is just a list of 2 scenarios.
3. Yes, CourgetteRunLevel.SCENARIO is set.

Waiting for 7.0.0 sounds fair, we've reduced the number of threads and so far didn't hit the crash again, so we can wait with current setup.

If I'm able to create a small project that reproduces the error I'll share it with you.

[prashant-ramcharan]

I've just released Courgette version 6.0.0.

Let me know if you see the same issue as I've upgraded Cucumber to version 7.0.0

Thanks

[prashant-ramcharan]

Closing issue - please reopen if still an issue and can be reproduced.