Initialization of io.cucumber.core.runner.TestStep causes performance issues

[flowerrrr]

After we upgrade our cucumber libs to a version that contained commit https://github.com/cucumber/cucumber-jvm/commit/58319035b7d3220a89bee376dc04ef068cf7c4ab we experienced a significant performance problem when running our cucumber scenarios.

Our test suite is quite big, we have ~10.000 scenarios with 5 to 30 steps. Execution time dropped from ~5 mins to ~10 mins after the version upgrade. That is, when the tests are run with Intellij. Same tests run with gradle build only shows a moderate performance decrease of about 20%, from 5 min to 6 min.

We could trace the problem to the initialization code of the class io.cucumber.core.runner.TestStep. Every time a new instance of the class is created (which happens for every step executed, ~100.000 times during a run of all scenarios) the method io.cucumber.core.runner.TestAbortedExceptions#createIsTestAbortedExceptionPredicate is executed. Since we don't have all the tested exceptions in our class path (when run in the IDE), every invocation of the method produces 3 ClassNotFoundExceptions and these seem to be the cause of our performance problems.

I see two solutions:

• make io.cucumber.core.runner.TestStep#isTestAbortedException static, so it is initialized only once
• defer initialization of the Predicate until it is actually needed (in case an exception has to be mapped).

If need be, i could provide a PR.

Thank you.

[mpkorstanje]

I would be happy to accept a PR for this.

@jkronegg which of these do you think is the better option?

[jkronegg]

@flowerrrr Are you sure this performance issue is related to the Predicate<Throwable> instance creation ? IMHO, most of the execution time comes from the Predicate<Throwable> execution (you can't get a ClassNotFoundException until the predicate is executed), see https://github.com/cucumber/cucumber-jvm/blob/02af97eeae2a75289d08d622d0c3ee36e31fc684/cucumber-core/src/main/java/io/cucumber/core/runner/TestAbortedExceptions.java#L26 If I'm correct, you should have a lot of Exceptions that are thrown in your code. If you don't notice them, could it be possible that they are swallowed somewhere ? There are some swallowing mechanisms in Cucumber and I not sure this is working properly, e.g. see https://github.com/cucumber/cucumber-jvm/issues/2748

The Predicate<Throwable> execution performance has been improved through https://github.com/cucumber/cucumber-jvm/issues/2666 : performance can still be 2x faster by caching the classes in a static classloader, but this will change the code behavior, so I'm not sure this is the way to go.

This aside, regarding your proposed solutions, and considering that Exceptions should be exceptions, I would defer the Predicate creation until we need to evaluate it. This way, TestCase instance creation will be faster.

[flowerrrr]

Oh, i see that the issue is already fixed in version v7.11.0 with this commit: https://github.com/cucumber/cucumber-jvm/commit/f53abb64dd9b3990c34d09514f3bba3112e34bd5 Sorry for the confusion, i was still on an old version.

[jkronegg]

@flowerrrr 5 minutes for 10'000 scenarios is relatively fast, but you could have even better performance with Cucumber 7.12.0 by adding the following property:

cucumber.uuid-generator=io.cucumber.core.eventbus.IncrementingUuidGenerator

see https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-core#event-bus

[flowerrrr]

Adding the property did produce a modest improvement of ~5%. Thank you for the tip.