Out of IncrementingUuidGenerator capacity

tester12345132 commented 1 month ago

👓 What did you see?

It looks like IncrementingUuidGenerator cannot close/reuse sessionId´s if testcases have a short execution time
CucumberException: Out of IncrementingUuidGenerator capacity. Please reuse existing instances or use another UuidGenerator implementation instead.

Stacktrace: 
Exception in thread "main" java.util.ServiceConfigurationError: io.cucumber.core.eventbus.UuidGenerator: Provider io.cucumber.core.eventbus.IncrementingUuidGenerator could not be instantiated
    at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:804)
    at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:722)
    at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1395)
    at io.cucumber.core.runtime.UuidGeneratorServiceLoader.loadSingleUuidGeneratorOrDefault(UuidGeneratorServiceLoader.java:60)
    at io.cucumber.core.runtime.UuidGeneratorServiceLoader.loadUuidGenerator(UuidGeneratorServiceLoader.java:46)
    at io.cucumber.core.runtime.Runtime$Builder.build(Runtime.java:180)
    at io.cucumber.core.cli.Main.run(Main.java:85)
    at ta.TestRunner.main(TestRunner.java:59)
Caused by: io.cucumber.core.exception.CucumberException: Out of IncrementingUuidGenerator capacity. Please reuse existing instances or use another UuidGenerator implementation instead.
    at io.cucumber.core.eventbus.IncrementingUuidGenerator.<init>(IncrementingUuidGenerator.java:102)
    at jdk.internal.reflect.GeneratedConstructorAccessor87.newInstance(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:780)
    ... 7 more

✅ What did you expect to see?

I would expect, that some other UUidGenerator would be used, if MAX_SESSION_ID from IncrementingUuidGenerator is out of capacity or the existing instances are reused

📦 Which tool/library version are you using?

io.cucumber, cucumber-junit, v7.19.0 org.seleniumhq.selenium, selenium-java, v.4.17.0

🔬 How could we reproduce it?

Not sure about steps to reproduce, any calls against an api should do the trick

📚 Any additional context?

Workaround: Use io.cucumber, cucumber-junit v7.11.2 (last version before changes on UUID generator mentioned below)
Found this issue: https://github.com/cucumber/cucumber-jvm/pull/2703 merged in 04/2023
Some testcases have > 100 Steps
Some testcases are very quick < 200 ms
Nearly no break in between test executions
Before the Exception there are two big AND fast testcases (111 Steps - 0,354s and 52 Steps - 0,229s)
Steps and times in log

Executions.txt

mpkorstanje commented 1 month ago

The IncrementingUuidGenerator is designed to support a single Cucumber run with 256 sessions per class loader (one session per worker thread, one for main) and approximately 4.6*10^18 events per session. Both of these seem like reasonable parameters.

But from your stacktrace and log output it looks like you are running Cucumber multiple times. Once for each test case. Used in this way the IncrementingUuidGenerator can only support up to 256 runs. So you are operating somewhat out of the design parameters. Could you explain why? That might help inform a solution as I would normally expect a single Cucumber run to run all scenarios.

From your stack trace it also looks like you are not using cucumber-junit. While your project might have a dependency on cucumber-junit, but in your problem are only using functionality from cucumber-core. Could you demonstrate exactly how you run Cucumber?

mpkorstanje commented 1 month ago

Workaround: Use io.cucumber, cucumber-junit v7.11.2 (last version before changes on UUID generator mentioned below)

This I don't understand.

The IncrementingUuidGenerator was introduced in v7.12.0 and is not used by default. You must actively choose to use it. By rolling back this far, before its introduction, you are actively choosing not to use it.

So it seems unlikely that this problem was introduced by a change in Cucumber. Have you been adding test cases and are now exceeding 256 cases?

jkronegg commented 1 month ago

I confirm that IncrementingUuidGenerator is not used by default. To enable it, you need to add the following into your cucumber.properties:

cucumber.uuid-generator=io.cucumber.core.eventbus.IncrementingUuidGenerator

Alternatively, you may have a custom META-INF/services/io.cucumber.core.eventbus.UuidGenerator file which contains:

io.cucumber.core.eventbus.IncrementingUuidGenerator

The default Cucumber META-INF file contains both the RandomUuidGenerator and the IncrementingUuidGenerator, and the UuidGeneratorServiceLoader uses the RandomUuidGenerator by default.

Are you in one of these configuration ?

The IncrementingUuidGenerator was designed to get the highest performance from Cucumber. The typical use-case is when you have very short duration test scenarios (e.g. about 1 millisecond): in this situation, the Cucumber framework overhead starts to be bigger in percentage, so IncrementingUuidGenerator helps to reduce the framework impact. However, when having long-running test scenarios (>100 ms), I don't think you'll get a true advantage with IncrementingUuidGenerator.

Changing the IncrementingUuidGenerator logic with a more complex approach which reuse the session ids would probably decrease the performance, which is against the original idea.

Piscenois commented 1 month ago

Hi,

Same problem without configuring any custom UUID generator :

java.util.ServiceConfigurationError: io.cucumber.core.eventbus.UuidGenerator: Provider io.cucumber.core.eventbus.IncrementingUuidGenerator could not be instantiated
Caused by: io.cucumber.core.exception.CucumberException: Out of IncrementingUuidGenerator capacity. Please reuse existing instances or use another UuidGenerator implementation instead.

It doesn't happen with 7.0.0. To give you context, I'm upgrading Cucumber version (from version 1.2.6 to 7.20.0) following your guidelines. I change only some code for TypeRegistry and @CucumberContextConfiguration. The problem appears when I launch all the tests (maven command on Jenkins). With IntelliJ and only one test, it works.

mpkorstanje commented 1 month ago

@Piscenois looks like you are running into https://github.com/cucumber/cucumber-jvm/issues/2930. Can you try upgrading to 7.19.0 instead.

jkronegg commented 1 month ago

I reproduced the issue with the following unit test in UuidGeneratorServiceLoaderTest:

@Test
void test_case_2_robustness() {
    Options options = () -> null;
    UuidGeneratorServiceLoader loader = new UuidGeneratorServiceLoader(
            UuidGeneratorServiceLoaderTest.class::getClassLoader,
            options);
    UuidGenerator actual = null;
    for (int i=0; i<1000; i++) {
        actual = loader.loadUuidGenerator();
    }
    assertThat(actual, instanceOf(RandomUuidGenerator.class));
}

The error is:

java.util.ServiceConfigurationError: io.cucumber.core.eventbus.UuidGenerator: Provider io.cucumber.core.eventbus.IncrementingUuidGenerator could not be instantiated

    at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:586)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:813)
    at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:729)
    at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1403)
    at io.cucumber.core.runtime.UuidGeneratorServiceLoader.loadSingleUuidGeneratorOrDefault(UuidGeneratorServiceLoader.java:60)
    at io.cucumber.core.runtime.UuidGeneratorServiceLoader.loadUuidGenerator(UuidGeneratorServiceLoader.java:46)
    at io.cucumber.core.runtime.UuidGeneratorServiceLoaderTest.test_case_2_robustness(UuidGeneratorServiceLoaderTest.java:81)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: io.cucumber.core.exception.CucumberException: Out of IncrementingUuidGenerator capacity. Please reuse existing instances or use another UuidGenerator implementation instead.
    at io.cucumber.core.eventbus.IncrementingUuidGenerator.<init>(IncrementingUuidGenerator.java:102)
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
    at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:789)
    ... 8 more

I think this will be solved by https://github.com/cucumber/cucumber-jvm/pull/2931

Piscenois commented 1 month ago

@Piscenois looks like you are running into #2930. Can you try upgrading to 7.19.0 instead.

Thank you @mpkorstanje ! It works fine. 👍

cucumber / cucumber-jvm