Cucumber support

[j-martinez-dev]

Description Cucumber is the most used tool for BDD testing and it supports Junit5, the problem is that the support for Junit 5 is using the Test engine instead the junit extensions. It would be great to have @QuarkusCucumberTest that take care of this problem

Implementation ideas I think that something like https://github.com/cucumber/cucumber-jvm/tree/main/spring could be implemented

[j-martinez-dev]

Hi @stuartwdouglas , im writing you because i want to know if we could work together to make this happen. I know that you show interest in the past

461

Do you have a simple @QuarkusTest + Cucumber example I could look at to see if I could get it to work? I have never used cucumber, so I am not really sure what the expectation is here, but I think I would need to write a custom quarkus-cucmber module to integrate them.

In fact, I think that the cucumber integration is not difficult but it needs a lot of quarkus testing knowledge.

Cucumber has already integration with other similary cases (spring, micronaut) and as you can see in the cucumber documentation, we need to implement a simple class: ObjectFactory

In this class we need to start/stop the application and provide a method to acces to the beans. In quarkus, i think that all this is wrapped in @QuarkusTest annotation.

@juanmanuelz said in the same post that he make it work, but not in a clean way, so i think that you can help up to make it work for quarkus.

I can contribute with an example of quarkus + cucumber projet, is need it, i can contact @juanmanuelz to have a working example before you take a look.

Do you think that you can help us?

I'm really excited about all the work in the quarkus ecosystem and i would be really thankfull if you can help us

(Sorry for my english)

[tisoft]

I am currently using this class as a workaround to combine cucumber with QuarkusTest:

import java.lang.reflect.Field;
import java.net.URI;
import java.time.Clock;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicReference;

import javax.enterprise.inject.Instance;
import javax.enterprise.inject.spi.CDI;

import io.cucumber.core.backend.ObjectFactory;
import io.cucumber.core.eventbus.EventBus;
import io.cucumber.core.feature.FeatureParser;
import io.cucumber.core.options.CommandlineOptionsParser;
import io.cucumber.core.options.RuntimeOptions;
import io.cucumber.core.options.RuntimeOptionsBuilder;
import io.cucumber.core.plugin.PluginFactory;
import io.cucumber.core.plugin.Plugins;
import io.cucumber.core.plugin.PrettyFormatter;
import io.cucumber.core.runner.Runner;
import io.cucumber.core.runtime.CucumberExecutionContext;
import io.cucumber.core.runtime.ExitStatus;
import io.cucumber.core.runtime.FeaturePathFeatureSupplier;
import io.cucumber.core.runtime.FeatureSupplier;
import io.cucumber.core.runtime.ObjectFactorySupplier;
import io.cucumber.core.runtime.ScanningTypeRegistryConfigurerSupplier;
import io.cucumber.core.runtime.TimeServiceEventBus;
import io.cucumber.core.runtime.TypeRegistryConfigurerSupplier;
import io.cucumber.java.JavaBackendProviderService;
import io.cucumber.plugin.event.EventHandler;
import io.cucumber.plugin.event.PickleStepTestStep;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestStepFinished;
import io.quarkus.test.junit.QuarkusTest;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.DynamicContainer;
import org.junit.jupiter.api.DynamicNode;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import org.junit.jupiter.api.TestInstance;
import org.junit.platform.console.ConsoleLauncher;

@QuarkusTest
@TestInstance (TestInstance.Lifecycle.PER_CLASS)
class DynamicCucumberTest {
   private static String [] mainArgs = new String [] {};

   @TestFactory
   List <DynamicNode> getTests () {
      try {
         // We run in a different ClassLoader then "main", so we need to grab any cli arguments from the SystemClassLoader
         Class <?> aClass = ClassLoader.getSystemClassLoader ().loadClass (this.getClass ().getName ());
         Field aClassDeclaredField = aClass.getDeclaredField ("mainArgs");
         aClassDeclaredField.setAccessible (true);
         DynamicCucumberTest.mainArgs = (String []) aClassDeclaredField.get (aClass);
      } catch (NoSuchFieldException | ClassNotFoundException | IllegalAccessException e) {
         e.printStackTrace ();
      }

      EventBus eventBus = new TimeServiceEventBus (Clock.systemUTC (), UUID::randomUUID);

      final FeatureParser parser = new FeatureParser (eventBus::generateId);

      RuntimeOptionsBuilder commandlineOptionsParser = new CommandlineOptionsParser (System.out).parse (mainArgs);

      RuntimeOptionsBuilder runtimeOptionsBuilder = new RuntimeOptionsBuilder ();
      runtimeOptionsBuilder.addDefaultFeaturePathIfAbsent ();
      runtimeOptionsBuilder.addDefaultGlueIfAbsent ();
      runtimeOptionsBuilder.addDefaultFormatterIfAbsent ();
      runtimeOptionsBuilder.addDefaultSummaryPrinterIfAbsent ();

      runtimeOptionsBuilder.addGlue (URI.create ("classpath:/" + Steps.class.getPackage ().getName ().replace (".", "/")));

      RuntimeOptions runtimeOptions = runtimeOptionsBuilder.build (commandlineOptionsParser.build ());
      FeatureSupplier featureSupplier = new FeaturePathFeatureSupplier ( () -> Thread.currentThread ().getContextClassLoader (), runtimeOptions, parser);

      final Plugins plugins = new Plugins (new PluginFactory (), runtimeOptions);
      plugins.addPlugin (new PrettyFormatter (System.out));

      final ExitStatus exitStatus = new ExitStatus (runtimeOptions);
      plugins.addPlugin (exitStatus);
      if (runtimeOptions.isMultiThreaded ()) {
         plugins.setSerialEventBusOnEventListenerPlugins (eventBus);
      } else {
         plugins.setEventBusOnEventListenerPlugins (eventBus);
      }
      ObjectFactory objectFactory = new CdiObjectFactory ();

      ObjectFactorySupplier objectFactorySupplier = () -> objectFactory;

      TypeRegistryConfigurerSupplier typeRegistryConfigurerSupplier = new ScanningTypeRegistryConfigurerSupplier ( () -> Thread.currentThread ()
                                                                                                                               .getContextClassLoader (),
                                                                                                                   runtimeOptions);

      Runner runner = new Runner (eventBus,
                                  Collections.singleton (new JavaBackendProviderService ().create (objectFactorySupplier.get (),
                                                                                                   objectFactorySupplier.get (),
                                                                                                   () -> Thread.currentThread ()
                                                                                                               .getContextClassLoader ())),
                                  objectFactorySupplier.get (),
                                  typeRegistryConfigurerSupplier.get (),
                                  runtimeOptions);

      CucumberExecutionContext context = new CucumberExecutionContext (eventBus, exitStatus, () -> runner);

      List <DynamicNode> features = new LinkedList <> ();
      features.add (DynamicTest.dynamicTest ("Start Cucumber", context::startTestRun));

      featureSupplier.get ().forEach (f -> {
         List <DynamicTest> tests = new LinkedList <> ();
         tests.add (DynamicTest.dynamicTest ("Start Feature", () -> context.beforeFeature (f)));
         f.getPickles ().forEach (p -> tests.add (DynamicTest.dynamicTest (p.getName (), () -> {
            AtomicReference <TestStepFinished> resultAtomicReference = new AtomicReference <> ();
            EventHandler <TestStepFinished> handler = event -> {
               if (event.getResult ().getStatus () != Status.PASSED) {
                  // save the first failed test step, so that we can get the line number of the cucumber file
                  resultAtomicReference.compareAndSet (null, event);
               }
            };
            eventBus.registerHandlerFor (TestStepFinished.class, handler);
            context.runTestCase (r -> r.runPickle (p));
            eventBus.removeHandlerFor (TestStepFinished.class, handler);

            if (mainArgs.length == 0) {
               // if we have no main arguments, we are running as part of a junit test suite, we need to fail the junit test explicitly
               if (resultAtomicReference.get () != null) {
                  Assertions.fail ("failed in " + f.getUri () + " at line " + ((PickleStepTestStep) resultAtomicReference.get ().getTestStep ()).getStep ().getLocation ().getLine (),
                                   resultAtomicReference.get ().getResult ().getError ());
               }
            }
         })));
         features.add (DynamicContainer.dynamicContainer (f.getName ().orElse (f.getSource ()), tests.stream ()));
      });

      features.add (DynamicTest.dynamicTest ("Finish Cucumber", context::finishTestRun));

      return features;
   }

   public static class CdiObjectFactory implements ObjectFactory {
      public CdiObjectFactory () {
      }

      public void start () {

      }

      public void stop () {

      }

      public boolean addClass (Class <?> clazz) {
         return true;
      }

      public <T> T getInstance (Class <T> type) {
         Instance <T> selected = CDI.current ().select (type);
         if (selected.isUnsatisfied ()) {
            throw new IllegalArgumentException (type.getName () + " is no CDI bean.");
         } else {
            return selected.get ();
         }
      }
   }

   public static void main (String [] args) {
      mainArgs = args;
      ConsoleLauncher.main ("-c", DynamicCucumberTest.class.getName ());
   }
}

It converts each scenario into a dynamic junit5 test, which is then executed inside the normal QuarkusTest scope. Steps get injected by CDI, so they need to be properly annotated e.g. with @ApplicationScoped. Additionally if run as main class, it behaves as if it would be a cucumber CLI, this allows e.g. IntelliJ to use this class as a runner for cucumber features, so you can run and debug your tests from the IDE. Surefire treats it as an ordinary junit test and also works.

I'm sure its full of bugs, but it works for me :tm:

[j-martinez-dev]

@tisoft Thank your for sharing your idea

I tested in my project and i see that the test are running. But in my case I need to use @InjectSpy and @InjectMock from quarkus, do you have any idea to add this to your example?

Regards,

[tisoft]

Yes. They need to be added into the DynamicCucumberTest class. Here is an excerpt of my real class:

@QuarkusTest
@QuarkusTestResource (MssqlTestResource.class)
@TestProfile (DynamicCucumberTest.Profile.class)
@TestInstance (TestInstance.Lifecycle.PER_CLASS)
class DynamicCucumberTest {

   @InjectMock
   @RestClient
   @Inject
   MirthClient mirthClient;

   @BeforeAll
   public void setup () {
      // Can't use @InjectMock, since we need a sensible default return
      TimeKeeper timeKeeper = mock (TimeKeeper.class);
      when (timeKeeper.getClock ()).thenReturn (Clock.systemDefaultZone ());

      QuarkusMock.installMockForType (timeKeeper, TimeKeeper.class);
   }

And inside your Step classes you can then mock/verify, but since the mocks in the step classes are CDI injected, you need to fiddle the real object out of the wrapper:

   @SuppressWarnings ("unchecked")
   public static <E> E getMock (E m) {
      return (E) (((ClientProxy) (m)).arc_contextualInstance ());
   }

   @When ("the time advances to {string}")
   public void theTimeAdvancesTo (String arg0) {
      Mockito.reset (getMock (timeKeeper));
      Mockito.when (getMock (timeKeeper).getClock ())
             .thenReturn (Clock.fixed (HL7MessageUtils.fromHl7 (arg0, ZoneId.systemDefault ()).toInstant (),
                                       ZoneId.systemDefault ()));
   }

As you can see, you can use test profiles, test resources, @InjectMocks, or manual mocks. @InjectSpy should also work. Of course these are the same for all scenarios/features. If you need different setups per feature/scenario, you would need to have a DynamicCucumberTest runner class for each different setup and then filter the scenarios/feature files inside each class. Maybe you could even reuse the @CucumberOptions annotation and get the values from that. Luckily I don't need that so I didn't implement it :dancers:

[j-martinez-dev]

Sorry for the delay, i tested your solution and it works :)

Its funny that i need it for the same reason that your example (mock the time)

I think that is a reasonable workaround, I hope that someone in the quarkus team (and the cucumber team) have the time to see your code and give us some advice.

[stuartwdouglas]

My issue is that I don't really know anything about Cucumber, so even though I could probably get it working I don't really know how it is supposed to be used or what the expectation is.

If you can provide an example project that you would expect to work but doesn't because of @QuarkusTest then I can look at how to actually integrate it into our testing framework.

[j-martinez-dev]

HI @stuartwdouglas , I will work this weekend to give you a example project. :) Thank for your answer

[j-martinez-dev]

Hi @stuartwdouglas , i created a project to see if you can help us with this problem:

https://github.com/panchitoboy/quarkus-cucumber

[drubio-tlnd]

Hi, I'm using the workaround provided by @tisoft but I'm having an issue when trying to have a @ConfigProperty inside a step class. Would it be related to classloader issue (between QuarkusTest and JUnit DynamicTest) discussed in https://github.com/quarkusio/quarkus/issues/10623 ? @stuartwdouglas Also, with this workaround, I was able to add a tag policy (like not @foo but it seems that it's nos applied (meaning @foo scenarios are executed). Any idea?

Thanks!

[stuartwdouglas]

To fix the ClassLoader issue in the workaround add this to set the context ClassLoader: https://github.com/panchitoboy/quarkus-cucumber/compare/quarkus...stuartwdouglas:proposal#diff-d075952340d927de0ca3077a67f5c52ee81ea7a0820b6d380fa0d90b99c6e2f5R120

[stuartwdouglas]

So looking into Cucumber the existing support is based on using the vintage test engine, which we don't support and have no plans to, so @Cucumber is never going to work.

I propose we create a quarkiverse-cucumber that has the current workaround as an abstract CucumberTest class: https://github.com/panchitoboy/quarkus-cucumber/compare/quarkus...stuartwdouglas:proposal

To use this you would then simply add a test class to your project that extends this class, and cucumber will work (at the moment you also need a CDI scope on the test class, but we can fix that in Quarkus itself).

Does this sound like a reasonable approach?

[drubio-tlnd]

This does the trick for the ClassLoader issue 👍 As for the tag issue, i'll keep looking into it and post the solution if i find one.

Thanks for your help

Update: I've finally been able to activate Cucumber tags by doing this: 1- add Predicate<Pickle> filter = new Filters(runtimeOptions); 2- replace f.getPickles().forEach(...) by f.getPickles().stream().filter(filters).forEach(...)

[antoniomacri]

I'm also interested in this.

I noticed that after the CucumberTest no other tests are run, because of this line:

        features.add(DynamicTest.dynamicTest("Finish Cucumber", context::finishTestRun));

If I remove it, along with the corresponding

        features.add(DynamicTest.dynamicTest("Start Cucumber", context::startTestRun));

it continues with the other tests.

[TimeActual]

It seems the QuarkusTest context closes before the HTML & JSON reports are written completely. I have added below code: runtimeOptionsBuilder.addPluginName("html:target/cucumber-reports.html"); Generated report has incomplete body.

Is there a way to fix this?

[artysidorenko]

After looking through the startup steps used by the QuarkusTestExtension, I put together a simplified version that manages to start the application through the CdiObjectFactory. It requires junit-vintage-engine but aside from that re-uses most things from quarkus.

Here's an example forking from that quarkus-cucumber test repo, if it helps:

https://github.com/panchitoboy/quarkus-cucumber/commit/d7e0a1d84c8770098e57897d1fc62b7d87d8c398

Probably there's lot of things missing/incomplete, but it's running the tests ok so far (let me know if you see any major issues with it).

[ghost]

Hi, any progress on this? We would be interested in an extension for Cucumber in Quarkus.

[j-martinez-dev]

HI, at this moment i think the more "clean way" to use cucumber in quarkus is here:

https://github.com/panchitoboy/quarkus-cucumber/tree/quarkus-target (thanks @artysidorenko for clean the branch).

I dont know if in Quarkus 2.0 (released today) there would be an easy way to start quarkus for testing without @QuarkusTest . In that case it would be easy create an integration using the cucumber Object factory.

[grandmaximum]

@artysidorenko I tried integrating your solution into my project using gradle but struggles with getting it to work due to the following error: "Caused by: io.quarkus.bootstrap.BootstrapException: Failed to determine the Maven artifact associated with the application C:/XXXXX"

Basically QuarkusBootstrap trying to set "mavenArtifactResolver" which cannot be fetched in my environment.

I traced the error to BootstrapMavenContext where: public LocalProject getCurrentProject() { return currentProject; }

returns null.

Is your CdiObjectFactory strictly developed for Maven or is there something that I'm just not getting?

[artysidorenko]

Is your CdiObjectFactory strictly developed for Maven or is there something that I'm just not getting?

@grandmaximum the startup logic is a stripped down version of the regular QuarkusBootstrap workflow (these docs and this class were my inspiration). I'm on maven, but in any case looks like maven is the "default" one that Quarkus support; they say that gradle support in general is still incomplete/beta (see here and here)

That being said.... I notice there's a gradle resolver available in their bootstrap codebase. There might be a way to swap out the default maven resolver for the gradle resolver and get it to work with some extra configuration?

If you have a minimal reproducible repo available I can try to take a look

[stuartwdouglas]

So there are a few issues here.

• We don't support junit-vintage-engine, so it won't really be possible to use the existing Cucumber junit stuff, at least not in the same package as @QuarkusTest
• If you can't re-use @QuarkusTest then either Cucumber will need to bootstrap Quarkus or a quarkus extension will need to bootstrap cucumber, either approach will likely result in the use of internal API's subject to change.

I think the best approach would be to try and contribute a proper JUnit 5 module to cucumber, and then it should be possible to get them to play nicely with each other.

[mpkorstanje]

I think the best approach would be to try and contribute a proper JUnit 5 module to cucumber, and then it should be possible to get them to play nicely with each other.

It is probably important to understand that Junit 5 isn't a monolith. Rather:

JUnit 5 = JUnit Platform + JUnit Jupiter + JUnit Vintage.

Cucumber currently supports JUnit 5 with the cucumber-junit-platform-engine. Quarkus also supports JUnit 5 but as a JUnit Jupiter extension. While both JUnit 5, these are different things. JUnit Jupiter is a test engine that executes class based tests, Cucumber has a test engine that executes tests based on feature files written in Gherkin. Conceptually trying to run Cucumber as a JUnit Jupiter extension would be the wrong solution.

So Cucumber will have to bootstrap Quarkus. As shown by panchitoboy https://github.com/quarkusio/quarkus/issues/11045#issuecomment-825185972 Cucumber has a public API that allows this to be done. And Qaurkus could provide this implementation. This is less then ideal as it makes Quarkus dependent on Cucumber.

So I think the ideal outcome would be to reduce the coupling between Quarkus and JUnit Jupiter a bit by providing a stable public api. This could be something like Springs TestContextManager. It is/was used by Spring to integrate with JUnit4, JUnit5 and TestNG. Cucumber uses this API in the cucumber-spring module. This could also be a more generic API to bootstrap Qaurkus.

Either way as long as Cucumber, or any one else, can bootstrap Qaurkus then both projects can stick to their core business.

[khush2704]

Hello @tisoft , I am trying something similar but with the Springboot Application. It is possible to integrate Kogito Springboot with Cucumber framework for acceptance testing. Any thoughts or any comment ? Thanks in advance.

[stuartwdouglas]

This should be a Quarkiverse extension, I have some ideas around how to do it but I just need to find the time. @gastaldi can you create a quarkiverse repo and give me access?

[gastaldi]

@stuartwdouglas of course: https://github.com/quarkiverse/quarkus-cucumber

Have fun! 😉

[stuartwdouglas]

I have made a very quick start based on the example code. Basically you can do:

@QuarkusTest
public class BasicTest extends CucumberQuarkusTest {

}

And it should work. It's not released yet so you will need to build it to try it out.

[stuartwdouglas]

0.1.0 has been released, lets move all further conversation to the Quarkiverse repo.

[mpkorstanje]

@stuartwdouglas I do not believe the right discussion can be had in the quarkus-cucumber repo.

Fundamentally the problem is that Quarkus can not be bootstrapped independently from JUnit5.

And while I could point out all the problems with the current implementation of quarkus-cucumber, they all come back to that same fundamental problem.

[stuartwdouglas]

I don't really understand what you think the issues are? Jupiter support test factories, so the current implementation uses this integration to integrate with our existing Quarkus test framework.

[mpkorstanje]

The problem, fundamentally is that Quarkus couples tightly with JUnit Jupiter. As a result no other test framework can be used. As a result you have coupled tightly to Cucumber through JUnit 5.

I think the ideal outcome would be to reduce the coupling between Quarkus and JUnit Jupiter a bit by providing a stable public api. This could be something like Springs TestContextManager.

https://github.com/quarkusio/quarkus/issues/11045#issuecomment-846401054

[stuartwdouglas]

Quarkus is an opinionated framework, we have standardized on JUnit 5. Why do you even care what framework is being used when all that framework does is integrate with Cucumber and Quarkus?

[mpkorstanje]

Very well, it is your maintenance burden.