search

arquillian / arquillian-core

Arquillian provides a component model for integration tests, which includes dependency injection and container life cycle management. Instead of managing a runtime in your test, Arquillian brings your test to the runtime.
http://arquillian.org
Apache License 2.0
372 stars 195 forks source link

JUnit 5 integration spike #137

Closed bartoszmajsak closed 3 years ago

bartoszmajsak commented 6 years ago
Issue Overview

JUnit 5 Final has been around for a while and we see more demand to bring integration with Arquillian for the community.

Goal of this spike is to explore what needs to be implemented and plan following tasks based on this.

There has been a discussion on the forum and a PoC started by Payara folks.

MatousJobanek commented 6 years ago

Based on my very short investigation, we will probably need to create our own TestEngine. The PoC you have referenced is using an approach of the extensions (callbacks) when you don't have the context of the whole test suite. If I understand it correctly, this is something similar to the current @Rules and @ClassRules In addition, with the ability of the Launcher we should be able to finally bring the ability of sub-suite deployments. For more information take a look at the Advanced Topics paragraph.

bartoszmajsak commented 6 years ago

Thanks for checking it out. What are the next steps then?

MatousJobanek commented 6 years ago

Definitely, spend more time with the JUnit 5 extensions - if there is some way of getting the whole test-suite context - eg. from the storage that is provided by JUnit. If not, then start moving the old runner logic to a new ArquillianTestEngine. One of the first steps would be also taking a look at this PR: https://github.com/arquillian/arquillian-core/pull/107 as it contains logic that would allow using Arquillian as a JUnit @Rule. This work could be reused for new Arquillian JUnit 5 extensions - for cases when a user cannot or doesn't want to use the engine. Both implementations the TestEngline and the extension approach should be done as one task as it will require complex refactoring and having in mind both approaches - if we agree on that implementing both. When the current bits are refactored to support JUnit 5, then we can start working on the sub-suite deployment feature or something more. Summing this up:

  1. finish the research a) investigate extensions & test engines more deeply b) take a look at already existing implementations c) take a look at PR #107 d) prepare a proposal of the final design
  2. agree on the concept/design we will want to implement (only one or both approaches)
  3. Implementation a) create a new module (if it is possible to keep it in one project) b) implement the support of JUni5 c) execute all current Arquillian extension and container adapter test suites using JUnit5
  4. plan next steps a) sub-suite deployment b) ?? anything else ??
dipak-pawar commented 6 years ago

Looks like as per Matous's above comment we don't have suite context if we decided to go through junit 5 extension. https://github.com/junit-team/junit5/issues/1145.

Now if we decided to go with implementing custom TestEngine for Arquillian. We will be having lot of duplications from junit-jupiter-engine due to final & private variable.

Also I found this interesting comment - https://github.com/junit-team/junit5/issues/20#issuecomment-312210399 about discovering all tests twice if you tries to use jupiter-engine as composition in your engine to avoid duplications

Now digging into test extension, to see any other possible ways.

MatousJobanek commented 6 years ago

So, I've tried to create the integration using a combination of a new TestEngine implementation with JUnit 5 extension implementation. The engine is necessary for having the context of the whole test suite and for starting and stopping of the containers. The extension is for managing the lifecycle on the level of test classes and test methods.

During the implementation I found a few obstacles: The first problem was that the JUnit 5 surefire provider calls the engine for every test class separately. This has been fixed in the latest version r5.1.0-M1

So, we need to listen on the very beginning and then delegate the test execution to the standard JupiterTestEngine. This is pretty easy, the problematic part is providing an instance of the TestRunnerAdaptor to the extension implementation. TestRunnerAdaptor is used in Arquillian for firing events, managing lifecycle, keeping the scopes instances etc. In other words, without the adaptor (or any other similar class) we cannot share the context between the test engine and the extension, and thus, we are not able to do the Arquillian magic.

Technically speaking, that could be managed by overriding some Jupiter's internal classes and using delegation. It isn't easy to do, it would require several hacks, code duplications, and reflections, but is probably doable.

The last obstacle is that we need the junit-jupiter-engine dependency for the delegation of the test execution. But in the platform implementation, there are retrieved all TestEngine implementations that are on classpath. In our case it would mean that all classes would be executed twice - with our arquillian engine and with jupiter engine. There is probably a way to workaround that using shaded jar and relocation with breaking inner jupiter's SPI, but this is nothing that I would like to distribute.

Summing it up. We can provide Arquillian functionality in the context of JUnit 5 extension, but not for the context of the whole test suite execution. There is already a JUnit 5 issue that covers this missing functionality/SPI - it's that one that was referenced by @dipak-pawar : https://github.com/junit-team/junit5/issues/1145

MatousJobanek commented 6 years ago

As the Arquilian engine for JUnit 5 is not feasible for now, I've been trying to create an extension that would do similar work as JUnit 4 Arquillian Rules do.

JUnit 5 versions I'm been working with JUnit 5 5.1.0-M1 and Platform Launcher 1.1.0-M1

Code The resulting code can be found here: https://github.com/MatousJobanek/arquillian-core/tree/junit5-extension The specific JUnit 5 module is here: https://github.com/MatousJobanek/arquillian-core/tree/junit5-extension/junit5

Dependencies: standalone: org.jboss.arquillian.junit5:arquillian-junit5-standalone container: org.jboss.arquillian.junit5:arquillian-junit5-container

Extension class The JUnit 5 extension class: ArquillianExtension

Implementation Standalone The extension works in the standalone mode without any problem and as expected.

Container Container startup/shutdown, as well as a deployment, is managed perfectly and without any obstacles. The problem comes with the in-container testing (client tests are OK). If any test should be executed inside of the container, then (as JUnit 5 doesn't provide any way to use my own test method executer) I need to use a workaround using nasty hacks and Java reflection. Using ExecutionCondition callback, I need to skip the in-container test methods (to prevent executing them on client) and then manually invoke them inside of the container. This works fine as well, but the main problem is with the results - as I had to skip them, then the JUnit 5 listeners marked those test methods as skipped. I'm able to partially change the restult, but some reports (surefire, IDE) can already contain the skipped one, so the reported number of the executed tests can be different than the actuall ones.

OndroMih commented 6 years ago

I've tried the approach of creating a JUnit5 extensions almost a year ago when I started this discussion (I'm the Payara folk mentioned above). I came to a similar conclusion as @MatousJobanek - it's not possible to intercept test execution so that it's routed to the test executed in the container. The only thing I could do is to run the test in the Arquillian container before it was executed by Junit5 outside the container. But I couldn't prevent JUnit5 running the tests outside the container just with a JUnit5 extension.

Therefore my conclusion is the same - we either need to build a JUnit5 test runner, possibly duplicating lots of code from the Jupiter engine, or wait until JUnit5 implements https://github.com/junit-team/junit5/issues/1145 (which may take too long according to the comments)

My source code is here: OndrejM/arquillian-junit5-hacks, with the extendion class ArquillianExt.java. I didn't get as far as @MatousJobanek and the code is much simpler, but it demonstrates the same problem when you try running the BasicJunit5ArquillianTest - the test runs succesfully in the container but then it fails when running out of container because a managed resource isn't available (the test uses embedded Payara Server so it runs out of the box without external container running).

MatousJobanek commented 6 years ago

Hi @OndrejM, I saw your work. I was curious if there had been changed something in the JUnit 5 from the time you were trying it which would make the implementation feasible. There have been changed and fixed a lot of things, unfortunately, a possibility to provide a custom test method runner is still missing. I've created an issue for it: https://github.com/junit-team/junit5/issues/1248 My extension is working and nothing is failing, the only thing that is weird are the test results when the in-container tests are used. If you want to use my implementation, feel free to take it ;-). If you would like to have it in the main repo (not my forked one) I can push it into a separated branch.

keilw commented 6 years ago

@OndrejM @MatousJobanek Has there been any progress or is Arquillian still not working properly with JUnit5?

bartoszmajsak commented 6 years ago

@keilw if you follow this thread and linked issues you will see that we are missing few pieces on JUnit 5 side to make this integration happen.

keilw commented 6 years ago

@bartoszmajsak Thanks for the update. I tried to add the "Vintage" module which looks like it works fine for now. If JUnit 5 can solve this, we would be happy to use those new versions without the "Vintage Look" ;-)

bartoszmajsak commented 6 years ago

@keilw we would love to make it happen. hopefully we can follow up on that any time soon. Another approach could be to use JUnit rules instead of runner we have introduced late last year http://arquillian.org/blog/2017/12/20/arquillian-core-1-2-0-Final/

davidwittesz commented 6 years ago

I had have a look at https://github.com/junit-team/junit5/issues/1248 seems that there is no progress up to now?

I Would be very happy about this feature.

Any guesses when it would be possible to use JU5 with arquillian?

cardil commented 5 years ago

Arquillian's implicit behavior of running tests on remote container and then silently tunneling them to client JUnit was always hard to understand for developers. Maybe it is good opportunity to switch testing to more explicit solution.

I'm thinking of something like that:

@Test
void assertAtContainer(Arquillian arq) {
  // when
  int result = arg.runAtContainer(new ArquillianExecution<Integer>() {
    @Inject
    @Transactional
    Integer execute(MyDao dao) {
      return dao.countActiveUsers();
    }
  });

  // then
  assertThat(result).isGreatherThen(0);
}
lordofthejars commented 5 years ago

@bartoszmajsak this would be also really great to have support.

jameschensmith commented 5 years ago

@bartoszmajsak, has there been any additional push for this? I'd love to be able to use JUnit-5 with Arquillian. What's the current status of this?

bartoszmajsak commented 5 years ago

@james-r-smith nothing besides the PoC @MatousJobanek worked on back in the day. Two crucial feature requests in JUnit 5 are still not implemented:

You would probably be fine with JUnit5 Vintage Engine but I guess that's not what you are looking for.

poikilotherm commented 5 years ago

@bartoszmajsak You might have noticed that junit-team/junit5#157 has been implemented... :smile:

blabno commented 5 years ago

@bartoszmajsak Hi! Any idea if junit5 will be supported in near future?

gastaldi commented 5 years ago

Is there anyone working on this feature? It would be really cool to adopt JUnit 5 in Arquillian projects

bartoszmajsak commented 5 years ago

Is there anyone working on this feature? It would be really cool to adopt JUnit 5 in Arquillian projects

@gastaldi no one that I'm aware of. It's up for grabs.

zforgo commented 5 years ago

Since Junit Jupiter 5.5.x has released intercepting test methods are available. I made an extension based on that. https://github.com/zforgo/arquillian-junit5-extension

tandraschko commented 5 years ago

Nice! It would be great if it could make it into arquillian directly.

dmatej commented 5 years ago

@zforgo I got only one NPE (I will create PR soon), but generally it seems it really works! You are a hero! :-)

EDIT: one more issue - tests were always successful; see PR. (and in my master is added logging)

EDIT: and in-container injections do not work because interceptors are out of game there. Maybe it would help to separate interceptors to "inner" and "outer", but this would need a change in Arquillian.

But for remote testing it works.

arjantijms commented 4 years ago

Hi, just wondering. Any progress here?

OndroMih commented 4 years ago

I'm pretty sure the current solution is stable enough to become part of the Arquillian project. I'm not an Arquillian committer but I can submit this to the Arquillian project if nobody plans to do so. Another option would be to submit this as a separate project to the Eclipse foundation as an Arquillian extension.

I've rewritten the Jakarta Batch TCK to use Arquillian and JUnit 5, I wrote about it here: https://ondro.inginea.eu/index.php/possible-ways-to-use-arquillian-in-jakarta-ee-tcks/. I had no problems running the rewritten test suite against Payara Server with all tests passed. Therefore I believe that the current solution is stable enough. It would be good if some other people reported their success stories with this solution or tried to rewrite their current Arquillian tests to JUnit 5 and report how it went.

lprimak commented 4 years ago

I would second that opinion. Something needs to be done with JUnit 5 and since there is no other solution, this would be a great addition

bartoszmajsak commented 4 years ago

Hey @OndroMih, thanks for sharing your experience with JUnit 5 and Arquillian. I'm all for moving this forward so apologies there was not much traction on this issue - I have other things which are consuming both my professional and personal time.

That said I'm really happy to see people in the community involved and interested. I believe that should be part of arquillian-core as it is with junit4 and TestNG. Is there anything we should add/extend or porting @zforgo work is enough?

dmatej commented 4 years ago

@bartoszmajsak There are still unmerged my two pull requests, but generally it works well. https://github.com/zforgo/arquillian-junit5-extension/pulls

arjantijms commented 4 years ago

Hope this can be moved forward. As I manage several projects that use junit and arquillian intensively, I'll try to test out something soon on these projects to see where we stand.

OndroMih commented 4 years ago

Hi @bartoszmajsak, I'm pretty sure it's safe to start by porting @zforgo 's work. Then @dmatej can raise his pull requests against the ported code in arquillian-core to fix some known issues.

bartoszmajsak commented 4 years ago

I integrated @zforgo work as a module in arquillian-core. Again thanks for this milestone!

You can find all the adjustments on the junit5_extension branch, including two PRs from @dmatej. If anyone can try it out against their code it would be just great.

The original code has no tests, so I don't feel comfortable with merging it yet. There are, however, some external integration tests which seems to be working fine with a few tweaks. Any contributions here would be more than awesome, but I will try to find some time this week to do a bit of test-after-development :)

bartoszmajsak commented 4 years ago

@OndroMih @arjantijms do you have any larger projects you could give it a spin? If needed I can push snapshot artifact so that it's easier for you to consume.

Thihup commented 4 years ago

I've got a try using the javaee-samples/jakartaee8-tck project and it seems to be working fine. https://github.com/Thihup/jakartaee8-tck/commit/a503d3a30b232d0cfc34860439bf96378ed5efbf

bartoszmajsak commented 4 years ago

Thanks for your feedback @Thihup!

lprimak commented 4 years ago

Definitely please put out a snapshot

lprimak commented 4 years ago

I can't get it to work... every test throws an exception:

java.lang.ClassCastException: org.junit.jupiter.engine.descriptor.JupiterEngineDescriptor cannot be cast to org.junit.jupiter.engine.descriptor.MethodBasedTestDescriptor

Any ideas?

lprimak commented 4 years ago

More context:

java.lang.ClassCastException: org.junit.jupiter.engine.descriptor.JupiterEngineDescriptor cannot be cast to org.junit.jupiter.engine.descriptor.MethodBasedTestDescriptor
    at org.jboss.arquillian.junit5.container.JUnitJupiterTestRunner.lambda$execute$0(JUnitJupiterTestRunner.java:41)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.lambda$applyPostDiscoveryFilters$3(EngineDiscoveryOrchestrator.java:122)
    at org.junit.platform.engine.TestDescriptor.accept(TestDescriptor.java:249)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.lambda$acceptInAllTestEngines$8(EngineDiscoveryOrchestrator.java:167)
    at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.acceptInAllTestEngines(EngineDiscoveryOrchestrator.java:167)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.applyPostDiscoveryFilters(EngineDiscoveryOrchestrator.java:128)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:92)
    at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:92)
    at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:67)
    at org.jboss.arquillian.junit5.container.JUnitJupiterTestRunner.execute(JUnitJupiterTestRunner.java:49)
    at org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.executeTest(ServletTestRunner.java:139)
    at org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.execute(ServletTestRunner.java:117)
    at org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.doGet(ServletTestRunner.java:86)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1636)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:259)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
    at java.lang.Thread.run(Thread.java:748)
lprimak commented 4 years ago

I think it's a bug, when I check instanceof for the failure, my tests pass

lprimak commented 4 years ago

The preceding PR #285 fixes the issue and gets my tests to pass

lprimak commented 4 years ago

@bartoszmajsak Please check & merge

lprimak commented 3 years ago

Not sure if the snapshot is available, but I have put the latest branch version up on Payara's Nexus:

Arquillian artifacts:

        <dependency>
            <groupId>org.jboss.arquillian.junit5</groupId>
            <artifactId>arquillian-junit5-container</artifactId>
            <version>1.7.0.Alpha5</version>
            <scope>test</scope>
        </dependency>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.jboss.arquillian</groupId>
                <artifactId>arquillian-bom</artifactId>
                <version>1.7.0.Alpha5</version>
                <type>pom</type>
                <scope>test</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

   <repositories>
        <!-- Once Arq 1.7.0 or later is released with JUnit 5 support, this can be removed -->
        <repository>
            <id>payara</id>
            <url>https://nexus.payara.fish/repository/payara-artifacts</url>
        </repository>
    </repositories>

Payara connector artifacts (example):

        <dependency>
            <groupId>fish.payara.arquillian</groupId>
            <artifactId>arquillian-payara-server-remote</artifactId>
            <version>2.3.1-arq-1.7.0.Alpha5</version>
            <scope>test</scope>
        </dependency>

Payara artifacts can be browsed here: https://nexus.payara.fish/#browse/browse:payara-artifacts under fish/payara/arquillian

Sample project can be found here:

https://github.com/flowlogix/flowlogix/tree/master/jakarta-ee/jee-examples

bartoszmajsak commented 3 years ago

Thanks @lprimak. I'm actually about to cut the next alpha release today with all the changes (including yours) :)

bartoszmajsak commented 3 years ago

1.7.0.Alpha5 is out with initial support for JUnit 5. Feel free to open an issue if you find anything and I'll be happy to address them. I'm closing this long thread.

Thank you all for your help! Much appreciated!