Running cucumber specs as part of the sbt test task

[masterzen]

Hi,

Your plugin is very good, and works very fine when running "cucumber" from the sbt command prompt.

This might be my global misunderstanding of SBT, but I can't find a way to have the cucumber specs to be run as part of the sbt test task.

Ideally, I'd like to be able to have a sub module project that when running the test task, in fact triggers a run of the cucumber tests, with the test results reported back to sbt.

Am I overlooking something? Thanks,

[skipoleschris]

Hi, Thanks for the feedback. You are correct that the plugin is currently implemented as a standalone task. My personal experience is that cucumber specs tend to take time to execute. On my projects I find I like to have the unit tests/scala check run very fast using the test target and then run the cucumber specs separately before checking and as part of continuous integration.

However, I can understand the benefit of having the plugin integrated as part of the test task. I'm not familiar with how that works within SBT, but I'm happy to take a look and see what I can work out. I'll update this issue as soon as I figure out whether it is possible and how to do it.

Regards, Chris

[masterzen]

Thanks for the feedback Chris. I also had a look, and started creating a patch. For the moment it is non-functional, but I have some hope for today :) If I have something that works, I'll submit a pull request.

You are right that the tests shouldn't be run as the test phase without a manual configuration of the user. I was thinking about "Additional Test Configurations" as explained in: http://www.scala-sbt.org/release/docs/Detailed-Topics/Testing.html

[skipoleschris]

Hi. I've also started taking a look at implementing the sbt test interface. Looks doable, but I'm still experimenting with how to get all the sbt settings passed through successfully. I'm also working on how to run cucumber without forking as this is the recommended approach for running as a test framework. Will also look at implementing the test listener interface as well.

Would be great to see your patch when you have something done as I can then pool your ideas with my own in order to build the best solution.

Thanks.

[masterzen]

Right now, I have the task running but it doesn't pull-in the cucumber test dependencies :( If I have something ready, I'll send you a pointer to the code.

[skipoleschris]

Right, I've been looking into this in some detail and it's not easy, but is certainly doable. I think it's a good idea, so I'll carry on working on it.

Basically, the challenge is that the cucumber plugin is build for a specific SBT version (and thus Scala version) and is loaded by the SBT classloader. However test frameworks (that implement the test interface) are loaded onto the project test classpath based on the project Scala version. These two classpaths are not shared.

What I am therefore going to have to do is have two projects:

• xsbt-cucumber-plugin which implements the plugin logic, configuration and the cucumber goal and is built for specific SBT versions
• xsbt-cucumber-test which implements the test interface and is cross-built for all the different Scala versions in the same way as a normal dependency

I will probably also need to create a common project that holds the abstractions for actually running Cucumber JVM.

What will probably then happen is that I'll add a config setting to the plugin which when enabled will add the test interface to the test class path and enable the integration into the test target and the reporting.

A bit convoluted, but I think it should all come together just fine. Propbably take me a few days though!

[skipoleschris]

Okay, I've uploaded an experimental branch for a 0.7.0 release. It's called test-interface. It just about works except that it seems to lock up after the tests run. It's also missing a few polishing touches.

I've updated the README to describe how it works and how to integrate it. The exact list of things that don't yet work are also on this page. Hope to have these fixed in the next few days. Any early feedback would be appreciated.

[masterzen]

@skipoleschris I will make sure to review and test it tomorrow and report you any success or failure.

[masterzen]

@skipoleschris, so I tried both ways of running the tests. Both ran the features with success.

A few of things to note:

• in Integration mode, after the features ran, sbt started running the other tests apparently, but nothing was appearing on the console (as if the cucumber plugin was still buffering the input/output).
• in regular mode, when an exception occurs, it can happen the cucumber plugin is stuck and no sbt prompt appears (I already got this issue with the previous version).
• the plugin namespace changed, so the addSbtPlugin in the README should be addSbtPlugin("templemore" % "sbt-cucumber-plugin" % "0.7.0")

Let me know if you want me to test/troubleshoot anything.

[skipoleschris]

OK, I've released version 0.7.0, which should be in the repository now. It should allow the test task to run cucumber tests and works against Scala 2.9.2 and 2.10.0-RC1/2.

There might still be a few rough edges and I still can't figure out how to stop multi-module projects from running in parallel and interleaving their output. However, I didn't think this was enough of a blocker to stop the release and I can always fix in a minor version if a solution comes along.

Please raise any further problems as new issues so that I can attack them in another release.