Release-only tests

[eparejatobes]

We frequently have the need for tests that create resources, perform expensive (time and money) operations etc. It would good to be able to mark those tests as "release-only", making sure that they will be run when (and only when) you are making a release.

I'm guessing this could be done using scalatest tags

• Scalatest docs -- tagging your tests

and redefining sbt test so that normally it would exclude "release-only" tests. Found this about it:

• Hootsuite blog -- How to Run Tagged Scala Tests with SBT and ScalaTest

[laughedelic]

Related in sbt-0.13.9:

Supports excluding tests in testOnly/testQuick with -, for example -MySpec.

[laughedelic]

By the way, the list of fixes in 0.13.9 looks good: https://github.com/sbt/sbt/releases/tag/v0.13.9

[laughedelic]

• http://www.edgezero.de/2015/03/partitioning-your-test-code-with-scalatest-and-sbt/

[eparejatobes]

We need this ASAP. A simple approach:

Tag for release-only tests

As in

package ohnosequences.tests
import org.scalatest.Tag
// NOTE use the String inside the Tag constructor later
case object releaseOnly extends Tag("ohnosequences.tests.tags.releaseOnly")

This is used like so

import ohnosequences.tests._

class FunnyTests extends FunSuite {

  test("launch 100 machines, do stuff", releaseOnly) {
    // ...
  }
}

Ignore that tag by default

In build.sbt or as part of this plugin (much better)

testOptions in Test += Tests.Argument("-l", "ohnosequences.tests.tags.releaseOnly")

Run them in the release task

Through some sbt magic that I don't quite understand. I guess it is possible to change the value of a setting temporarily right?

References

• http://www.scalatest.org/user_guide/using_the_runner#filtering
• http://www.scalatest.org/user_guide/using_scalatest_with_sbt
• https://github.com/sbt/sbt-release/blob/master/src/main/scala/ReleaseExtra.scala#L87-L96

[laughedelic]

Good! Yes, it's doable 👍 I can spend some time on it this weekend

[laughedelic]

This is finally done:

• added releaseOnlyTestTagPackage and releaseOnlyTestTag settings for the tag's name (with some default values)
• test:test runs everything except the test tagged with that tag
• test:testAll runs everything (used in the release process)

Not the best solution, because testAll is defined through testOnly and this tag is excluded only from the test task. The other solution I have is through an sbt "configuration" scoping, but I didn't find how to add custom configs from a plugin.

[laughedelic]

I found a way to add configs with plugins. But I will change here all things to AutoPlugins first.