search

JetBrains / sbt-tc-logger

SBT plugin generates TeamCity service messages to be used by SBT runner
Apache License 2.0
10 stars 12 forks source link

TC's "sbt test" build step succeed even though it throws an exception during initialization #12

Open jsancio opened 7 years ago

jsancio commented 7 years ago

We have that TC Version 2017.1.2 (build 46812) succeeds the CI even though it was unable to initialize and run all of the test. We created a minimal sbt project that reproduces this behavior. The full project is at https://github.com/jsancio/teamcity-test. Requirements:

  1. Include sbt-tc-logger version 0.4.0.
  2. Force the test initialization to fail by throwing an exception during object initialization.

When you run the test using the normal sbt behavior this is what we get:

$ sbt test
[info] Loading project definition from /home/jose/work/teamcity-test/project
[info] Set current project to Hello (in build file:/home/jose/work/teamcity-test/)
[info] HelloSpec:
[info] The Hello object
[info] example.HelloSpec *** ABORTED ***
[info]   java.lang.ExceptionInInitializerError:
[info]   at example.HelloSpec.$anonfun$new$1(HelloSpec.scala:7)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info]   at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685)
[info]   ...
[info]   Cause: java.lang.IllegalArgumentException: Test TC sbt logger plugin
[info]   at example.Hello$.trickCompiler(Hello.scala:10)
[info]   at example.Hello$.<init>(Hello.scala:6)
[info]   at example.Hello$.<clinit>(Hello.scala)
[info]   at example.HelloSpec.$anonfun$new$1(HelloSpec.scala:7)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
[info]   ...
[error] Could not run test example.HelloSpec: java.lang.ExceptionInInitializerError
[info] Run completed in 292 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 1
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] *** 1 SUITE ABORTED ***
[error] Error during tests:
[error]     example.HelloSpec
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1 s, completed Jul 9, 2017 4:40:08 PM

$ echo $?
1

Trying to run this test with the TC logger enable results in:

$ TEAMCITY_VERSION=ignore sbt test
[info] Loading project definition from /home/jose/work/teamcity-test/project
[info] Set current project to Hello (in build file:/home/jose/work/teamcity-test/)
##teamcity[compilationStarted compiler='Scala compiler in Test' flowId='-946421287']
##teamcity[compilationStarted compiler='Scala compiler' flowId='-946421287']
##teamcity[compilationFinished compiler='Scala compiler' flowId='-946421287']
##teamcity[compilationFinished compiler='Scala compiler in Test' flowId='-946421287']
##teamcity[testSuiteStarted name='example.HelloSpec' flowId='27']
[info] HelloSpec:
[info] The Hello object
[info] example.HelloSpec *** ABORTED ***
[info]   java.lang.ExceptionInInitializerError:
[info]   at example.HelloSpec.$anonfun$new$1(HelloSpec.scala:7)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info]   at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685)
[info]   ...
[info]   Cause: java.lang.IllegalArgumentException: Test TC sbt logger plugin
[info]   at example.Hello$.trickCompiler(Hello.scala:10)
[info]   at example.Hello$.<init>(Hello.scala:6)
[info]   at example.Hello$.<clinit>(Hello.scala)
[info]   at example.HelloSpec.$anonfun$new$1(HelloSpec.scala:7)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
[info]   ...
[error] Could not run test example.HelloSpec: java.lang.ExceptionInInitializerError
##teamcity[message status='ERROR' flowId='-946421287' text='Could not run test example.HelloSpec: java.lang.ExceptionInInitializerError']
##teamcity[testSuiteFinished name='example.HelloSpec' message='null' details='|[Ljava.lang.StackTraceElement;@2511a2c3' flowId='27']
[success] Total time: 1 s, completed Jul 9, 2017 4:43:28 PM

$ echo $?
0

Even thought the logger is printing an error message in:

##teamcity[message status='ERROR' flowId='-946421287' text='Could not run test example.HelloSpec: java.lang.ExceptionInInitializerError']

It results in the build step succeeding in TeamCity.

nvollmar commented 5 years ago

We've run into the same issue, when forked tests fail (with sbt.ForkMain 37823 failed with exit code 255) it get suppressed and the build succeeds anyway (resulting in not all tests being executed without notice!).