broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
988 stars 358 forks source link

Fix flakey test: RobustClientHelperSpec #4351

Open geoffjentry opened 5 years ago

geoffjentry commented 5 years ago

https://fc-jenkins.dsp-techops.broadinstitute.org/job/cromwell-test-runner/884/ https://fc-jenkins.dsp-techops.broadinstitute.org/job/cromwell-test-runner/1179/

java.lang.IllegalStateException: cannot create children while terminating or terminated at akka.actor.dungeon.Children.makeChild(Children.scala:270) at akka.actor.dungeon.Children.attachChild(Children.scala:48) at akka.actor.dungeon.Children.attachChild$(Children.scala:47) at akka.actor.ActorCell.attachChild(ActorCell.scala:431) at akka.actor.ActorSystemImpl.systemActorOf(ActorSystem.scala:745) at akka.testkit.TestKitBase.$init$(TestKit.scala:170) at akka.testkit.TestKit.(TestKit.scala:896) at akka.testkit.TestProbe.(TestKit.scala:954) at akka.testkit.TestProbe.(TestKit.scala:956) at akka.testkit.TestProbe$.apply(TestKit.scala:990) at cromwell.core.actor.RobustClientHelperSpec.$anonfun$new$7(RobustClientHelperSpec.scala:109) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682) at org.scalatest.TestSuite.withFixture(TestSuite.scala:196) at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195) at cromwell.core.actor.RobustClientHelperSpec.withFixture(RobustClientHelperSpec.scala:14) at org.scalatest.FlatSpecLike.invokeWithFixture$1(FlatSpecLike.scala:1680) at org.scalatest.FlatSpecLike.$anonfun$runTest$1(FlatSpecLike.scala:1692) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289) at org.scalatest.FlatSpecLike.runTest(FlatSpecLike.scala:1692) at org.scalatest.FlatSpecLike.runTest$(FlatSpecLike.scala:1674) at cromwell.core.actor.RobustClientHelperSpec.runTest(RobustClientHelperSpec.scala:14) at org.scalatest.FlatSpecLike.$anonfun$runTests$1(FlatSpecLike.scala:1750) at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396) at scala.collection.immutable.List.foreach(List.scala:389) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373) at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410) at scala.collection.immutable.List.foreach(List.scala:389) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461) at org.scalatest.FlatSpecLike.runTests(FlatSpecLike.scala:1750) at org.scalatest.FlatSpecLike.runTests$(FlatSpecLike.scala:1749) at cromwell.core.actor.RobustClientHelperSpec.runTests(RobustClientHelperSpec.scala:14) at org.scalatest.Suite.run(Suite.scala:1147) at org.scalatest.Suite.run$(Suite.scala:1129) at cromwell.core.TestKitSuite.org$scalatest$BeforeAndAfterAll$$super$run(TestKitSuite.scala:16) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at cromwell.core.actor.RobustClientHelperSpec.org$scalatest$FlatSpecLike$$super$run(RobustClientHelperSpec.scala:14) at org.scalatest.FlatSpecLike.$anonfun$run$1(FlatSpecLike.scala:1795) at org.scalatest.SuperEngine.runImpl(Engine.scala:521) at org.scalatest.FlatSpecLike.run(FlatSpecLike.scala:1795) at org.scalatest.FlatSpecLike.run$(FlatSpecLike.scala:1793) at cromwell.core.actor.RobustClientHelperSpec.run(RobustClientHelperSpec.scala:14) at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314) at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507) at sbt.TestRunner.runTest$1(TestFramework.scala:113) at sbt.TestRunner.run(TestFramework.scala:124) at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282) at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246) at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282) at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282) at sbt.TestFunction.apply(TestFramework.scala:294) at sbt.Tests$.processRunnable$1(Tests.scala:347) at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353) at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46) at sbt.std.Transform$$anon$4.work(System.scala:67) at sbt.Execute.$anonfun$submit$2(Execute.scala:269) at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16) at sbt.Execute.work(Execute.scala:278) at sbt.Execute.$anonfun$submit$1(Execute.scala:269) at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178) at sbt.CompletionService$$anon$2.call(CompletionService.scala:37) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

danbills commented 5 years ago

Another error w/ this test:

https://fc-jenkins.dsp-techops.broadinstitute.org/view/Testing/view/Test%20Runners/job/cromwell-test-runner/2444/testReport/junit/cromwell.core.actor/RobustClientHelperSpec/RobustClientHelper_should_reset_timeout_when_backpressured_is_received/

   java.lang.AssertionError: assertion failed: received unexpected message RealMessage(ServiceUnreachable,TestActor[akka://TestSystem-78f39f37-cc73-481d-8e7a-e59e623aa020/user/$$i]) after 0 millis
      at akka.testkit.TestKitBase.expectNoMsg_internal(TestKit.scala:696)
      at akka.testkit.TestKitBase.expectNoMessage(TestKit.scala:661)
      at akka.testkit.TestKitBase.expectNoMessage$(TestKit.scala:660)
      at akka.testkit.TestKit.expectNoMessage(TestKit.scala:896)
      at cromwell.core.actor.RobustClientHelperSpec.$anonfun$new$7(RobustClientHelperSpec.scala:140)
      at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at cromwell.core.actor.RobustClientHelperSpec.withFixture(RobustClientHelperSpec.scala:14)
      at org.scalatest.FlatSpecLike.invokeWithFixture$1(FlatSpecLike.scala:1680)
      at org.scalatest.FlatSpecLike.$anonfun$runTest$1(FlatSpecLike.scala:1692)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
      at org.scalatest.FlatSpecLike.runTest(FlatSpecLike.scala:1692)
      at org.scalatest.FlatSpecLike.runTest$(FlatSpecLike.scala:1674)
      at cromwell.core.actor.RobustClientHelperSpec.runTest(RobustClientHelperSpec.scala:14)
      at org.scalatest.FlatSpecLike.$anonfun$runTests$1(FlatSpecLike.scala:1750)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
      at scala.collection.immutable.List.foreach(List.scala:389)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410)
      at scala.collection.immutable.List.foreach(List.scala:389)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
      at org.scalatest.FlatSpecLike.runTests(FlatSpecLike.scala:1750)
      at org.scalatest.FlatSpecLike.runTests$(FlatSpecLike.scala:1749)
      at cromwell.core.actor.RobustClientHelperSpec.runTests(RobustClientHelperSpec.scala:14)
      at org.scalatest.Suite.run(Suite.scala:1147)
      at org.scalatest.Suite.run$(Suite.scala:1129)
      at cromwell.core.TestKitSuite.org$scalatest$BeforeAndAfterAll$$super$run(TestKitSuite.scala:16)
      at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
      at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
      at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
      at cromwell.core.actor.RobustClientHelperSpec.org$scalatest$FlatSpecLike$$super$run(RobustClientHelperSpec.scala:14)
      at org.scalatest.FlatSpecLike.$anonfun$run$1(FlatSpecLike.scala:1795)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
      at org.scalatest.FlatSpecLike.run(FlatSpecLike.scala:1795)
      at org.scalatest.FlatSpecLike.run$(FlatSpecLike.scala:1793)
      at cromwell.core.actor.RobustClientHelperSpec.run(RobustClientHelperSpec.scala:14)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
      at sbt.TestRunner.runTest$1(TestFramework.scala:113)
      at sbt.TestRunner.run(TestFramework.scala:124)
      at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
      at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
      at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
      at sbt.TestFunction.apply(TestFramework.scala:294)
      at sbt.Tests$.processRunnable$1(Tests.scala:347)
      at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
      at sbt.std.Transform$$anon$4.work(System.scala:67)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
      at sbt.Execute.work(Execute.scala:278)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
geoffjentry commented 5 years ago

Again 2779

java.lang.AssertionError: assertion failed: received unexpected message RealMessage(ServiceUnreachable,TestActor[akka://TestSystem-a47da50b-5587-413b-bbc6-4773a965cb41/user/$$i]) after 0 millis
      at akka.testkit.TestKitBase.expectNoMsg_internal(TestKit.scala:696)
      at akka.testkit.TestKitBase.expectNoMessage(TestKit.scala:661)
      at akka.testkit.TestKitBase.expectNoMessage$(TestKit.scala:660)
      at akka.testkit.TestKit.expectNoMessage(TestKit.scala:896)
      at cromwell.core.actor.RobustClientHelperSpec.$anonfun$new$7(RobustClientHelperSpec.scala:140)
      at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at cromwell.core.actor.RobustClientHelperSpec.withFixture(RobustClientHelperSpec.scala:14)
      at org.scalatest.FlatSpecLike.invokeWithFixture$1(FlatSpecLike.scala:1680)
      at org.scalatest.FlatSpecLike.$anonfun$runTest$1(FlatSpecLike.scala:1692)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
      at org.scalatest.FlatSpecLike.runTest(FlatSpecLike.scala:1692)
      at org.scalatest.FlatSpecLike.runTest$(FlatSpecLike.scala:1674)
      at cromwell.core.actor.RobustClientHelperSpec.runTest(RobustClientHelperSpec.scala:14)
      at org.scalatest.FlatSpecLike.$anonfun$runTests$1(FlatSpecLike.scala:1750)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
      at scala.collection.immutable.List.foreach(List.scala:389)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410)
      at scala.collection.immutable.List.foreach(List.scala:389)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
      at org.scalatest.FlatSpecLike.runTests(FlatSpecLike.scala:1750)
      at org.scalatest.FlatSpecLike.runTests$(FlatSpecLike.scala:1749)
      at cromwell.core.actor.RobustClientHelperSpec.runTests(RobustClientHelperSpec.scala:14)
      at org.scalatest.Suite.run(Suite.scala:1147)
      at org.scalatest.Suite.run$(Suite.scala:1129)
      at cromwell.core.TestKitSuite.org$scalatest$BeforeAndAfterAll$$super$run(TestKitSuite.scala:16)
      at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
      at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
      at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
      at cromwell.core.actor.RobustClientHelperSpec.org$scalatest$FlatSpecLike$$super$run(RobustClientHelperSpec.scala:14)
      at org.scalatest.FlatSpecLike.$anonfun$run$1(FlatSpecLike.scala:1795)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
      at org.scalatest.FlatSpecLike.run(FlatSpecLike.scala:1795)
      at org.scalatest.FlatSpecLike.run$(FlatSpecLike.scala:1793)
      at cromwell.core.actor.RobustClientHelperSpec.run(RobustClientHelperSpec.scala:14)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
      at sbt.TestRunner.runTest$1(TestFramework.scala:113)
      at sbt.TestRunner.run(TestFramework.scala:124)
      at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
      at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
      at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
      at sbt.TestFunction.apply(TestFramework.scala:294)
      at sbt.Tests$.processRunnable$1(Tests.scala:347)
      at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:353)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
      at sbt.std.Transform$$anon$4.work(System.scala:67)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
      at sbt.Execute.work(Execute.scala:278)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
geoffjentry commented 5 years ago
  1. Not on VPN so can't get stack trace but there are already a few here