Question on running unit tests with `CassandraLauncher`

[lukewyman]

I've been using your akka-customer sample project as one of my resources for grokking testing Akka Persistence Cassandra, and have a question about cassandra-env.sh issues when using CassandraLoader: I'm seeing the following in my terminal window near the beginning of my test suite: 11/05 13:16:32 ERROR[main] o.a.c.s.StartupChecks - cassandra.jmx.local.port missing from cassandra-env.sh, unable to start local JMX service.

I may be off, but I can't help but think that the error above is responsible for the rest...

Further down, I'm seeing multiple connection errors: 13:17:14.830 [cluster2-nio-worker-0] DEBUG com.datastax.driver.core.Connection - Connection[/127.0.0.1:19042-1, inFlight=0, closed=false] connection error java.io.IOException: Connection reset by peer ... And further multiple connection errors: com.datastax.driver.core.exceptions.TransportException: [/127.0.0.1:19042] Cannot connect ... Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:19042

I did some research on the LOCAL_JMX issue, and that is something that would usually be dealt with in the cassandra-env.sh file, which we don't have for the embedded C*. And I tried doing a fork in Test := true, followed by a envVars in Test := Map("LOCAL_JMX" -> "no") in my build.sbt, but to no avail.

Any thoughts?

cassandralauncher-unittests-stacktrace.txt

[murphye]

cassandra.jmx.local.port missing from cassandra-env.sh, unable to start local JMX service I believe is a red herring. I was getting that too but still able to run the test.

https://github.com/murphye/akka-customer/blob/98fce78d5b076ba86c213a3a6c9567fa59284225/src/test/resources/application.conf#L1

This is where I am setting the port #. I did this because without it there could be a conflict with another instance of Cassandra running. It is not necessary however. Try commenting out that line and see what happens.

I didn't get very deep with creating these tests. I had my own issue with how it was creating the actor system and what I had to name the test. Watch out for that one.

https://github.com/murphye/akka-customer/blob/98fce78d5b076ba86c213a3a6c9567fa59284225/src/test/java/lightbend/customer/customer.java#L18

[murphye]

13:37:47.084 [cluster1-worker-3] DEBUG c.d.driver.core.ControlConnection - [Control connection] Refreshing schema for spike.messages (TABLE)
[info] - should report the correct state *** FAILED ***
[info]   java.lang.AssertionError: assertion failed: timeout (5 seconds) during expectMsg while waiting for 5
[info]   at scala.Predef$.assert(Predef.scala:219)
[info]   at akka.testkit.TestKitBase.expectMsg_internal(TestKit.scala:401)
[info]   at akka.testkit.TestKitBase.expectMsg(TestKit.scala:387)
[info]   at akka.testkit.TestKitBase.expectMsg$(TestKit.scala:387)
[info]   at akka.testkit.TestKit.expectMsg(TestKit.scala:881)
[info]   at spike.cass.persist.AdderSpec.$anonfun$new$6(AdderSpec.scala:44)
[info]   at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   ...
13:37:47.864 [cluster1-nio-worker-0] DEBUG com.datastax.driver.core.Cluster - Received event EVENT DOWN /127.0.0.1:19042, scheduling delivery
13:37:47.864 [cluster2-nio-worker-0] DEBUG com.datastax.driver.core.Cluster - Received event EVENT DOWN /127.0.0.1:19042, scheduling delivery

This seems to be where your problem is stemming from.

[lukewyman]

Okay, thanks for that. I've left questions in the gitter room for akka-persistence-cassandra. I've been toggling your 19042 with 9042. I either use your configuration with 19042 and let CassandraLauncher do it's thing in test setup and teardown, or I use 9042, disable the setup/teardown, and run the tests against a Cassandra instance running on my laptop.

So, I'm definitely in a world of hurt when it comes to the Launcher. When I disable the Launcher and run against my local instance of C*, things go pretty well, with the exception of seeing dead letters warnings when the persistent actor tries to recover (the test intentionally stops and starts the actor to test this).

...and thanks for picking up the timeout, will give that a look over.

[murphye]

Ok, good luck. If you look at last 2 lines you can see that C* is being shut down as a result of that timeout. Look at your actor system name you may need to rename your test. See my first comment and link where I have the TODO

[lukewyman]

OMG!!! EVERYTHING WORKS!!! Thanks for your help on this, Eric. My tests run against the CassandraLauncher just dandy. I thought that after the timeout issue that I was going to get down and wrestle with this another night for the recovery part of my tests, but my test also verifies the recovery/message replay of the actor, so all good. I didn't have any problems with the test name, but that may be an issue with Java/JUnit that doesn't exist with Scala/scala-test. Thanks again!

[murphye]

No problem! Are you using Akka HTTP calling Akka Peristence? If so, and you're not already, definitely check out using Cluster Sharding as I do in this example application. Pretty much is the recommended way to do it. Are you working on your project for your company?

[lukewyman]

Yeah, I'll be building onto what I have so far and do the whole shebang with Akka HTTP that calls a sharded cluster where my persistent actors live. I'll definitely be perusing your project further for the HTTP and cluster sharding pieces.

This little project is a spike for a company I'm freelancing with at the moment. We're about to upgrade to Scala 2.12 and Akka 2.5.6 on our main initiative, and we'll use persistence/CQRS for the first time for this particular module. So, it's going to have a spike, develop, spike, develop rhythm to it for a bit.