Upgrading to Spark Version 1.0.0 - StackOverflowError

[tsindot]

I have upgraded a local copy of Calliope to the recently released Spark Version 1.0.0. It compiles just fines. When I execute the test > sbt test

I get a StackOverflowError:

[info] Loading project definition from /Users/tnist/projects/radtech.io/ies/new/calliope/project
[info] Set current project to calliope (in build file:/Users/tnist/projects/radtech.io/ies/new/calliope/)
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] No tests to run for calliope-macros/test:test
java.lang.StackOverflowError
    at java.util.concurrent.atomic.AtomicLong.compareAndSet(AtomicLong.java:149)
    at java.util.concurrent.atomic.AtomicLong.getAndIncrement(AtomicLong.java:177)
    at java.util.logging.LogRecord.<init>(LogRecord.java:183)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:575)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:651)
    at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:224)
    at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:301)
    at java.util.logging.Logger.log(Logger.java:565)

It is not a total bust at least 14 of the 25 tests pass:

[error] Could not run test com.tuplejump.calliope.CassandraRDDFunctionsSpec: java.lang.StackOverflowError
[error] Error: Total 14, Failed 0, Errors 0, Passed 14
[error] Error during tests:
[error]     com.tuplejump.calliope.NativeCassandraRDDSpec
[error]     com.tuplejump.calliope.CassandraRDDFunctionsSpec
[error]     com.tuplejump.calliope.ThriftCassandraRDDSpec
[error]     com.tuplejump.calliope.Cql3CassandraRDDSpec
[error] (calliope/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 6 s, completed Jun 1, 2014 5:21:03 PM

I have bumped the sbt version up to 0.13.2 in the project/build.properties, though 0.13.0 generates the same error.

In the project/CalliopeBuild.scala if have changed it as show below:

  lazy val SPARK_VERSION = "1.0.0"

  lazy val calliope = {
    val dependencies = Seq(
      "org.apache.cassandra" % "cassandra-all" % CAS_VERSION % "provided" intransitive(),
      "org.apache.cassandra" % "cassandra-thrift" % CAS_VERSION % "provided" intransitive(),
      "org.apache.thrift" % "libthrift" % THRIFT_VERSION exclude("org.slf4j", "slf4j-api") exclude("javax.servlet", "servlet-api"),
      "com.datastax.cassandra" % "cassandra-driver-core" % DS_DRIVER_VERSION intransitive(),
      "org.slf4j" % "slf4j-jdk14" % "1.7.5",
      "org.apache.spark" %% "spark-core" % SPARK_VERSION % "provided" exclude("org.apache.hadoop", "hadoop-core")  exclude("org.slf4j", "slf4j-api") exclude("org.slf4j", "slf4j-jdk14") exclude("org.slf4j", "slf4j-log4j12"),
      "org.apache.spark" %% "spark-streaming" % SPARK_VERSION % "provided" exclude("org.slf4j", "slf4j-api") exclude("org.slf4j", "slf4j-jdk14") exclude("org.slf4j", "slf4j-log4j12"),
      "org.apache.hadoop" % "hadoop-core" % "1.0.4" % "provided",
      "org.apache.commons" % "commons-lang3" % "3.1",
      "com.github.nscala-time" %% "nscala-time" % "1.0.0",
      "org.scalatest" %% "scalatest" % "1.9.1" % "test"
    )

Any ideas on what may be going on? I can push a branch if that would be of assistance. It maybe something straight forward, is it related to the macros at all? Is there something these four test have in common:

com.tuplejump.calliope.NativeCassandraRDDSpec
com.tuplejump.calliope.CassandraRDDFunctionsSpec
com.tuplejump.calliope.ThriftCassandraRDDSpec
com.tuplejump.calliope.Cql3CassandraRDDSpec

TIA for the assistance.

[milliondreams]

I think you caught our work in progress issue. Should be fixed in the latest dev. Let me know if you still see it.

[tsindot]

@milliondreams

Still seeing the error.

Just did a fetch and rebase then a stash pop of my stashed changes to get the 1.0.0 version of Spark and sbt 0.13.2. Compile is fine as before, but the same 4 tests still fail:

[error] Error: Total 14, Failed 0, Errors 0, Passed 14
[error] Error during tests:
[error]     com.tuplejump.calliope.NativeCassandraRDDSpec
[error]     com.tuplejump.calliope.CassandraRDDFunctionsSpec
[error]     com.tuplejump.calliope.ThriftCassandraRDDSpec
[error]     com.tuplejump.calliope.Cql3CassandraRDDSpec
[error] (calliope/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 25 s, completed Jun 1, 2014 8:46:46 PM

with the same StackoverFlowError:

java.lang.StackOverflowError
    at java.lang.Class.getComponentType(Native Method)
    at java.util.Arrays.copyOf(Arrays.java:2245)
    at java.util.concurrent.CopyOnWriteArrayList.toArray(CopyOnWriteArrayList.java:355)
    at java.util.logging.Logger.getHandlers(Logger.java:1296)
    at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:1333)
    at java.util.logging.Logger.log(Logger.java:564)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:579)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:651)
    at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:224)
    at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:301)
    at java.util.logging.Logger.log(Logger.java:565)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:579)
    at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:651)
    at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:224)
    at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:301)
    at java.util.logging.Logger.log(Logger.java:565)

Looks like an issue with SLF4J, but I'm not seeing the root cause of it right now.

[milliondreams]

@tsindot I just pushed in a change to upgrade spark to 1.0.0. We were already using Scala 2.10.4.

The tests are running now. I have removed the explicit dependency we had on slf4j and commons-lang3. That might have fixed things.

I have not build it with 0.13.5 yet... giving it a go now. But I don't think sbt version has anything to do with this.

[milliondreams]

We are now using sbt 0.13.5 and everything is working.

@tsindot If you are still seeing issues, we can jump in a hangout and try to resolve it.

[tsindot]

@milliondreams

Thanks it works as advertised. Just did a fetch and rebase and all tests are passing. Any ideas on when this may be prompted to an actual release and the bits pushed for public access?

Thanks again for the quick fix.

[milliondreams]

We are working on a major release with Macros and Native CQL RDD and a few other goodies thrown in. We should have a release by end of the next week.