ThriftServer is missing from the classpath while testing

ThriftServer & Controller is missing from the classpath while testing

Expected behavior

I want to test thrift client when testing thrift server

Actual behavior

Test Code:

package com.martin.ktz.server.controller

import com.twitter.finatra.thrift.{EmbeddedThriftServer, ThriftClient}
import com.twitter.util.Future
import com.martin.ktz.server.{MyThriftServer}
import com.martin.ktz.thriftscala.TMyService
import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner

@RunWith(classOf[JUnitRunner])
class MyControllerTest extends IntegrationTest {
  private val modules = Seq(...)

  val testInjector = TestInjector(modules).create

  override protected def injector: Injector = testInjector
  val server = new EmbeddedThriftServer(new MyThriftServer) with ThriftClient
  lazy val client: TMyService[Future] = server.thriftClient[TMyService[Future]](clientId = "client123")
  ...
}

Server Main code

package com.martin.ktz

import com.twitter.finatra.thrift.ThriftServer
import com.twitter.finatra.thrift.filters._
import com.twitter.finatra.thrift.routing.ThriftRouter
import com.twitter.inject.TwitterModule
import com.twitter.inject.server.Ports
import com.martin.ktz.server.v1.modules.{HttpClientModules, ThriftModule}
import com.martin.ktz.server.v2.modules.{JinroLoggerModule, UserInfoCacheModules}

/**
  * Created by ktz on 2016. 11. 15..
  */
object MyThriftServerMain extends MyThriftServer

class MyThriftServer extends ThriftServer with Ports {
  override val name = "my"

  override val defaultFinatraThriftPort: String = ":8080"

  override def thriftPort: Option[Int] = Some(8080)

  override val modules: Seq[TwitterModule] = Seq(...)

  override protected def configureThrift(router: ThriftRouter): Unit = {
    router
      .filter[LoggingMDCFilter]
      .filter[TraceIdMDCFilter]
      .filter[ThriftMDCFilter]
      .filter[MyAccessLoggingFilterFilter]
      .filter[StatsFilter]
      .add[MyController]
  }

  override def warmup(): Unit = {
    handle[ThriftWarmupHandler]
  }
}

And error occures like this.

[error] /Users/ktz/IdeaProjects/My/my-server/src/test/scala/com/martin/ktz/server/controller/MyControllerTest.scala:12:45: Symbol 'type com.twitter.finatra.thrift.ThriftServer' is missing from the classpath.
[error] This symbol is required by 'class com.martin.ktz.server.MyThriftServer'.
[error] Make sure that type ThriftServer is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[error] A full rebuild may help if 'MyThriftServer.class' was compiled against an incompatible version of com.twitter.finatra.thrift.
[error]   val server = new EmbeddedThriftServer(new MyThriftServer) with ThriftClient

And also I have classpath in test dependencyClasspath

...
[info] * Attributed(/Users/ktz/.ivy2/cache/com.twitter/finatra-thrift_2.12/jars/finatra-thrift_2.12-18.4.0-tests.jar)
[info] * Attributed(/Users/ktz/.ivy2/cache/com.twitter/inject-thrift_2.12/jars/inject-thrift_2.12-18.4.0.jar)
...

Sorry for poor documented issue.. If any more information to solve, I will check... Thanks!

@knightpop can you share the configuration for whatever build system you are using? It's not quite clear what is going on from just the code and the exception since this seems to be a problem with the build setup. Thanks!

@cacoco Sorry for the late reply, Ok, is giving the part of build.sbt file can help?

It is multi-project, and serverside build.sbt is here...

lazy val baseSettings = Seq(
  organization := "com.martin",
  scalaVersion := "2.12.6",
  crossScalaVersions := Seq("2.11.11", "2.12.4"),
  releaseCrossBuild := true,
  releaseProcess := Seq[ReleaseStep](
    checkSnapshotDependencies,
    inquireVersions,
    runClean,
    runTest,
    setReleaseVersion,
    commitReleaseVersion,
    tagRelease,
    //        releaseStepCommand("publishSigned"),
    publishArtifacts,
    setNextVersion,
    commitNextVersion,
    //releaseStepCommand("sonatypeReleaseAll"),
    pushChanges
  ),
  scalafmtVersion in ThisBuild := "1.4.0",
  scalafmtOnCompile in ThisBuild := true,
  scalafmtTestOnCompile in ThisBuild := true,
  aggregate in assembly := false,
  assemblyMergeStrategy in assembly := {
    case "BUILD"                                                  => MergeStrategy.discard
    case "version.properties"                                     => MergeStrategy.concat
    case "META-INF/io.netty.versions.properties"                  => MergeStrategy.last
    case PathList("org", "apache", "commons", "logging", xs @ _*) => MergeStrategy.last
    case PathList("org", "apache", "thrift", xs @ _*)             => MergeStrategy.last
    case PathList("org", "apache", "log4j", xs @ _*)              => MergeStrategy.discard
    case other                                                    => MergeStrategy.defaultMergeStrategy(other)
  },
  aggregate in assembly := false,
  assemblyOutputPath in assembly := file("target/scala-2.11/katalk_gift_iu.jar"),
  fork in run := true,
  fork in Test := true,
  scalacOptions ++= Seq(
    //"-Xfatal-warnings",
    "-deprecation",
    "-encoding",
    "UTF-8",
    "-feature",
    "-language:implicitConversions",
    "-target:jvm-1.8",
    "-unchecked",
    "-Xlint",
    "-Ylog-classpath",
    //    "-Yno-adapted-args",
    //   "-Ywarn-dead-code",
    //    "-Ywarn-numeric-widen",
    //    "-Ywarn-value-discard",
    "-Xfuture",
    //  "-Ywarn-unused-import"
    "-language:postfixOps",
    "-language:higherKinds",
    "-Ypartial-unification"
  ),
  javacOptions ++= Seq(
    "-encoding",
    "UTF-8"
  )
)

...

lazy val `my-server` = (project in file("iu-server"))
  .settings(baseSettings)
  .settings(
    name := "iu-server",
    libraryDependencies ++= Seq(
      "com.martin" %% "durian" % versions.durian, // It has Finatra 18.4.0
      "com.twitter" %% "bijection-core" % versions.bijection,
      "com.twitter" %% "bijection-util" % versions.bijection,
      "com.getsentry.raven" % "raven-logback" % versions.raven_logback,
      "com.github.ikhoon" %% "scalacache-arcus" % versions.scalaCache,
      "io.bfil" %% "automapper" % versions.autoMapper,
      // test
      "com.martin" %% "durian-test" % versions.durian % Test, // It has Finatra Test 18.4.0
      "org.specs2" %% "specs2-mock" % versions.specs2 % Test,
      "org.json4s" %% "json4s-native" % versions.json4s % Test,
      "com.github.kardapoltsev" %% "json4s-java-time" % versions.json4sJavaTime % Test
    )
  )
  .dependsOn(`iu-core` % "test->test;compile->compile")

It's all and tell me whatever can help. Thanks for your help!

@knightpop I don't see where you depend on finatra-thrift at all let alone in the necessary compile scope? Can you perhaps specify your Finatra dependencies explicitly rather than relying on them transitively? It would make things easier.

For your case you want the compile scope of finatra-thrift_2.XX in both test and compile scopes, you additionally want the test scope (the jar with the "tests" classifier) of finatra-thrift_2.XX in your test scope. If you don't have that you will run into issues.

More information here: https://twitter.github.io/finatra/index.html#test-dependencies

An older example here: https://github.com/twitter/finatra-activator-thrift-seed

Ok, finatra-thrift is in some internal library called durian. I will disassemble that dependency.

lazy val `my-server` = (project in file("iu-server"))
  .settings(baseSettings)
  .settings(
    name := "iu-server",
    libraryDependencies ++= Seq(
     "com.fasterxml.jackson.core" % "jackson-databind" % versions.jackson,
      "com.fasterxml.jackson.datatype" % "jackson-datatype-jsr310" % versions.jackson,
      "com.chuusai" %% "shapeless" % versions.shapeless,
      "io.circe" %% "circe-core" % versions.circe,
      "io.circe" %% "circe-parser" % versions.circe,
      "co.fs2" %% "fs2-core" % versions.fs2,
      "com.github.zainab-ali" %% "fs2-reactive-streams" % versions.fs2Reactive,
      "com.twitter" %% "finatra-http" % versions.twitter,
      "com.twitter" %% "finatra-httpclient" % versions.twitter,
      "com.twitter" %% "finagle-redis" % versions.twitter,
      "com.github.ikhoon" %% "finatra-swagger" % versions.swagger,
      "io.circe" %% "circe-core" % versions.circe,
      "io.circe" %% "circe-generic" % versions.circe,
      "io.circe" %% "circe-parser" % versions.circe,
      "io.circe" %% "circe-java8" % versions.circe,
      "com.twitter" %% "finatra-thrift" % versions.twitter,
      "com.twitter" %% "finagle-thrift" % versions.twitter,
      "com.twitter" %% "finagle-thriftmux" % versions.twitter,
      ("com.twitter" %% "util-zk" % versions.twitter).exclude("org.slf4j", "slf4j-log4j12"),
      "com.github.pathikrit" %% "better-files" % versions.betterFiles % Test,
      "com.typesafe.akka" %% "akka-actor" % versions.akka % Test,
      "org.scalatest" %% "scalatest" % versions.scalatest % Test,
      "com.twitter" %% "finatra-http" % versions.twitter % Test,
      "com.twitter" %% "finatra-jackson" % versions.twitter % Test,
      "com.twitter" %% "inject-server" % versions.twitter % Test,
      "com.twitter" %% "inject-app" % versions.twitter % Test,
      "com.twitter" %% "inject-core" % versions.twitter % Test,
      "com.twitter" %% "inject-modules" % versions.twitter % Test,
      "com.google.inject.extensions" % "guice-testlib" % versions.guice % Test,
      "org.mockito" % "mockito-core" % versions.mockito % Test,
      "com.github.alexarchambault" %% "scalacheck-shapeless_1.13" % versions.scalacheckShapeless % Test,
      "com.fortysevendeg" %% "scalacheck-datetime" % versions.scalacheckDatetime % Test,
      "junit" % "junit" % versions.junit % Test,
      ("com.twitter" %% "finatra-http" % versions.twitter).classifier("tests"),
      ("com.twitter" %% "finatra-jackson" % versions.twitter).classifier("tests"),
      ("com.twitter" %% "finatra-thrift" % versions.twitter).classifier("tests"),
      ("com.twitter" %% "inject-server" % versions.twitter).classifier("tests"),
      ("com.twitter" %% "inject-app" % versions.twitter).classifier("tests"),
      ("com.twitter" %% "inject-core" % versions.twitter).classifier("tests"),
      ("com.twitter" %% "inject-modules" % versions.twitter).classifier("tests"),
      ("com.martin.account" % "kakao-account-client" % versions.kakaoAccountClient)
        .exclude("org.slf4j", "slf4j-log4j12"),
      "com.martin.iceland" % "iceland-papi-client" % versions.icelandPapi,
      "com.twitter" %% "bijection-core" % versions.bijection,
      "com.twitter" %% "bijection-util" % versions.bijection,
      "com.getsentry.raven" % "raven-logback" % versions.raven_logback,
      "com.github.ikhoon" %% "scalacache-arcus" % versions.scalaCache,
      "io.bfil" %% "automapper" % versions.autoMapper,
      // test
      "org.specs2" %% "specs2-mock" % versions.specs2 % Test,
      "org.json4s" %% "json4s-native" % versions.json4s % Test,
      "com.github.kardapoltsev" %% "json4s-java-time" % versions.json4sJavaTime % Test
    )
  )
  .dependsOn(`iu-core` % "test->test;compile->compile")

I also add classifier in sbt also, but it is not work.. finatra version is 18.4.0. Hmm.. It's hard to find out by this way... Can you give me some hint to find the reason? Let's say like, we have to look into the classpath by show runtime:fullClasspath?

@knightpop you need not just the classifier tests, that only adds the test-jar to your compile classpath (and not your test classpath). I believe you need the classifier tests in the test scope which is the % "test->test" syntax, which says your project's "test" configuration uses dependency's "test" configuration. This would put the test-jar on your test classpath. This should be in addition to having the compile scope declared.

This isn't really a Finatra issue so much as an understanding sbt issue. I'm happy to help debug more but the sbt documentation (in addition to the examples I previously linked) is probably a better place to look for answers. Specfically the section on Ivy Configurations.

Ok!! I will try thank you very much! And always thans for making fantastic library, Finatra!!

twitter / finatra