Open lmlynik opened 6 years ago
While debugging to eventually create #162, I found out that the issue is different from what it seems.
it has a bunch of "recoverWith". In particular:
private lazy val _simpleImpl = {
val config = system.settings.config
val dynamic = system.dynamicAccess
def classNameFromConfig(path: String): String =
if (config.hasPath(path)) config.getString(path)
else "<nope>"
def create(clazzName: String) =
dynamic
.createInstanceFor[SimpleServiceDiscovery](clazzName, (classOf[ExtendedActorSystem] → system) :: Nil)
.recoverWith {
case _ ⇒
dynamic.createInstanceFor[SimpleServiceDiscovery](clazzName, (classOf[ActorSystem] → system) :: Nil)
}
.recoverWith {
case _ ⇒
dynamic.createInstanceFor[SimpleServiceDiscovery](clazzName, Nil)
}
// format: OFF
val i = create {
classNameFromConfig("akka.discovery." + _simpleImplMethod + ".class")
}.recoverWith {
case _ ⇒ create(classNameFromConfig(_simpleImplMethod + ".class"))
}.recoverWith {
case _ ⇒ create(_simpleImplMethod) // so perhaps, it is a classname?
}
// format: ON
i.getOrElse(
throw new IllegalArgumentException(
s"Illegal `akka.discovery.method` value '${_simpleImplMethod}' or incompatible class! " +
"The implementation class MUST extend akka.discovery.SimpleServiceDiscovery and take an " +
"ExtendedActorSystem as constructor argument.", i.failed.get)
)
}
The first create
call will try to initialize the correct akka.discovery.kubernetes.KubernetesApiSimpleServiceDiscovery.class
class. However, it may fail for some reasons, like you are not running on kubernetes and therefore the file api-ca-path = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
doesn't exist (at least that was the issue I found in my experiment).
Once it blows up, it goes to next "recoverWith", etc. The final recoverWith will try to instantiate "_simpleImplMethod" which is "kubernetes-api" that is not a java class.
I would suggest you to debug to find out what the underlying issue is. Set the breakpoint in one of those recoverWith to see why akka cannot instantiate the class (maybe you missed some configuration?).
I agree that the exception should give you a more meaningful message, though. Maybe having some logging would not be bad.
@mbuccini good point,
So from what I see the first failure is,
akka.discovery.kubernetes.KubernetesApiSimpleServiceDiscovery.<init>(akka.actor.ExtendedActorSystem)
But the second recover I get the same as error as you mentioned.
Accidentally closed the issue, sorry.
But the second recover I get the same as error as you mentioned.
Yeah, it implies that you are not running it in a valid k8 cluster (or something like that).
I'll just close it and create an issue for better logging.
I think this should still be kept open here @lmlynik, the issue is well described here. i got this issue testing with my local kubernetes as well
I'll reopen to investigate more soon
I'm facing the same issue, but the root cause in my case is:
java.lang.NoSuchMethodException: akka.discovery.kubernetes.KubernetesApiSimpleServiceDiscovery.<init>(akka.actor.ExtendedActorSystem)
Dependencies
[INFO] +- com.typesafe.akka:akka-actor_2.12:jar:2.5.12:compile
[INFO] | +- com.typesafe:config:jar:1.3.2:compile
[INFO] | \- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.8.0:compile
[INFO] +- com.typesafe.akka:akka-testkit_2.12:jar:2.5.12:test
[INFO] +- com.typesafe.akka:akka-cluster-sharding_2.12:jar:2.5.12:compile
[INFO] | +- com.typesafe.akka:akka-persistence_2.12:jar:2.5.12:compile
[INFO] | | \- com.typesafe.akka:akka-protobuf_2.12:jar:2.5.12:compile
[INFO] | \- com.typesafe.akka:akka-cluster-tools_2.12:jar:2.5.12:compile
[INFO] +- com.typesafe.akka:akka-cluster_2.12:jar:2.5.12:compile
[INFO] | \- com.typesafe.akka:akka-remote_2.12:jar:2.5.12:compile
[INFO] | +- io.netty:netty:jar:3.10.6.Final:compile
[INFO] | +- io.aeron:aeron-driver:jar:1.7.0:compile
[INFO] | \- io.aeron:aeron-client:jar:1.7.0:compile
[INFO] | \- org.agrona:agrona:jar:0.9.12:compile
[INFO] +- com.typesafe.akka:akka-distributed-data_2.12:jar:2.5.12:compile
[INFO] | \- org.lmdbjava:lmdbjava:jar:0.6.0:compile
[INFO] | \- com.github.jnr:jnr-constants:jar:0.9.9:compile
[INFO] +- com.typesafe.akka:akka-slf4j_2.12:jar:2.5.12:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.7.22:compile
[INFO] +- com.lightbend.akka.management:akka-management-cluster-bootstrap_2.12:jar:0.14.0:compile
[INFO] | +- com.lightbend.akka.management:akka-management_2.12:jar:0.14.0:compile
[INFO] | +- com.lightbend.akka.discovery:akka-discovery_2.12:jar:0.14.0:compile
[INFO] | +- com.typesafe.akka:akka-http-core_2.12:jar:10.0.13:compile
[INFO] | | \- com.typesafe.akka:akka-parsing_2.12:jar:10.0.13:compile
[INFO] | +- com.typesafe.akka:akka-http-spray-json_2.12:jar:10.0.13:compile
[INFO] | \- io.spray:spray-json_2.12:jar:1.3.3:compile
[INFO] +- com.lightbend.akka.management:akka-management-cluster-http_2.12:jar:0.14.0:compile
[INFO] +- com.lightbend.akka.discovery:akka-discovery-dns_2.12:jar:0.14.0:compile
[INFO] +- com.lightbend.akka.discovery:akka-discovery-kubernetes-api_2.12:jar:0.14.0:compile
[INFO] | +- com.typesafe.akka:akka-http_2.12:jar:10.0.13:compile
[INFO] | \- com.typesafe.akka:akka-stream_2.12:jar:2.5.7:compile
[INFO] | +- org.reactivestreams:reactive-streams:jar:1.0.1:compile
[INFO] | \- com.typesafe:ssl-config-core_2.12:jar:0.2.2:compile
[INFO] | \- org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.0.4:compile
[INFO] \- org.scala-lang:scala-library:jar:2.12.6:compile
Hi everyone, I had similar issue, fix for it was application.conf file
akka.management.cluster.bootstrap.contact-point-discovery {
discovery-method = akka.discovery.kubernetes-api
}
and after adding this line - no problem
@isatimur This actually helps to get a proper error from the discovery class if there is something misconfigured. Imho it is not really fix for the problem. Will look on adding better logging if there is not already a PR for it - so the discovery module doesn't swallow the root exception.
PR would be welcome @teroxik
application.conf
akka {
jvm-exit-on-fatal-error = false
coordinated-shutdown.exit-jvm = on
stdout-loglevel = "DEBUG"
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "DEBUG"
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
#log-config-on-start = on
actor {
provider = "cluster"
allow-java-serialization = off
enable-additional-serialization-bindings = on
serializers { }
serialization-bindings {
"com.messages.JsonSerializable" = jackson-json
}
debug {
autoreceive = on
lifecycle = on
unhandled = on
event-stream = on
fsm = on
}
}
http {
host = "0.0.0.0"
port = 8080
}
}
akka.management {
cluster.bootstrap {
contact-point-discovery {
discovery-method = kubernetes-api
}
}
}
buid.sbt
enablePlugins(JavaServerAppPackaging)
sources in (Compile, doc) := Seq.empty
publishArtifact in (Compile, packageDoc) := false
lazy val akkaHttpVersion = "10.1.10"
lazy val akkaTypedVersion = "2.6.0-M7"
lazy val akkaStreamKafkaTeskitVerion = "1.0.5"
lazy val akkaTypedTestkitVersion = "2.5.8"
lazy val akkaStreamAlpakkaKafka = "1.0.3"
lazy val akkaStreamAlpakkaCassandra = "1.0.0"
lazy val cassandraDriverMappingVersion = "3.5.1"
lazy val cassandraDriverExtrasVersion = "3.5.1"
lazy val chillAkka = "0.9.3"
lazy val libphonenumberVersion = "8.10.16"
lazy val commonsCodedVersion = "1.13"
lazy val logbackVersion = "1.2.3"
lazy val akkaManagement = "1.0.5"
lazy val guice = "4.2.2"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor-typed" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-stream-typed" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-stream-kafka" % akkaStreamAlpakkaKafka,
"com.lightbend.akka" %% "akka-stream-alpakka-cassandra" % akkaStreamAlpakkaCassandra,
"com.typesafe.akka" %% "akka-persistence-typed" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-cluster-sharding" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-cluster-typed" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-cluster-tools" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-http-jackson" % akkaHttpVersion,
"com.typesafe.akka" %% "akka-discovery" % akkaTypedVersion,
"com.datastax.cassandra" % "cassandra-driver-mapping" % cassandraDriverMappingVersion,
"com.datastax.cassandra" % "cassandra-driver-extras" % cassandraDriverExtrasVersion,
"com.twitter" %% "chill-akka" % chillAkka,
"com.typesafe.akka" %% "akka-serialization-jackson" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaTypedVersion,
"ch.qos.logback" % "logback-classic" % logbackVersion,
"com.lightbend.akka.management" %% "akka-management" % akkaManagement,
"com.lightbend.akka.management" %% "akka-management-cluster-http" % akkaManagement,
"com.lightbend.akka.management" %% "akka-management-cluster-bootstrap" % akkaManagement,
"com.googlecode.libphonenumber" % "libphonenumber" % libphonenumberVersion,
"commons-codec" % "commons-codec" % commonsCodedVersion,
"com.google.inject" % "guice" % guice,
"com.typesafe.akka" %% "akka-multi-node-testkit" % akkaTypedVersion,
"com.typesafe.akka" %% "akka-actor-testkit-typed" % akkaTypedVersion % Test,
"com.typesafe.akka" %% "akka-stream-testkit" % akkaTypedVersion % Test,
"com.typesafe.akka" %% "akka-stream-kafka-testkit" % akkaStreamAlpakkaKafka % Test,
"com.typesafe.akka" %% "akka-http-testkit" % akkaHttpVersion % Test,
"junit" % "junit" % "4.12" % Test,
"com.novocode" % "junit-interface" % "0.10" % Test
)
dockerExposedPorts := Seq(8080, 8558, 2552)
dockerBaseImage := "openjdk:8-jre-alpine"
Error
2019-11-20 19:37:56,126 INFO a.m.c.bootstrap.ClusterBootstrap run-main-0 - Bootstrap using `akka.discovery` method: kubernetes-api
[error] (run-main-0) java.lang.IllegalArgumentException: akka.discovery.kubernetes-api.class must contain field `class` that is a FQN of a `akka.discovery.ServiceDiscovery` implementation
[error] java.lang.IllegalArgumentException: akka.discovery.kubernetes-api.class must contain field `class` that is a FQN of a `akka.discovery.ServiceDiscovery` implementation
[error] at akka.discovery.Discovery.classNameFromConfig$1(Discovery.scala:66)
[error] at akka.discovery.Discovery.akka$discovery$Discovery$$createServiceDiscovery(Discovery.scala:82)
[error] at akka.discovery.Discovery$$anon$1.apply(Discovery.scala:21)
[error] at akka.discovery.Discovery$$anon$1.apply(Discovery.scala:20)
[error] at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
[error] at akka.discovery.Discovery.loadServiceDiscovery(Discovery.scala:51)
[error] at akka.management.cluster.bootstrap.ClusterBootstrap.<init>(ClusterBootstrap.scala:52)
[error] at akka.management.cluster.bootstrap.ClusterBootstrap$.createExtension(ClusterBootstrap.scala:114)
[error] at akka.management.cluster.bootstrap.ClusterBootstrap$.createExtension(ClusterBootstrap.scala:108)
[error] at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1137)
[error] at akka.management.scaladsl.AkkaManagement.$anonfun$loadRouteProviders$1(AkkaManagement.scala:259)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
[error] at scala.collection.immutable.List.foreach(List.scala:388)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:233)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
[error] at scala.collection.immutable.List.map(List.scala:294)
[error] at akka.management.scaladsl.AkkaManagement.loadRouteProviders(AkkaManagement.scala:236)
[error] at akka.management.scaladsl.AkkaManagement.<init>(AkkaManagement.scala:75)
[error] at akka.management.scaladsl.AkkaManagement$.createExtension(AkkaManagement.scala:51)
[error] at akka.management.scaladsl.AkkaManagement$.createExtension(AkkaManagement.scala:45)
[error] at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1137)
[error] at akka.actor.ExtensionId.apply(Extension.scala:77)
[error] at akka.actor.ExtensionId.apply$(Extension.scala:76)
[error] at akka.management.scaladsl.AkkaManagement$.apply(AkkaManagement.scala:45)
[error] at akka.management.javadsl.AkkaManagement$.get(AkkaManagement.scala:21)
[error] at akka.management.javadsl.AkkaManagement.get(AkkaManagement.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
@brabo-hi you don't have it on the class path https://doc.akka.io/docs/akka-management/current/discovery/kubernetes.html
This is what I getting when setting up akka boostrapping with k8s discovery.
and dependencies