search

kamon-io / Kamon

Distributed Tracing, Metrics and Context Propagation for applications running on the JVM
https://kamon.io
Other
1.41k stars 327 forks source link

Kamon breaks "affinity-pool-executor" #776

Open eugenemiretsky opened 4 years ago

eugenemiretsky commented 4 years ago

Kamon seems to break (bring app down), when using affinity-pool-executor

app_1 | akka.actor.ActorInitializationException: akka://quota-service-actor-system/user/redis2-6379-listener-actor: exception during creation app_1 | at akka.actor.ActorInitializationException$.apply(Actor.scala:202) app_1 | at akka.actor.ActorCell.create(ActorCell.scala:696) app_1 | at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:547) app_1 | at akka.actor.ActorCell.systemInvoke(ActorCell.scala:569) app_1 | at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:293) app_1 | at akka.dispatch.Mailbox.run(Mailbox.scala:228) app_1 | at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedThreadPool$TimingRunnable.run(ExecutorInstrumentation.scala:437) app_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) app_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) app_1 | at java.lang.Thread.run(Thread.java:748) app_1 | Caused by: java.lang.NullPointerException: null app_1 | at kamon.instrumentation.akka.instrumentations.akka_25.InstrumentNewExecutorServiceOnAkka25$.around(DispatcherInstrumentation.scala:143) app_1 | at akka.dispatch.affinity.AffinityPoolConfigurator$$anon$1.createExecutorService(AffinityPool.scala) app_1 | at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor$lzycompute(Dispatcher.scala:43) app_1 | at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor(Dispatcher.scala:43) app_1 | at akka.dispatch.ExecutorServiceDelegate.execute(ThreadPoolBuilder.scala:217) app_1 | at akka.dispatch.ExecutorServiceDelegate.execute$(ThreadPoolBuilder.scala:217) app_1 | at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:42) app_1 | at akka.dispatch.Dispatcher.registerForExecution(Dispatcher.scala:126) app_1 | at akka.dispatch.MessageDispatcher.attach(AbstractDispatcher.scala:140) app_1 | at akka.actor.dungeon.Dispatch.start(Dispatch.scala:116) app_1 | at akka.actor.dungeon.Dispatch.start$(Dispatch.scala:114) app_1 | at akka.actor.ActorCell.start(ActorCell.scala:447) app_1 | at akka.actor.LocalActorRef.start(ActorRef.scala:347) app_1 | at akka.actor.dungeon.Children.makeChild(Children.scala:315) app_1 | at akka.actor.dungeon.Children.actorOf(Children.scala:47) app_1 | at akka.actor.dungeon.Children.actorOf$(Children.scala:46) app_1 | at akka.actor.ActorCell.actorOf(ActorCell.scala:447) app_1 | at scredis.io.ListenerActor.createIOActor(ListenerActor.scala:73) app_1 | at scredis.io.ListenerActor.preStart(ListenerActor.scala:270) app_1 | at akka.actor.Actor.aroundPreStart(Actor.scala:550) app_1 | at akka.actor.Actor.aroundPreStart$(Actor.scala:550) app_1 | at scredis.io.ListenerActor.aroundPreStart(ListenerActor.scala:22) app_1 | at akka.actor.ActorCell.create(ActorCell.scala:676)

dlmutart commented 4 years ago

I'm also seeing a NPE with Kamon as soon as I add an affinity-pool-executor to my configuration.

[ERROR] [07/13/2020 11:54:09.375] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user] null java.lang.NullPointerException at java.util.regex.Matcher.getTextLength(Matcher.java:1283) at java.util.regex.Matcher.reset(Matcher.java:309) at java.util.regex.Matcher.(Matcher.java:229) at java.util.regex.Pattern.matcher(Pattern.java:1093) at kamon.util.Filter$Glob.accept(Filter.scala:197) at kamon.util.Filter$IncludeExclude.$anonfun$includes$1(Filter.scala:140) at kamon.util.Filter$IncludeExclude.$anonfun$includes$1$adapted(Filter.scala:140) at scala.collection.LinearSeqOptimized.exists(LinearSeqOptimized.scala:95) at scala.collection.LinearSeqOptimized.exists$(LinearSeqOptimized.scala:92) at scala.collection.immutable.List.exists(List.scala:89) at kamon.util.Filter$IncludeExclude.includes(Filter.scala:140) at kamon.util.Filter$IncludeExclude.accept(Filter.scala:137) at kamon.instrumentation.akka.instrumentations.InstrumentNewExecutorServiceOnAkka25$.around(DispatcherInstrumentation.scala:155) at akka.dispatch.affinity.AffinityPoolConfigurator$$anon$1.createExecutorService(AffinityPool.scala) at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor$lzycompute(Dispatcher.scala:43) at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor(Dispatcher.scala:43) at akka.dispatch.ExecutorServiceDelegate.execute(ThreadPoolBuilder.scala:217) at akka.dispatch.ExecutorServiceDelegate.execute$(ThreadPoolBuilder.scala:217) at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:42) at akka.dispatch.Dispatcher.registerForExecution(Dispatcher.scala:126) at akka.dispatch.MessageDispatcher.attach(AbstractDispatcher.scala:140) at akka.actor.dungeon.Dispatch.start(Dispatch.scala:116) at akka.actor.dungeon.Dispatch.start$(Dispatch.scala:114) at akka.actor.ActorCell.start(ActorCell.scala:447) at akka.actor.ActorCell.start(ActorCell.scala:447) at akka.actor.RepointableActorRef.point(RepointableActorRef.scala:111) at akka.actor.ActorCell.handleSupervise(ActorCell.scala:720) at akka.actor.ActorCell.supervise(ActorCell.scala:706) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:554) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:569) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:293) at akka.dispatch.Mailbox.run(Mailbox.scala:228) at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:653) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

wfudeacon97 commented 4 years ago

We've hit a similar issue. We were not hitting this issue when running on OpenJdk 8, but we did hit this issue with OpenJdk11. On OpenJdk 11, the issue presents with both Kanela-agent 1.0.0 and 1.0.6. The issue goes away when I change my executor away from an affinity-pool.

We are working with kamon-bundle_2.13: 2.1.4 and akka 2.13 (multiple libraries) version: 2.5.24.

Has anyone found a solution to this problem?

dlmutart commented 4 years ago

I'm also seeing a NPE with Kamon as soon as I add an affinity-pool-executor to my configuration.

[ERROR] [07/13/2020 11:54:09.375] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user] null java.lang.NullPointerException at java.util.regex.Matcher.getTextLength(Matcher.java:1283) at java.util.regex.Matcher.reset(Matcher.java:309) at java.util.regex.Matcher.(Matcher.java:229) at java.util.regex.Pattern.matcher(Pattern.java:1093) at kamon.util.Filter$Glob.accept(Filter.scala:197) at kamon.util.Filter$IncludeExclude.$anonfun$includes$1(Filter.scala:140) at kamon.util.Filter$IncludeExclude.$anonfun$includes$1$adapted(Filter.scala:140) at scala.collection.LinearSeqOptimized.exists(LinearSeqOptimized.scala:95) at scala.collection.LinearSeqOptimized.exists$(LinearSeqOptimized.scala:92) at scala.collection.immutable.List.exists(List.scala:89) at kamon.util.Filter$IncludeExclude.includes(Filter.scala:140) at kamon.util.Filter$IncludeExclude.accept(Filter.scala:137) at kamon.instrumentation.akka.instrumentations.InstrumentNewExecutorServiceOnAkka25$.around(DispatcherInstrumentation.scala:155) at akka.dispatch.affinity.AffinityPoolConfigurator$$anon$1.createExecutorService(AffinityPool.scala) at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor$lzycompute(Dispatcher.scala:43) at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor(Dispatcher.scala:43) at akka.dispatch.ExecutorServiceDelegate.execute(ThreadPoolBuilder.scala:217) at akka.dispatch.ExecutorServiceDelegate.execute$(ThreadPoolBuilder.scala:217) at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:42) at akka.dispatch.Dispatcher.registerForExecution(Dispatcher.scala:126) at akka.dispatch.MessageDispatcher.attach(AbstractDispatcher.scala:140) at akka.actor.dungeon.Dispatch.start(Dispatch.scala:116) at akka.actor.dungeon.Dispatch.start$(Dispatch.scala:114) at akka.actor.ActorCell.start(ActorCell.scala:447) at akka.actor.ActorCell.start(ActorCell.scala:447) at akka.actor.RepointableActorRef.point(RepointableActorRef.scala:111) at akka.actor.ActorCell.handleSupervise(ActorCell.scala:720) at akka.actor.ActorCell.supervise(ActorCell.scala:706) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:554) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:569) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:293) at akka.dispatch.Mailbox.run(Mailbox.scala:228) at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:653) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

We are seeing it on Java HotSpot(TM) 64-Bit Server VM (build 25.261-b25, mixed mode)