Metrics report stops

[jfuentescpp]

Integration worked perfectly but after days or weeks some instances stop reporting metrics. Checking the logs:

[kamon-akka.actor.default-dispatcher-4] [akka://kamon/user/kamon-jmx-reporter] Supervisor strategy STOPPING actor from incorrect invocation of JMX registration  
javax.management.MalformedObjectNameException: Unterminated key property part
javax.management.ObjectName.construct(ObjectName.java:556)  
kamon.jmx.MBeanManager$.createOrUpdateMBean(JMXReporterActor.scala:135)
kamon.jmx.JMXReporterActor$.kamon$jmx$JMXReporterActor$$updateCounterMetrics(JMXReporterActor.scala:183)
kamon.jmx.JMXReporterActor$$anonfun$receive$1$$anonfun$applyOrElse$2$$anonfun$apply$6.apply(JMXReporterActor.scala:202)  

and after that a few of:

[akka://kamon/user/kamon-jmx-reporter/kamon-jmx-actor] Message [kamon.metric.SubscriptionsDispatcher$TickMetricSnapshot] from Actor[akka://kamon/user/kamon-jmx-reporter#-391116912] to Actor[akka://kamon/user/kamon-jmx-reporter/kamon-jmx-actor#1569427122] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.  

My configuration looks like:

    kamon {
      jmx {
        percentiles = [50.0, 70.0, 90.0, 95.0, 99.0, 99.9]

        subscriptions {
          histogram = ["**"]
          counter = ["**"]
          akka-actor = ["**"]
          akka-dispatcher = ["**"]
          akka-router = ["**"]
          system-metric = ["**"]
          trace = ["**"]
          trace-segment = ["**"]
          akka-http-server = ["**"]
        }
      }

      akka-http {

        # Header name used when propagating the `TraceContext.token` value across applications.
        trace-token-header-name = "X-Trace-Token"

        # When set to true, Kamon will automatically set and propogate the `TraceContext.token` value.
        automatic-trace-token-propagation = true

        # Fully qualified name of the implementation of kamon.akka.http.AkkaHttpNameGenerator that will be used for assigning names
        # to traces and client http segments.
        name-generator = com.mypackage.AkkaHttpNameGenerator

        client {
          # Strategy used for automatic trace segment generation when issue requests with akka-http-client. The possible values
          # are: request-level and host-level (Not implemented yet!).
          instrumentation-level = request-level
        }
      }

      system-metrics {
        #sigar is enabled by default
        sigar-enabled = true

        #jmx related metrics are enabled by default
        jmx-enabled = true
      }

      metric {
        tick-interval = 60 second
      }
    }

Name generator:

class AkkaHttpNameGenerator extends NameGenerator {
  def generateTraceName(request: HttpRequest): String = request.getUri().path()
  def generateRequestLevelApiSegmentName(request: HttpRequest): String = "request-level " + request.uri.path.toString()
  def generateHostLevelApiSegmentName(request: HttpRequest): String = "host-level " + request.uri.path.toString()
}

[dpsoft]

@jfuentescpp are you using kamon-akka module?

[jfuentescpp]

Hi @dpsoft, I'm using kamon-akka-http. I updated description to give more details, including NameGenerator.

[jfuentescpp]

Hi @dpsoft any ideas about this problem ?

[dpsoft]

@jfuentescpp sorry for the late response, Sounds like the issue could be related to https://github.com/kamon-io/kamon-akka/pull/19 . In order to verify you can do the following:

1- in your buld.sbt import the latest kamon-akka version: 0.6.8 2- in your buld.sbt in the kamon-akka-http exclude the'kamon-akka' transitive dependecy withexclude("io.kamon", "kamon-akka_2.12")`