Metrics integration

[ujjwal]

I am wondering if there an end-end example showing parcel, CSD with metrics integration?

I have a Spring boot based application that I want to deploy as a CSD. I can export the metrics to JMX or StatsD but trying to figure out how it would connect with the MDL to the Cloudera Dashboard.

[cmacdonald]

I'm also asking this question: see http://community.cloudera.com/t5/Cloudera-Manager-Installation/Process-monitoring-for-a-custom-service-in-Cloudera-Manager/m-p/37150 for the latest of what we have heard from Cloudera

[alxn]

@ujjwal @cmacdonald Looks like KAFKA got an .mdl file.

[cmacdonald]

@alxn thanks. their .kdl file is not where one might expect it, i.e. its in src/main/resources, rather than src/descriptor. The referenced counters are defined in Kafka through Codahale, but I'm not sure how the CSD can calculate the JMX name.

Clearly the Java source now included in the CSD is related somehow - any ideas?

[cmacdonald]

The java source in the KAFKA csd generates the final src/descriptor/service.mdl. I cannot get the KAFKA generator to run, as it relies on a com.cloudera.csd package that is apparently not open source.

On the other hand, the generated service.mdl for KAFKA can be found in a Cloudera distribution:

unzip -c  /usr/share/cmf/csd/KAFKA-5.7.0.jar descriptor/service.mdl | less
<snip>
"metricDefinitions" : [ {
      "context" : "kafka.controller.KafkaController::ActiveControllerCount::value",
      "counter" : false,
      "description" : "Will be 1 if this broker is the active controller, 0 otherwise",
      "label" : "Active Controller",
      "name" : "kafka_active_controller",
      "numeratorUnit" : "controller"
    }

I believe the context attribute to be some form of pattern for a JMX query. HDFS has more complex formats, e.g.:

{
      "context" : "REGEX$$Hadoop:service=DataNode,name=DataNodeActivity::BlockReportsAvgTime",
      "counter" : false,
      "description" : "Block Reports Average Time",
      "label" : "Block Reports Average Time",
      "name" : "hdfs_block_reports_avg_time",
      "numeratorUnit" : "ms",
      "weightingMetricName" : "hdfs_block_reports_num_ops"
    }

[alxn]

@cmacdonald They did update the wiki with the SMDL format... but I didn't notice the repo:

<repository>
      <id>cdh.releases.repo</id>
      <url>http://maven.jenkins.cloudera.com:8081/artifactory/libs-release-local</url>
      <name>Releases Repository</name>
    </repository>

That's brutal...

[cmacdonald]

The repo not accessible.

I also think the documentation has been there a while, but the crucial "context" parameter was not documented.

[cmacdonald]

After more investigation, I can confirm that "context" does not entirely refer to something that can be directly obtained over JMX....

In fact, metrics appears to work is as follows

• the Cloudera Manager agent process contacts the process over an HTTP connection, e.g. http://namenode:50070/jmx
• the exact URL definition and how to parse the output is customised in the agent using an adaptor for each service.

Until the generic adaptor of Cloudera Manager has more functionality by default ( in file /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.0-py2.7.egg/cmf/monitor/generic/adapter.py ), no useful statistics can be distributed in the a custom CSD. Cloudera Manager is hard-coded to the services they currently support (see /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.0-py2.7.egg/cmf/monitor/generic/adapter_factory.py).

[knoguchi]

Same thing for Kafka. curl -H "Content-Type: application/json" http://localhost:24042/api/metrics | jq . gives you the Kafka metrics in JSON that is slightly massaged Yammer(Codahale) metrics.

Then this script monitor/generic/kafka_adapters.py reads the JSON and ingest the metric.

CDH Kafka uses the 3rd party metric exporter class. This class is loaded by the Kafka Broker, and runs the HTTP server on a separate thread.