search

logzio / jmx2graphite

JMX to Graphite every x seconds in one command line (Docker based) (also come in Java Agent flavour)
MIT License
77 stars 23 forks source link
graphite java jmx jolokia metrics opensource

readme

example workflow name

jmx2graphite

jmx2graphite is a one liner tool for polling JMX and writes into Graphite (every 30 seconds by default). You install & run it on every machine you want to poll its JMX.

Currently it has two flavors:

  1. Docker image which reads JMX from a jolokia agent running on a JVM, since exposing JMX is the simplest and easiest through Jolokia agent (1 liner - see below).
  2. Run as a java agent, and get metrics directly from MBean Platform

The metrics reported have the following names template:

[service-name].[service-host].[metric-name]

How to run?

Using Docker (preferred)

If you don't have docker, install it first - instructions here.

Run the docker either with environment variables or with a configuration file.

Docker with env variables

docker run -i -t -d --name jmx2graphite \
   -e "JOLOKIA_URL=http://172.1.1.2:11001/jolokia/" \
   -e "SERVICE_NAME=MyApp" \
   -e "GRAPHITE_HOST=graphite.foo.com" \
   -e "GRAPHITE_PROTOCOL=pickled" \
   -v /var/log/jmx2graphite:/var/log/jmx2graphite \
   --rm=true \
   logzio/jmx2graphite

Environment variables

Optional environment variables

Docker with config file

Create a .conf file, set the input parameter and provide it to the docker.

See our sample config file].

You can find a complete list of the required parameters here

docker run -d --name jmx2graphite -v path/to/config/myConfig.conf:/application.conf logzio/jmx2graphite

Note: The config file at the docker end must be name application.conf

Rest of command

Using bash + Jolokia agent

  1. get the java agent jar from the releases page

  2. Create a config file that will contain the input parameters, see our sample config file - The mandatory items are:

    1. service.jolokiaFullUrl - Fill in the full URL to the JVM running Jolokia (It exposes your JMX as a REST service, normally under port 8778).
    2. service.name - The role name of the service.
    3. graphite.hostname - Graphite host name the metrics will be sent to
    4. graphite.port - The port which Graphite listen to.
    5. graphite.connectTimeout - Timeout in seconds for the connection with graphite.
    6. graphite.socketTimeout - Timeout in seconds for the socket.

  3. Run your app with Jolokia agent (instructions below)

  4. run the jar: java -jar jmx2graphite.jar

  5. If you wish to run this as a service you need to create a service wrapper for it. Any pull requests for making it are welcome!

As Java Agent

This lib can also get the metrics from MBean Platform instead of jolokia. In order to do so, we need to run inside the JVM.

How to expose JMX Metrics using Jolokia Agent

  1. Download Jolokia JVM Agent JAR file here.
  2. Add the following command line option to your line running java: 
    -javaagent:path-to-jolokia-jar-file.jar

    For example:

    -javaagent:/opt/jolokia/jolokia-jvm-1.3.2-agent.jar

By default it exposes an HTTP REST interface on port 8778. See here if you want to change it and configure it more. We run all of ours apps using Docker, so to avoid clashes when we map the 8778 port to a unique external port belonging only to this application.

Installing and Configuring Graphite

If you never installed Graphite, this small guide below might be a good place to start. I'm using Docker since it's very easy to install this way.

Installing Graphite

We will install Graphite using a great docker image by hopsoft. I tried several and it was by far the easiest to work with.

  1. Run the following to get basic Graphite up and running

    docker run -d \
  --name graphite \
  -p 80:80 \
  -p 2003:2003 \
  -p 2004:2004 \
  -p 8125:8125/udp \
  -p 8126:8126 \
  hopsoft/graphite-statsd

  2. Now, let's copy out all of its existing configuration files so it will be easy to modify. I will assume you will place it at /home/ubuntu

    cd /home/ubuntu
mkdir graphite
docker cp graphite:/opt/graphite/conf graphite
docker cp graphite:/opt/graphite/webapp/graphite graphite/webapp

  3. Stop graphite by running docker stop graphite

  4. Configuring Graphite: Now edit the following files:

    1. /home/ubuntu/graphite/conf/carbon.conf:
      • MAX_CREATES_PER_MINUTE: Make sure to place high values - for example 10000. The default of 50 means that the first time you run jmx2graphite, all of your metrics are reported at once. If you have more than 50, all other metrics will be dropped.
      • MAX_UPDATES_PER_SECOND: I read a lot there should be a formula for calcualting the value for this field, but that is once you reach high I/O disk utilization. For now, simply place 10000 value there. Otherwise you will get a 1-2 minute lag from the moment jmx2graphite pushes the metric to Graphite until it is viewable in Graphite dashboard
    2. /home/ubuntu/graphite/conf/storage-schemas.conf:
      • in the default section (default_1min_for_1day) make sure retentions is set to the same interval as you are using in jmx2graphite (30seconds by default). Here is an example 
        [default_1min_for_1day]
pattern = .*
retentions = 30s:24h,1min:7d,10min:1800d

        If you have 10s:24h then when doing derivative, you will get null values for each missing 2 points in the 30sec window and the graph will be empty

  5. Create some directories which normally are created by the docker image but since we're mounting /var/log to an empty directory of ours in the host, they don't exists:

    mkdir -p /home/ubuntu/log/nginx
mkdir -p /home/ubuntu/log/carbon
mkdir -p /home/ubuntu/log/graphite

  6. Run Graphite. I use the following short bash script run-graphite.sh:

    #!/bin/bash
 docker run -d \
  --name graphite \
  --rm=true \
  --restart=always \
  -p 80:80 \
  -p 2003:2003 \
  -p 2004:2004 \
  -p 8125:8125/udp \
  -p 8126:8126 \
  -v /home/ubuntu/graphite/storage:/opt/graphite/storage \
  -v /home/ubuntu/log:/var/log \
  -v /home/ubuntu/graphite/conf:/opt/graphite/conf \
  -v /home/ubuntu/graphite/webapp/graphite:/opt/graphite/webapp/graphite \
  hopsoft/graphite-statsd

Configuring Graphite

If you have an existing Graphite installation see the section above "configuring Graphite: Now edit the following files:".

Motivation

I was looking for a tool I can just drop in place, have a 1-liner run command which will then run every 10 seconds, poll my JVM JMX entirely and dump it to Graphite. Of course I started Googling and saw the following:

So after spending roughly 1.5 days fighting with those tools and not getting what I wanted, I sat down to write my own.

Why Docker?

Docker enables jmx2graphite to install and run in one command line! Just about any other solution will requires more steps for installation, and not to mention the development efforts.

Why Jolokia?

Features Roadmap

Contributing

We welcome any contribution! You can help in the following way:

Building and Deploying

Build

Build Java Agent

mvn clean install

Deploy

docker login 
docker push logzio/jmx2graphite

Changelog

License

See the LICENSE file for license rights and limitations (MIT).