Add ElasticSearchOutputWriter

[cyrille-leclerc]

Follow-up on the email thread "Elastichsearch output writer strategies" initiated by @zepouet .

https://groups.google.com/forum/#!topic/jmxtrans/jIyYctTW2go

It would be nice to have an ElasticSearchOutputWriter.

ElasticSearch Data Structure

• Question: One unique Index or rolling Indexes suffixing the index by the date? If we store all the metrics in the same ElasticSearch Index, it will be harder to implement a retention policy purging the old metrics.
  • @evgeniy-khist has decided to on his ElasticSearchOutputWriter have a configurable index name capable of using SimpleDateFormat to add a temporal element in the index name. Default value jmxtrans-%{yyyy.MM.dd}. See here
• Question: What should be the data structure? A nested/document-oriented data structure or a flat structure?
  • Question: JMXTrans heavily relies on "." characters in metric names, usually to provide grouping levels. How do we do with ElasticSearch that sees "." as document nesting instructions.
  • @evgeniy-khist has decided to have very small documents (@timestamp, metricName, metricValue). See here
  • jmxtrans has decided to also have one document per collected metric but also stores the metrics meta data here

ElasticSearch Client and Protocol

• Question: We don't want jmxtrans-agent to have dependencies that would mess with the classpath, is it reasonable to bypass standard elasticsearch java client and implement a basic HTTP POST call to send the metrics? We will probably also need to create the index mapping.

[gehel]

Question: We don't want jmxtrans-agent to have dependencies that would mess with the classpath, is it reasonable to bypass standard elasticsearch java client and implement a basic HTTP POST call to send the metrics?

Yes, it is perfectly reasonable. Elasticsearch clients for most languages talk to the REST API and do not use the Java based internal protocol.

We will probably also need to create the index mapping.

Elasticsearch has an auto create mode for indices / mappings. If I remember correctly, that's even the default configuration. If the index / mapping used by jmxtrans will probably be simple enough that the auto mapping should work just fine. Explicitly creating the index / mapping would be nice, but not strictly required.

[zepouet]

You are right. I just watched how you did with InfluxDB and it seems good to have a minimalist API. So no hard dependencies with the product.

[zepouet]

Hello,

I created a fork from your jmxtrans-agent to test the addon of Elastic https://github.com/Treeptik/jmxtrans-agent

It is inspired by your work with Jest API (come from ElasticOutputWriter into JmxTrans) and the work from @evgeniy-khist

At the beginning I tried tu use Elastic Java Transport Client but I have a segmentation fault in JVM ! I will open a ticket at Elastic (tomorrow I must have a call with David P. from Elastic) The crash came after 15m with intensive usage.

For the moment, it is nice with Jest. I prefer Jest to Elastic Client Rest. My fork is not ended (tests to do but I will use these days).

Thanks !