Feature/kafka support

[claudiouzelac]

This pull request (PR) provides Kafka support. This allows metrics to be written to a configured Kafka instance to be ingested and analytics to be performed in near real-time.

---

This change is 

[cyrille-leclerc]

Hello @bowlofstew, thank you very much for your Pull Request. Unfortunately, the message may not have been visible enough on our readme page, "JmxTrans Agent has zero dependencies to ease integration."

We don't want to pull any dependency / library to be sure to not have integration issues. We are also reluctant to shadow libraries for simplicity and for the size of the jmxtrans-agent jar file.

Didn't you have classloader issues with your kafka enabled jmxtrans-agent jar dependending on kafka-clients, jackson-core, guava and kafka_2.11?

I looked at the Kafka Protocol, I didn't find a plain text protocol.

I don't know how we could integrate your contribution. I need to brainstorm to see how we could include this outputwriter not adding constraints and dependencies to people who don't use this KafkaOutputWriter.

[evmin]

@cyrille-leclerc

Interesting topic - side contributions. I would be interested in expanding the agent with Google Cloud Stackdriver Monitoring Writer. While the API is REST based, no issues, the authentication is OAuth - which in theory can be mimicked. But I am not sure I would be able to reproduce it easily - I usually rely on the Google libs.

One option to consider is to leave the dependencies out, but include the code that relies on them.

Logic is simple - if someone needs the extension, they should take care of providing the necessary libs.

I have tried this against 1.2.6-SNAPSHOT. Seems to work. If the writer is not defined in the config - it is not loaded and no exceptions are thrown. If it is defined, and the dependencies are provided by the app - all good too.

Another option - some sort of plugin mechanism.

[claudiouzelac]

Thanks for the response. Unfortunately there is not a plain-text protocol that it exposes directly and to make it harder there is metadata around each topic that has to be handled.

[cyrille-leclerc]

@bowlofstew thanks for the feedback. Can you explain us how you handled the dependencies having a java agent dependending on a bunch of jars (kafka-clients, jackson-core, guava and kafka_2.11)?

@evmin

One option to consider is to leave the dependencies out, but include the code that relies on them. Logic is simple - if someone needs the extension, they should take care of providing the necessary libs.

How would they do this?

• Adding the dependencies (e.g. kafka-clients and guava) in the bootclasspath?
• Creating an uber-jar for the agent?
• ...

[evmin]

@cyrille-leclerc

I have tried both. And opted out for the class path option:

I.e. the required java agent jars need to be bundled in with the application itself. While I include the agent with the JVM docker image, the application developers have to resolve any potential conflicts when they build their docker images.

If they are not going to use certain functionality (i.e. Writer) - the class is not being instantiated by the jmxtrans - and dependencies do not need to be included. Everyone is happy.

Again, this is my case. An uber jar would be an option too.

[claudiouzelac]

Sorry for the delay in response! Much like the above stated, I went down a classpath, path for the consumer. This was done in the context of a fat jar from SBT in my use case.

[cyrille-leclerc]

@bowlofstew few questions please

• Fat jar loaded as an agent, correct?
• Have you been afraid of collisions with the jars loaded by the application?
• Did you evaluate the idea to add the jars in the bootclasspath instead of building your fat jar?

[evmin]

@cyrille-leclerc @bowlofstew

May I provide my 2 cents?

Points raised by @cyrille-leclerc is exactly what we have experienced.

We have tried fat jar and experienced collisions. Solved this, no issues, but the risk is there. So what we have opted out for is to explicitly require the dependency on the classpath for the app itself. This way the developer is forced to face the collisions in the early development cycle and solve them. Either by fixing the app or contributing to the agent.

[cstroe]

A possible resolution to the classpath collisions is to repackage all the agent dependencies with a unique namespace. This could be done at build time with something like the maven-shade-plugin.