sematext / logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint
https://sematext.com/logagent
Apache License 2.0
389 stars 79 forks source link

Implement JMX input #48

Closed otisg closed 6 years ago

otisg commented 7 years ago

Maybe via https://www.npmjs.com/package/jmx (but didn't look into options deeply), though I see this depends on some node-java package and that node-java README mentions Python...

megastef commented 7 years ago

Evaluated and worked so far. Java installation and integration with node-java was a bit of trouble on Mac-OS-X https://github.com/joeferner/node-java/issues/90#issuecomment-45613235. A handy tool to discover JMX beans and property names is https://github.com/sematext/jmxc :)

megastef commented 7 years ago

This should be a separate npm module/plugin, to avoid dependencies on "java".

megastef commented 7 years ago

An alternative to get JMX stats into Logagent could be jmxtrans combined with Logagent command plugin: https://github.com/jmxtrans/jmxtrans/wiki/StdoutWriter

fbalicchia commented 6 years ago

Make some experiments with the last version of node-jmx and haven't trouble in installation on MacOs. Developing ingestion for Cassandra metrics I found that node-jmx seems to store every event emitted. I need to investigate a little more to figure out if is a my implementation problem or something else.

fbalicchia commented 6 years ago

Hi, I' dig a little in node-jmx but during experiments I went against some problems, infact using it in batch mode there was trouble in memory usage or too much event listener was added. I've forked library and remove client and node-jmx layers and use only a mbeanServerConnection with a minor refactoring. At the moment APIs are a little rougth but works, as soon as is possible they will come more fluent. now code look like

'use strict'
const mBeanconnection = require('./lib/adapters/mbeanServerConnectionNew')
const jmxServiceUrlBuilder = require("./lib/adapters/helpers/jmxServiceUrlBuilder"),
checkError = require("./lib/helpers/error").checkError

setInterval(readDatAll.bind(this), 1000)

var serviceOrHost = 'localhost'
var port = 7199
var protocol
var urlPath

var keyValuecqlMetrics = [
["org.apache.cassandra.metrics:type=CQL,name=PreparedStatementsEvicted","Count"],
["org.apache.cassandra.metrics:type=CQL,name=PreparedStatementsCount","Value"],
["org.apache.cassandra.metrics:type=CQL,name=PreparedStatementsExecuted","Count"],
["org.apache.cassandra.metrics:type=CQL,name=PreparedStatementsRatio","Value"],
["org.apache.cassandra.metrics:type=CQL,name=RegularStatementsExecuted","Count"]]

this.cqlMetrics = new Map(keyValuecqlMetrics)

this.url = jmxServiceUrlBuilder(serviceOrHost, port, protocol, urlPath);

function readDatAll(){
    var self = this
    self.cqlMetrics.forEach(function(value, key){
        connectBean(self.url, key,value);
    })
}

function connectBean(url,bean,measure)
{
    var self = this
    var mBean = new mBeanconnection()
    mBean.myconnect(url,function(){
        var self = this;
        mBean.queryMBeans(null, bean, function(instance) {
          instance.getObjectName(function(err, objectName) {
            if (checkError(err, self)) return;
            mBean.getAttribute(objectName,measure, function(data){
                console.log(bean + ' value ' + data)
            });
            mBean.close
            mBean = null
          });
        });
    });
}

At the moment it works for me but any suggestion are welcome ;-) Asap I'll send a Cassandra input metrics plugin

otisg commented 6 years ago

I think we can close this. https://github.com/sematext/sematext-agent-java has JMX support.