Config reload does not report new metrics

[bl1nk]

hey there,

This is the config file we use:

<jmxtrans-agent>
    <queries>
        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attributes="bytesReceived,bytesSent,errorCount,processingTime,requestCount" resultAlias="tomcat.%name%.#attribute#"/>
        <query objectName="Catalina:type=ThreadPool,name=*" attributes="currentThreadCount,currentThreadsBusy,maxThreads" resultAlias="tomcat.%name%.#attribute#"/>
        <query objectName="java.lang:type=ClassLoading" attributes="LoadedClassCount,TotalLoadedClassCount,UnloadedClassCount" resultAlias="tomcat.ClassLoading.#attribute#"/>
        <query objectName="java.lang:type=Compilation" attribute="TotalCompilationTime" resultAlias="tomcat.Compilation.TotalCompilationTime"/>
        <query objectName="java.lang:type=GarbageCollector,name=*" attributes="CollectionCount,CollectionTime" resultAlias="tomcat.%name%.#attribute#"/>
        <query objectName="java.lang:type=Memory" attributes="HeapMemoryUsage,NonHeapMemoryUsage" resultAlias="jvm.#attribute#.#key#"/>
        <query objectName="java.lang:type=Memory" attributes="ObjectPendingFinalizationCount" resultAlias="tomcat.Memory.ObjectPendingFinalizationCount"/>
        <query objectName="java.lang:type=MemoryPool,name=*" attribute="Usage" resultAlias="tomcat.%name%.Usage.#key#"/>
        <query objectName="java.lang:type=OperatingSystem" attributes="MaxFileDescriptorCount,OpenFileDescriptorCount,ProcessCpuLoad,SystemCpuLoad,SystemLoadAverage" resultAlias="tomcat.OperatingSystem.#attribute#"/>
        <query objectName="java.lang:type=Runtime" attribute="Uptime" resultAlias="tomcat.Runtime.Uptime"/>
        <query objectName="java.lang:type=Threading" attributes="DaemonThreadCount,PeakThreadCount,ThreadCount,TotalStartedThreadCount" resultAlias="tomcat.Threading.#attribute#"/>
        <query objectName="java.nio:type=*,name=*" attributes="Count,TotalCapacity,MemoryUsed" resultAlias="tomcat.%type%.%name%.#attribute#"/>
        <query objectName="com.zaxxer.hikari:type=*" resultAlias="hikari.%type%.#attribute#"/>
    </queries>
    <outputWriter class="org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter">
        <host>XXX</host>
        <port>2003</port>
        <namePrefix>jmx.undefined-env.undefined-service.#hostname#.</namePrefix>
    </outputWriter>
    <collectIntervalInSeconds>60</collectIntervalInSeconds>
    <reloadConfigurationCheckIntervalInSeconds>30</reloadConfigurationCheckIntervalInSeconds>
</jmxtrans-agent>

We use this script as a cronjob to update the config:

#!/bin/sh
cd /opt/jmxtrans-config
git reset --hard
git pull

sed -i.bak \
    -e "s#<namePrefix>.*#<namePrefix>jmx.${ECS_ENVIRONMENT}.${SERVICENAME}.${HOSTNAME}.</namePrefix>#" \
    /opt/jmxtrans-config/jmxtrans-agent.xml

touch /opt/jmxtrans-config/jmxtrans-agent.xml

Problem is that the config file does not get reloaded or is not adding new queries to be reported to our influxdb. Re-deploying the service makes reporting work. The config file definitely gets updated by the cronjob.

What could be the problem here?

[kerlandsson]

Could you post the agent part of your command line? Is anything logged (to stdout) from jmxtrans-agent in the minutes after you have touched the configuration?

[kerlandsson]

Also, if you could post what jmxtrans-agent logs at startup it would help.

[cyrille-leclerc]

Hi, to add to @kerlandsson comments, could you please activate:

• jmxtrans-agent diagnosis: https://github.com/jmxtrans/jmxtrans-agent/wiki/Troubleshooting#enabling-jmxtransagent-diagnostic-info
• jmxtrans-logs: https://github.com/jmxtrans/jmxtrans-agent/wiki/Troubleshooting#enabling-logs

[cyrille-leclerc]

@kerlandsson note: it may make sense to add a debug line to dump the new configuration with a TRACE log message at https://github.com/jmxtrans/jmxtrans-agent/blob/95915e1b639f25539e1cc4ec898e1b2868f997d3/src/main/java/org/jmxtrans/agent/JmxTransExporter.java#L74-L74

The method JmxTransExporterConfiguration.toString() dumps everything.

what do you think?

[kerlandsson]

@cyrille-leclerc Makes sense. I went ahead and added it: 47f8efbbf4489459c4547519f70265b1850cfbad

[cyrille-leclerc]

@bl1nk we have deployed a new snapshot with @kerlandsson's improvement.

Could you test with it?

https://oss.sonatype.org/content/repositories/snapshots/org/jmxtrans/agent/jmxtrans-agent/1.2.4-SNAPSHOT/jmxtrans-agent-1.2.4-20160519.084523-1.jar

[bl1nk]

I've deployed a service with your new jmxtrans agent and with logging and diagnosis enabled. I removed a query from my jmxtrans config and it stopped reporting metrics for that query. After adding the query to my config jmxtrans started reporting metrics again, logs confirm both of this.

I would close this... for now. It didn't work with another query before, but I can't reproduce it. Thanks for your help!

[kerlandsson]

I'm happy it is resolved! It might have been that you used a too old version - the configuration reload functionality is quite new.

[bl1nk]

We have been using 1.2.0 before, I just updated to 1.2.3