Potentially Creating Large Amount of String Objects

[huanggx-sea]

Hi,

We're using strimzi kafka which uses jmx_exporter as a javaagent to export the metrics to /metrics.

And we find out that Receiver.recordBean under JmxCollector in jmx_exporter keeps creating a large amount of String objects

public void recordBean(...){
    ...
    for(Rule rule:config.rules){
        ...
        String matchName=beanName+(rule.attrNameSnakeCase?attrNameSnakeCase:attrName)+": "+matchBeanValue;
        ...
    }
    ...
}

Arthas shows that recordBean is called about 10k per minute and config.rules in our case is 21 long, which ends up about 210k String per minute.

And we can see Young GC moving the memory from 6GB to 500MB happens about 12 minutes a time.

In the heap dump, there are lots of:

kafka.server<type=socket-server-metrics, listener=TLS-9093, networkProcessor=29><>expired-connections-killed-count: 0.0
kafka.server<type=socket-server-metrics, listener=TLS-9093, networkProcessor=29><>incoming-byte-rate: 0.0
kafka.server<type=socket-server-metrics, listener=TLS-9093, networkProcessor=29><>request-size-avg: NaN
kafka.network<type=RequestMetrics, name=RequestQueueTimeMs, request=AlterClientQuotas><>98thPercentile: 0.0
...

which seems to be the content of matchName here.

Any idea how to fix this problem? Many Thanks.

[fstab]

Hi, thanks a lot for your analysis. I will certainly look into it as soon as time allows. Meanwhile, if you have a proposal how to reduce the number of strings, please post it here for discussion or create a PR.

[huanggx-sea]

Hi, thanks for your reply! To be honest, we don't have much experience in in Java programming.

I'm not sure whether we can use the same memory space for the String matchName in the inner loop

for(Rule rule:config.rules){
        ...
        String matchName=beanName+(rule.attrNameSnakeCase?attrNameSnakeCase:attrName)+": "+matchBeanValue;
        ...
}

This would at least make the number of String not proportional to the number of rules.

[dhoard]

This logic was changed slightly in 0.19.0 so that the attribute snake case name is only created if required by the rule.

https://github.com/prometheus/jmx_exporter/blob/a765bdf99c40f38ca246106ea2e80be49775abc5/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L508-L512

@huanggx-sea can you retest using 0.19.0?