Closed Drewster727 closed 5 years ago
Update: Digging around I found KAFKA_HEAD_OPTS being set via kafka-run-class.sh:
# Memory options
if [ -z "$KAFKA_HEAP_OPTS" ]; then
KAFKA_HEAP_OPTS="-Xmx256M"
fi
I updated my environment vars to:
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/opt/scripts/prometheus/jmx_prometheus_javaagent-0.11.0.jar=8081:/opt/scripts/prometheus/kafkaMon.yaml"
export KAFKA_HEAD_OPTS="-Xms512m -Xmx1g"
That does resolve the OOM errors, but still getting:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_exporter_build_info
at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:139)
at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:132)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:51)
... 6 more
I found a separate post indicating the collector has already been registered. Why would it be stating this? I'm only executing/starting it once, and only one javaagent.
Update #2: This fixed it for me:
export KAFKA_OPTS="-javaagent:/opt/scripts/prometheus/jmx_prometheus_javaagent-0.11.0.jar=8081:/opt/scripts/prometheus/kafkaMon.yaml"
export KAFKA_HEAD_OPTS="-Xms512m -Xmx1g"
note that I removed the $KAFKA_OPTS
from the beginning... it was causing the javaagent to get registered twice.
Hope this helps someone!
-Drew
i have the same error and same solution (iam using 0.12.1 version), but i have problems when access zookeeper console or other tools that interact with kafka broker.. do you solve this problem definitely? should i use KAFKA_JMX_OPTS?
Hey guys--
Looking to use the jmx_exporter to pull kafka metrics. However, I'm having issues starting kafka after I setup my config, and I don't quite see a smoking gun.
jmx_prometheus_javaagent-0.11.0.jar kafka v0.10.1
Error in /var/log/kafka/kafka.err upon starting:
We're using ambari/zookeeper to manage/broker our kafka nodes. Here is what I'm setting in the kafka-env template in order to set the KAFKA_OPTS:
From the ambari kafka service start logs (my export KAFKA_OPTS is in kafka-env.sh):
My kafkaMon.yaml (config file) -- pulled from example_configs/kafka_0_8_2.yaml:
The kafka services fail to start. Path-ing and permissions looks fine. From the logs, it looks as though the javaagent is actually starting/working, but kafka services ultimately fail to start up.
Can anyone tell me what I can do to diagnose or resolve?
Thanks, Drew