Description: I am using the following helm chart to deploy Kafka rest proxy with the kafka-rest image tag: 7.0.1. We are encountering an issue with internal communication between two containers in our Kubernetes deployment: the kafka-rest-server-container and thejmx-exporter-container. The kafka-rest-server-container is exposing a JMX port, configured through an environment variable, for the jmx-exporter-container to scrape metrics. However, whenever the jmx-exporter-container attempts to connect to the JMX port on localhost, it encounters a "Connection Refused" error.
Logs: Below are the logs from the jmx-exporter-container that indicate the connection refusal:
SEVERE: [!dt dt.entity.host=HOST-BF4FF85A33F813E3,dt.entity.process_group=PROCESS_GROUP-E763B61EAEB2F724,dt.entity.process_group_instance=PROCESS_GROUP_INSTANCE-9C7A9A7773EF297F,dt.kubernetes.cluster.id=e3e50ceb-9180-4ac5-937e-e87169bb2b16,dt.kubernetes.workload.kind=Deployment,dt.kubernetes.workload.name=kafka-dev-shared-services-kafka-staging-cp-kafka-rest-proxy,k8s.namespace.name=kafka,k8s.pod.name=kafka-dev-shared-services-kafka-staging-cp-kafka-rest-proxcmhh6,k8s.pod.uid=a7b3a612-4dd5-4ff3-8afe-2fa8267f482b] JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
...
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
Expected Behavior: Thejmx-exporter-container should successfully connect to the JMX port on the kafka-rest-server-container to retrieve metrics data.
Current Behavior: The connection to the JMX port on localhost is being refused, as shown in the logs above.
Environment:
Kubernetes Deployment
Rest-Server Container JMX Port: Configured via KAFKAREST_JMX_PORT environment variable
Description: I am using the following helm chart to deploy
Kafka rest proxy
with thekafka-rest image tag: 7.0.1
. We are encountering an issue with internal communication between two containers in our Kubernetes deployment: thekafka-rest-server-container
and thejmx-exporter-container.
Thekafka-rest-server-container
is exposing a JMX port, configured through an environment variable, for the jmx-exporter-container to scrape metrics. However, whenever thejmx-exporter-container
attempts to connect to the JMX port on localhost, it encounters a "Connection Refused" error.Logs: Below are the logs from the
jmx-exporter-container
that indicate the connection refusal:Expected Behavior: The
jmx-exporter-container
should successfully connect to the JMX port on thekafka-rest-server-container
to retrieve metrics data.Current Behavior: The connection to the JMX port on localhost is being refused, as shown in the logs above.
Environment:
Relevant Configuration:
JMX Configuration:
Kubernetes Deployment Environment Variables for the
kafka-rest-server-container
:Note: tried to use
JMX_PORT
and got same issue !