Kafka monitoring & connection issues

[monimail]

Hello,

I am trying to monitor how Kafka behaves in terms of performance. I tried it with jolokia in the openbmp_aio container as described here : https://github.com/paksu/kafka-jolokia-telegraf-collector but jolokia never sends anything.

As a result, I installed kafka on my host server in order to try to install jolokia more easily, but now I'm getting those log lines :

2018-05-14T11:57:38.613781 | ERROR    | send_bmp_raw       | rtr=xx.xx.xx.xx: Failed to produce bmp raw message: Local: Queue full
2018-05-14T11:57:38.714158 | ERROR    | produce            | rtr=xx.xx.xx.xx: Failed to produce message: Local: Queue full
2018-05-14T11:57:38.814367 | ERROR    | produce            | rtr=xx.xx.xx.xx: Failed to produce message: Local: Queue full
2018-05-14T11:57:38.914610 | ERROR    | send_bmp_raw       | rtr=xx.xx.xx.xx: Failed to produce bmp raw message: Local: Queue full

I am also having issues with the openbmp_kafka container that was not accepting TCP connections from the other containers (as described in https://github.com/SNAS/website/issues/85).

Any help would be appreciated.

Thanks in advance.

[3fr61n]

For monitoring kafka, I used logstash (with jmx plugin)

input {
    jmx {
        path => "/usr/share/logstash/resources/jmx"
        polling_frequency => 60
        type => "kafka-jmx"
        nb_thread => 4
    }

In the jmx directory I have the folowing json with all beans to be monitored

.... cat kafka.json 
{
  "host" : "kafka",
  "port" : 9999,
  "alias" : "ns",
  "queries" : [
  {
   "object_name" : "kafka.server:type=*,name=*"
  },
  {
   "object_name" : "kafka.server:type=*,name=*,topic=*"
  },
  {
   "object_name" : "kafka.cluster:name=*,partition=*,topic=*,type=*"
  },
  {
   "object_name" : "kafka.log:name=*,partition=*,topic=*,type=*"
  } ]
}

Then you can insert that date in whatever database you like (and logstash support).

Hope this would be useful

Regards