Not able to connect to kafka server

[smitra2020]

1. Installed adapter with a docker-compose Here is the docker-compose file used:

   kafka_adapter:
    image: telefonica/prometheus-kafka-adapter:1.7.0
    container_name: prometheus_kafka_adapter
    environment:
      - LOG_LEVEL=debug
      - KAFKA_BROKER_LIST=localhost:9093
      - KAFKA_TOPIC=metrics
      - PORT=8080
    ports:
      - "8090:8080"

kafka broker running on 9093 port and topic metrics are in good health. Through a console producer, I can push messages to the topic. However, prom-kafka-adapter is not sending metric to the topic "metric". From prometheus to adapter, flow is working fine. We also do not see any error in the adapter log explaining why the adapter is not able to send message in the kafka topic. Please suggest. To check the log in docker, I was using the below command. docker logs prometheus_kafka_adapter -f

Here is the adapter log, sometimes, we are getting JSON parser error:

Failed to obtain reader, Failed to marshal fields to JSON, json: unsupported value: NaN
{"fields.time":"2020-07-01T18:05:52Z","ip":"10.73.172.234","latency":1981058,"level":"info","method":"POST","msg":"","path":"/receive","status":200,"time":"2020-07-01T18:05:52Z","user-agent":"Prometheus/2.19.1"}
Failed to obtain reader, Failed to marshal fields to JSON, json: unsupported value: NaN
{"fields.time":"2020-07-01T18:05:52Z","ip":"10.73.172.234","latency":1737660,"level":"info","method":"POST","msg":"","path":"/receive","status":200,"time":"2020-07-01T18:05:52Z","user-agent":"Prometheus/2.19.1"}
Failed to obtain reader, Failed to marshal fields to JSON, json: unsupported value: NaN
{"fields.time":"2020-07-01T18:05:52Z","ip":"10.73.172.234","latency":1815878,"level":"info","method":"POST","msg":"","path":"/receive","status":200,"time":"2020-07-01T18:05:52Z","user-agent":"Prometheus/2.19.1"}
{"level":"debug","msg":"","time":"2020-07-01T18:05:57Z","var":{"timeseries":[{"labels":[{"name":"__name__","value":"scrape_series_added"},{"name":"instance","value":"localhost:9090"},{"name":"job","value":"prometheus"}],"samples":[{"timestamp":1593626752552}]}]}}
{"fields.time":"2020-07-01T18:05:57Z","ip":"10.73.172.234","latency":160996,"level":"info","method":"POST","msg":"","path":"/receive","status":200,"time":"2020-07-01T18:05:57Z","user-agent":"Prometheus/2.19.1"}
{"level":"debug","msg":"","time":"2020-07-01T18:06:01Z","var":{"timeseries":[{"labels":[{"name":"__name__","value":"node_custom_rule"},{"name":"code","value":"200"},{"name":"instance","value":"localhost:9090"},{"name":"job","value":"prometheus"}],"samples":[{"value":51,"timestamp":1593626758007}]}]}}
{"fields.time":"2020-07-01T18:06:01Z","ip":"10.73.172.234","latency":155700,"level":"info","method":"POST","msg":"","path":"/receive","status":200,"time":"2020-07-01T18:06:01Z","user-agent":"Prometheus/2.19.1"}

[palmerabollo]

Thanks, @smitra2020. We'll try to reproduce it as soon as we can. It would be really helpful if you could try with a previous version (such as 1.6.0) to know if we have introduced a bug in 1.7.0.

[jpfe-tid]

Hi @smitra2020,

I see two separate issues here.

1. Unable to send metrics from adapter into kafka

   Unless you're running prometheus-kafka-adapter with network: host, there isn't a kafka broker listening on localhost:9093.

   Solutions:

   1. Deploy a kafka broker as a docker-compose container along with prometheus-kafka-adapter.

   2. Change KAFKA_BROKER_LIST environment variable value to your docker bridge IP (you can find it with ifconfig) or if you're using macOS, then change it to docker.for.mac.localhost. Please note you should change your kafka advertiser listeners to match the new hostname.

2. Unable to parse value (NaN)

   Can you attach your prometheus server config for further investigation?

Thanks, Javier.

[palmerabollo]

@smitra2020 were you able to solve the issue?