Need for documentation on InfluxDBPusherSink and "sinks" in general

[Thomas2511]

Describe the bug I was trying to setup an InfluxDB using kafka-lag-exporter to check the lag from Kafka and I couldn't get it to work simply using the configuration provided through the documentation for InfluxDB.

To Reproduce The original configuration file looked like this:

kafka-lag-exporter {
  reporters.influxdb.endpoint = "http://influxdb"
  reporters.influxdb.port = 8086
  reporters.influxdb.database = "kafka_lag_exporter"
  reporters.influxdb.username = "admin"
  reporters.influxdb.password = "admin"
  clusters = [
    {
      name = "kafka"
      bootstrap-brokers = "kafka:19092"
    }
  ]
}

With a Kafka + Zookeeper cluster with InfluxDB and Grafana for data reporting. Everything set up in containers.

The configuration file seemed to tick every box regarding the configuration elements proposed in the README file.

With this configuration when I launch my application and check the kafka-lag-exporter logs, this is the configuration section report:

[...]
2021-03-03 13:27:12,583 INFO  c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Starting Kafka Lag Exporter with configuration: 

Poll interval: 30 seconds
Lookup table size: 60
Metrics whitelist: [.*]
Admin client consumer group id: kafkalagexporter
Kafka client timeout: 10 seconds

Prometheus:
  port: 8000

[...]

And I have no data in my InfluxDB.

But when I change my configuration file (after reading through the code and finding about the InfluxDBPusherSink) to:

kafka-lag-exporter {
  sinks = [
    "InfluxDBPusherSink"
  ]
  reporters.influxdb.endpoint = "http://influxdb"
  reporters.influxdb.port = 8086
  reporters.influxdb.database = "kafka_lag_exporter"
  reporters.influxdb.username = "admin"
  reporters.influxdb.password = "admin"
  clusters = [
    {
      name = "kafka"
      bootstrap-brokers = "kafka:19092"
    }
  ]
}

This time in the kafka-log-exporter logs, my configuration seems good, and I now have data in my InfluxDB

[...]
2021-03-03 13:48:46,152 INFO  c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Starting Kafka Lag Exporter with configuration: 

Poll interval: 30 seconds
Lookup table size: 60
Metrics whitelist: [.*]
Admin client consumer group id: kafkalagexporter
Kafka client timeout: 10 seconds

InfluxDB:
  endPoint: http://influxdb
  port: 8086
  database: kafka_lag_exporter
  username: admin
  async: true

[...]

Environment

• Version : kafka-lag-exporter:0.6.6
• Version of Apache Kafka cluster : zookeeper:3.4.9, confluentinc/cp-kafka:5.5.1
• Run Standalone

Additional context This is not a bug 'per se' but a clarification in the documentation would have saved me a lot of time.

[yehonatanz]

This is the same problem I experienced in #195

[bapBardas]

I've been fighting hours to have the InfluxDB exporter work properly, thank you very much for this clarification ! I agree this must be added to the official doc, it would save a lot of time to future-me :).