Multiple Kafka inputs - 7.x

[robrankin]

It appears that multiple Kafka inputs all use the last defined set of bootstrap_servers, assuming the group coordinator in the below log output is the same.

The 1st defined Kafka input is never used.

Removing either one of the defined inputs allows the remaining input to work correctly.

• Version: 7.0.0

• Operating System: CentOS Linux release 7.4.1708 (Core)

• Config File (if you have sensitive info, please remove it):

  input {
  kafka {
  bootstrap_servers => "01.servicebus.windows.net:9093"
  security_protocol => "SASL_SSL"
  sasl_mechanism => "PLAIN"
  jaas_path => "/etc/logstash/01-jaas.config"
  decorate_events => true
  group_id => "logstash-01-1"
  client_id => "logstash-01"
  topics => ["logstash"]
  id => "logstash-kafka-01"
  tags => ["01"]
  }
  kafka {
  bootstrap_servers => "02.servicebus.windows.net:9093"
  security_protocol => "SASL_SSL"
  sasl_mechanism => "PLAIN"
  jaas_path => "/etc/logstash/02-jaas.config"
  decorate_events => true
  group_id => "logstash-01-2"
  client_id => "logstash-02"
  topics => ["logstash"]
  id => "logstash-kafka-02"
  tags => ["02"]
  }
  }

• Sample Data:

  [org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-en1bdedv02ehn-0, groupId=logstash-en1bdess01-2] Discovered group coordinator 02.servicebus.windows.net:9093 (id: 2147483647 rack: null)
  [org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-en1bdess01ehn-0, groupId=logstash-en1bdess01-1] Discovered group coordinator 02.servicebus.windows.net:9093 (id: 2147483647 rack: null)

[robrankin]

In fact, "The 1st defined Kafka input is never used." doesn't appear to be correct after repeated restarting. Seems random.

[vbohata]

jaas_path is JVM global config. Can not be two different configs for the same Java process. To solve this I created following pull request: https://github.com/logstash-plugins/logstash-input-kafka/pull/313

[yaauie]

I believe this issue is resolved with documentation about the global nature of the jaas_path setting and the introduction of sasl_jaas_config.