mesos / kafka

Apache Kafka on Apache Mesos
Apache License 2.0
414 stars 140 forks source link

Add discoveryInfo section for Broker Task #187

Closed tpolekhin closed 4 years ago

tpolekhin commented 8 years ago

Mesos provides a mechanism for service discovery applications to get the information for specific tasks, for example ports that are used and protocols. Right now kafka framework doesn't use this. I would be really helpful to have this information per broker added, especially now when Kafka 0.9.0.0 supports SSL on a different port.

Example: broker-1 with plaintext for broker-to-broker communication and SSL for producers and consumers , and JMX endpoint for monitoring Kafka with some app inside the cluster

discoveryInfo {
  name: broker-1,
  environment: prod,
  location: us,
  version: 0.9.0.0,
  ports: [{
    name: plaintext,
    number: 9092,
    protocol: tcp,
    visibility: FRAMEWORK  // only for internal framework use
  },{
    name: ssl,
    number: 9093,
    protocol: tcp,
    visibility: EXTERNAL  // for secure message producing and consuming
  },{
    name: jmx,
    number: 9010,
    protocol: tcp,
    visibility: CLUSTER  // for monitoring kafka from inside the cluster
  }]
}
joestein commented 8 years ago

This would be good to-do when we move to the new HTTP API and also move to supporting new Mesos features too. This should be near top of lists to-do with that imho.

tpolekhin commented 8 years ago

Also, it will be very helpful to have ADVERTISED_PORT from broker config to go into discoveryInfo as well, so we can build some proxying on top of that info.