linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.77k stars 595 forks source link

Connecting SASL_SSL Kafka Cluster #583

Open tommyJimmy87 opened 5 years ago

tommyJimmy87 commented 5 years ago

I'm using Security Protocol set to "SASL_SSL" which means I need to specify all the JKS and JAAS in order to connect to the Cluster. I'm providing those information through the property file, but I found out that when the KafkaAdminClient runs it gets wrong SSL configuration because of this :

// Configure SSL configs (if security protocol is SSL)
      if (securityProtocol.equals(SecurityProtocol.SSL.name)) {
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTMANAGER_ALGORITHM_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYMANAGER_ALGORITHM_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYSTORE_TYPE_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG);
        setStringConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_SECURE_RANDOM_IMPLEMENTATION_CONFIG);
        setPasswordConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG);
        setPasswordConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_KEY_PASSWORD_CONFIG);
        setPasswordConfigIfExists(configs, adminClientConfigs, SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG);
      }

I'm not using SSL but SASL_SSL and I still need those properties to be set in the KafkaClientAdmin configurations.

The class where i found that code portion is KafkaCruiseControlUtils.java.

This is the error I'm facing :

org.apache.kafka.common.KafkaException: Failed create new KafkaAdminClient
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:370)
    at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:62)
    at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.createAdminClient(KafkaCruiseControlUtils.java:176)
    at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.describeLogDirs(KafkaCruiseControlUtils.java:161)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.populateKafkaBrokerLogDirState(KafkaClusterState.java:323)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJsonStructure(KafkaClusterState.java:252)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJSONString(KafkaClusterState.java:85)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.discardIrrelevantAndCacheRelevant(KafkaClusterState.java:454)
    at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.discardIrrelevantResponse(AbstractCruiseControlResponse.java:43)
    at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.writeSuccessResponse(AbstractCruiseControlResponse.java:34)
    at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServletUtils.syncRequest(KafkaCruiseControlServletUtils.java:118)
    at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doGet(KafkaCruiseControlServlet.java:159)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:564)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
    at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
    at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:153)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:140)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:65)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:88)
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:346)
    ... 39 more
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
    at org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:303)
    at org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:92)
    at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:63)
    at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:114)
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:142)
    ... 43 more

Tag Cruise Control version is 2.0.36. Kafka Version is 1.1.x.

efeg commented 5 years ago

@tommyJimmy87 Thanks for reporting the issue! Cruise Control does not support SASL_SSL, yet -- i.e. it currently supports SSL and Plaintext. Would you be interested in submitting a patch to add SASL_SSL support?

becketqin commented 5 years ago

@tommyJimmy87 Are you trying to use Kerberos for authentication? If so, one of the following configs is needed:

  1. sasl.kerberos.service.name
  2. sasl.jaas.config

It would be more helpful if you can share the configurations you are using.

tommyJimmy87 commented 5 years ago

@becketqin We are not using kerberos... we are just using SASL_SSL with the username and password in the JAAS.. but apparently as @efeg said is not supported.

tommyJimmy87 commented 5 years ago

@efeg I can take a look how complicate is to patch it.

becketqin commented 5 years ago

@tommyJimmy87 Hmm, CC should support all the mechanisms support by Kafka itself. Have you set sasl.mechanism in your config? By default it is going to be GSSAPI and kerberos will be used. With SASL_SSL, change the that config to PLAIN and have a try.

becketqin commented 5 years ago

Also, please make sure you are using the org.apache.kafka.common.security.plain.PlainLoginModule with your username and password in the jaas config file.

tommyJimmy87 commented 5 years ago

@becketqin Thx, here my config file :

broker.metric.sample.store.topic=__KafkaCruiseControlModelTrainingSamples
sample.store.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore
ssl.key.password=xxxxxxxxx (password is present)
min.samples.per.broker.metrics.window=1
num.partition.metrics.windows=1
ssl.keystore.password=xxxxxxxxx (password is present)
sample.store.topic.replication.factor=2
ssl.keystore.location=/etc/kafka/secrets/jks/kafka.keystore.jks
capacity.config.file=/etc/kafka-cruise-control/config/capacity.json
webserver.ui.diskpath=/app/cruise-control-ui/dist/
zookeeper.connect=ZKIP:2181/devkafka
partition.metrics.window.ms=300000
metric.sampler.partition.assignor.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.DefaultMetricSamplerPartitionAssignor
ssl.truststore.location=/etc/kafka/secrets/jks/kafka.truststore.jks
ssl.client.auth=true
partition.metric.sample.store.topic=__KafkaCruiseControlPartitionMetricSamples
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
producer.security.protocol=SASL_SSL
client.id=kafka-cruise-control
broker.metrics.window.ms=300000
default.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal
producer.sasl.mechanism=PLAIN
bootstrap.servers=SASL_SSL://myIP:9094
cluster.configs.file=/etc/kafka-cruise-control/config/clusterConfigs.json
num.sample.loading.threads=8
num.broker.metrics.windows=20
ssl.truststore.password=xxxxxxxxx (password is present)
ssl.endpoint.identification.algorithm=
metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler
consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_SSL
num.metric.fetcher=1
auto.create.topics.enable=true
metric.sampling.interval.ms=120000
metric.reporter.topic.pattern=__CruiseControlMetrics

And also I'm providing my JAAS file path in the KAFKA_OPTS env variable that should be added when CC runs. We are using the same JAAS file for several application.

tommyJimmy87 commented 5 years ago

@becketqin The problem is just the KafkaAdminClient Configs, the ConsumerConf for example are created with the right values (jks.. password and jaas are present in the ConsumerConf but not in the KafkaAdminClient).

tommyJimmy87 commented 5 years ago

@becketqin Uhm.. with the tag 0.1.39 is working actually.. earlier I was using 2.0.36 Tag... what's the different ? Master branch should be at the 0.1.39 right ? I guess it was a little bit confusing.

becketqin commented 5 years ago

@tommyJimmy87 Thanks for the update. This sounds like a bug in 2.0.36. But I could be wrong. @efeg would be more familiar with the diff between those two versions.

efeg commented 5 years ago

@tommyJimmy87 The reason why the tag 0.1.39 is working, but 2.0.36 is not is because former versions (i.e. 0.1.*) do not call describeLogDirs to show logDirs in the response of kafka_cluster_state (i.e. the endpoint that your request is sent to). The describeLogDirs creates and uses a KafkaAdminClient to return the corresponding response (supported for Kafka: 1.1.0+ with Cruise Control: 2.*).

To support SASL_SSL in 2.*, I believe there are (at least) two places to update (and test) -- i.e. you may search for the string if (securityProtocol.equals(SecurityProtocol.SSL.name)) to find the exact locations:

  1. KafkaCruiseControlUtils#parseAdminClientConfigs
  2. CruiseControlMetricsUtils#addSslConfigs(Properties, CruiseControlMetricsReporterConfig).

^ So I suspect that all we need would be:

else if (securityProtocol.equals(SecurityProtocol.SASL_SSL.name)) {
  setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.SASL_JAAS_CONFIG);
  setPasswordConfigIfExists(configs, adminClientConfigs, SaslConfigs.*); // Needs update for any other relevant password configs
  setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.*); // Needs update for any other relevant string configs.
}
tommyJimmy87 commented 5 years ago

@efeg Ok as I thought. If you want I can contribute on this.

efeg commented 5 years ago

@tommyJimmy87 Sure, we appreciate contributions!

Sugaroverdose commented 5 years ago

I've encountered the same issue with SASL_PLAINTEXT and sasl.mechanism=SCRAM-SHA-256: Cruise Control does connects to the cluster but right after that it tries to start AdminClient which does not inherit client.id, jaas.config and sasl.mechanism

UPD: managed to fix this by adding import org.apache.kafka.common.config.SaslConfigs; and setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.SASL_MECHANISM); into KafkaCruiseControlUtils.java i guess the same should be added into CruiseControlMetricsUtils.java

efeg commented 5 years ago

@Sugaroverdose Sounds great! Would you consider creating a PR with the patch?

Sugaroverdose commented 5 years ago

@efeg i'll try to it seems like that CruiseControlMetricsUtils.java does not inherit securityProtocol, so idk how it works with something different than plaintext

Sugaroverdose commented 5 years ago

@tommyJimmy87 you may try latest migrate_to_kafka_2_0 branch, it should work in your case now

poyyi commented 5 years ago

I'm novice in adopting to cruise-contol using 2.029,trying with Sasl_ssl going by above discussion it looks I'm may not be able use it sasl_ssl could you suggest which release shld I pick .. Thank u

tommyJimmy87 commented 5 years ago

@poyyi What is your Cruise Control Metric Jar configuration?

poyyi commented 5 years ago

@tommy it is cruise-control-metrics-reporter-0.1.0-SNAPSHOT.jar META-INF/ META-INF/MANIFEST.MF com/ com/linkedin/ com/linkedin/kafka/ com/linkedin/kafka/cruisecontrol/ com/linkedin/kafka/cruisecontrol/metricsreporter/ com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter$1.class com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/ com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor$Context.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric$MetricClassId.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType$MetricScope.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/BrokerMetric.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde$1.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricsUtils.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/PartitionMetric.class com/linkedin/kafka/cruisecontrol/metricsreporter/metric/TopicMetric.class com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporterConfig.class com/linkedin/kafka/cruisecontrol/metricsreporter/exception/

on the kafka servers.. "2024" is sasl_ssl port metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter cruise.control.metrics.reporter.bootstrap.servers=0.0.0.0:2024 auto.create.topics.enable=true

CC property file bootstrap.servers=BK1-IP:2024,BK2-IP:2024,BK3-IP:2024,BK4-IP:2024,BK5-IP:2024 zookeeper.connect=ZK1-IP:2181,ZK2-IP:2181,ZK3-IP:2181 sample.store.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore ssl.key.password=XXXXX@123 ssl.keystore.password=XXXXX@123 ssl.keystore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks ssl.truststore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks ssl.client.auth=true sasl.mechanism=PLAIN security.protocol=SASL_SSL producer.security.protocol=SASL_SSL producer.sasl.mechanism=PLAIN ssl.truststore.password=XXXXX@123 ssl.endpoint.identification.algorithm= consumer.sasl.mechanism=PLAIN consumer.security.protocol=SASL_SSL

tommyJimmy87 commented 5 years ago

I guess you are missing all the Cruise Control configs for Sasl SSL.

Try with these also :


KAFKA_METRIC_REPORTERS: "com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SECURITY_PROTOCOL: "SASL_SSL"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SASL_MECHANISM: "PLAIN"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_CLIENT_AUTH: "true"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_KEYSTORE_LOCATION: '/path/to/kafka.server.keystore.jks'
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_TRUSTSTORE_LOCATION: '/path/to/kafka.server.truststore.jks'

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SASL_MECHANISM_INTER_BROKER_PROTOCOL: "{{.kafkaSaslMechanismInterBrokerProtocol}}"
  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SASL_ENABLED_MECHANISMS: "{{.kafkaSaslEnabledMechanisms}}"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:SASL_SSL,CLIENT:SASL_SSL,BROKER:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_SSL:SASL_SSL"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:SSL,CLIENT:SSL,BROKER:SSL,PLAINTEXT:PLAINTEXT,SSL:SSL"

  KAFKA_CRUISE_CONTROL_METRICS_REPORTER_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,BROKER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL"
 KAFKA_CRUISE_CONTROL_METRICS_REPORTER_ADVERTISED_LISTENERS={advertised_listeners_list}

KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_TRUSTSTORE_PASSWORD={trustStorePass}
KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_KEYSTORE_PASSWORD={keyStorePass}
KAFKA_CRUISE_CONTROL_METRICS_REPORTER_SSL_KEY_PASSWORD={keyPass}
KAFKA_CRUISE_CONTROL_METRICS_REPORTER_BOOTSTRAP_SERVERS={yoursBootstrapServer}
poyyi commented 5 years ago

@jimmy sorry could not try immly on SASL_SSL (but tired with SSL which i run into some other issue described below ).. i have few queries on your suggestions for SASL_SSL

1)Above mentioned properties shld go into server.properties on brokers? 2) shld they be given exactly like how you gave above KAFKA_METRIC_REPORTERS: ?, currently i have it as below in my server.properties metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter cruise.control.metrics.reporter.security.protocol=SSL

PL note version I'm using: Cruise control version: Cruise-control-2.0.42 Kafka :kafka_2.12-0.11.0.0 (Kafka version : 0.11.0.1)

I ran with SSL pl find my setting and Exception I'm getting for

Failures "GET /kafkacruisecontrol/kafka_cluster_state?json=true HTTP/1.1" 500 4462 79

Success "GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1" 200 2637 8 "GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1" 200 61 2 "GET /kafkacruisecontrol/state?substates=MONITOR&verbose=true&json=true HTTP/1.1" 200 288 1 "GET /kafkacruisecontrol/state?substates=ANALYZER&verbose=true&json=true HTTP/1.1" 200 2866 2 "GET /kafkacruisecontrol/load?allow_capacity_estimation=true&json=true HTTP/1.1" 200 3074 2 "GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1" 200 3075 2 "GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1" 200 61 1 "GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1" 200 3275 2

Exception: GET request '/kafka_cluster_state' due to: 'Failed to populate broker logDir state.'. (com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet) java.lang.RuntimeException: Failed to populate broker logDir state. at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJSONString(KafkaClusterState.java:90) at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.discardIrrelevantAndCacheRelevant(KafkaClusterState.java:456) at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.discardIrrelevantResponse(AbstractCruiseControlResponse.java:49) Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support DESCRIBE_LOG_DIRS ---------------Cruisecontrol.properties Setting---- Cruise control.properties security.protocol=SSL ssl.truststore.type=JKS ssl.truststore.location=/opt/app/workload/cruise-control-2.0.42/config/certs/kafka.server.truststore.jks ssl.truststore.password=pxxxxxxxx@123 ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="uxxxxxx" password="pxxxxxxxxx";

---------Broker server.properties---------------- listeners=PLAINTEXT://bk1:2021,SASL_PLAINTEXT://bk1:2022,SSL://bk1:2023,SASL_SSL://bk1:2024 ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 security.inter.broker.protocol=SASL_SSL sasl.mechanism.inter.broker.protocol=PLAIN sasl.enabled.mechanisms=PLAIN super.users=User:mxxxx allow.everyone.if.no.acl.found=true authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

advertised.listeners=PLAINTEXT://bk1:2021,SASL_PLAINTEXT://bk1:2022,SSL://bk1:2023,SASL_SSL://bk1:2024 ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

############################# Log Basics #############################

log.dirs=/opt/app/workload/enabler/kafka_2.12-0.11.0.0/kafka-logs

Cruise control setting

group.initial.rebalance.delay.ms=0 metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter cruise.control.metrics.reporter.security.protocol=SSL cruise.control.metrics.reporter.bootstrap.servers=0.0.0.0:2023 auto.create.topics.enable=true cruise.control.metrics.reporter.ssl.truststore.location=/opt/app/workload/kafka_2.12-0.11.0.0/config/certs/kafka.server.truststore.jks cruise.control.metrics.reporter.ssl.truststore.password=pxxxxxxxx@123 cruise.control.metrics.reporter.ssl.keystore.location=/opt/app/workload/kafka_2.12-0.11.0.0/config/certs/kafka.server.keystore.jks cruise.control.metrics.reporter.ssl.keystore.password=pxxxxxxxx@123 cruise.control.metrics.reporter.ssl.key.password=pxxxxxxxx@123 broker.rack=Rack1

poyyi commented 5 years ago

@tommyJimmy87 ...did you get chance to look at my query

ishita07-danaher commented 4 years ago

I am facing the same issue. Can you please update on this ?

Arullaldivakar commented 3 years ago

Does cruise control support TLS SASL/Scram enabled Kafka secure cluster?

HirossxD commented 4 months ago

I have the same issue using SASL_PLAINTEXT, CC not working on kerberized kafka cluster,

i am using older commit of this repo: "repo_commit": "6e8f2ef0209903dc1036857edff1e1020058f02e"

seems like AdminClientConfig does not fetch/load sasl.kereberos.service.name , it is null even if specified in cruise_control_jaas.conf and cruisecontrol.properties files (or kafka_opts)

[2024-07-23 04:29:52,734] INFO COMMIT INFO: 2.5.126---6e8f2ef0209903dc1036857edff1e1020058f02e (com.linkedin.kafka.cruisecontrol.KafkaCruiseControl)
[2024-07-23 04:29:52,744] INFO AdminClientConfig values:
        bootstrap.servers = [poc-brkfk-1:9092, poc-brkfk-2:9092, poc-brkfk-3:9092]
        client.dns.lookup = use_all_dns_ips
        client.id =
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 50
        reconnect.backoff.ms = 50
        request.timeout.ms = 180000
        retries = 2147483647
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = [hidden]
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.connect.timeout.ms = null
        sasl.login.read.timeout.ms = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.login.retry.backoff.max.ms = 10000
        sasl.login.retry.backoff.ms = 100
        sasl.mechanism = GSSAPI
        sasl.oauthbearer.clock.skew.seconds = 30
        sasl.oauthbearer.expected.audience = null
        sasl.oauthbearer.expected.issuer = null
        sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
        sasl.oauthbearer.jwks.endpoint.url = null
        sasl.oauthbearer.scope.claim.name = scope
        sasl.oauthbearer.sub.claim.name = sub
        sasl.oauthbearer.token.endpoint.url = null
        security.protocol = SASL_PLAINTEXT
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
 (org.apache.kafka.clients.admin.AdminClientConfig)
[2024-07-23 04:29:52,781] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2024-07-23 04:29:52,783] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2024-07-23 04:29:52,783] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2024-07-23 04:29:52,783] ERROR Uncaught exception on thread Thread[main,5,main] (com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain)
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:538) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:143) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49) ~[kafka-clients-3.1.0.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.createAdminClient(KafkaCruiseControlUtils.java:582) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.<init>(KafkaCruiseControl.java:120) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.async.AsyncKafkaCruiseControl.<init>(AsyncKafkaCruiseControl.java:34) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlApp.<init>(KafkaCruiseControlApp.java:36) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlServletApp.<init>(KafkaCruiseControlServletApp.java:32) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.getCruiseControlApp(KafkaCruiseControlUtils.java:920) ~[cruise-control-2.5.126.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain.main(KafkaCruiseControlMain.java:37) ~[cruise-control-2.5.126.jar:?]
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:509) ~[kafka-clients-3.1.0.jar:?]
        ... 9 more
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
        at org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:303) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:92) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:61) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kafka-clients-3.1.0.jar:?]
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:509) ~[kafka-clients-3.1.0.jar:?]
        ... 9 more

i will try same procedure on the latest commit, will see if issue persist.

HirossxD commented 4 months ago

switched to commit f23332a2ec4f8f3f0d69cd6e6f3b7aa9e33fb873 and cruisecontrol started without an issue..

here are some necessary configuration of cruisecontrol.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

cruise.control.metrics.reporter.security.protocol=SASL_PLAINTEXT
cruise.control.metrics.reporter.sasl.mechanism=GSSAPI
cruise.control.metrics.reporter.sasl.kerberos.service.name=kafka

# Kerberos principal and keytab for Cruise Control
cruise.control.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
    useKeyTab=true \
    storeKey=true \
    keyTab="/etc/security/keytabs/kafka.service.keytab" \
    principal="kafka/poc-cc-vm@KAFKA.POC.LOCAL";

# Kerberos configuration file location
java.security.auth.login.config=/opt/cruise-control/config/cruise_control_jaas.conf
java.security.krb5.conf=/etc/krb5.conf

but cruisecontrol logs says

 INFO Collected 1 broker metric samples for 1 brokers. (com.linkedin.kafka.cruisecontrol.monitor.sampling.SamplingFetcher)

i have just setup cc parameters only on one broker yet.

current broker configuration lines related to CC and kerberos are

metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.topic.auto.create=true
cruise.control.metrics.topic.num.partitions=1
cruise.control.metrics.topic.replication.factor=1

cruise.control.metrics.reporter.bootstrap.servers=poc-brkfk-1:9092

cruise.control.metrics.reporter.security.protocol=SASL_PLAINTEXT
cruise.control.metrics.reporter.sasl.mechanism=GSSAPI

cruise.control.metrics.reporter.sasl.kerberos.service.name=kafka

# Kerberos principal and keytab for Cruise Control
cruise.control.metrics.reporter.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
storeKey=true \
keyTab="/etc/security/keytabs/kafka.service.keytab" \
principal="kafka/poc-brkfk-1@KAFKA.POC.LOCAL";

so it is functional i guess.