search

confluentinc / kafka-rest

Confluent REST Proxy for Kafka
https://docs.confluent.io/current/kafka-rest/docs/index.html
Other
46 stars 647 forks source link

Property bootstrap.servers is not valid and other confusing config warnings. #425

Open josh-padnick opened 6 years ago

josh-padnick commented 6 years ago

I'm configuring REST Proxy by closely following the recommended configuration properties, however the behavior I'm seeing is not consistent with the documentation.

When I run REST Proxy, here's the log output I see:

[2018-04-12 22:02:36,394] INFO Logging initialized @1208ms (org.eclipse.jetty.util.log:186)
[2018-04-12 22:02:37,129] INFO Verifying properties (kafka.utils.VerifiableProperties:72)
[2018-04-12 22:02:37,197] WARN Property bootstrap.servers is not valid (kafka.utils.VerifiableProperties:87)
[2018-04-12 22:02:37,198] INFO Property group.id is overridden to  (kafka.utils.VerifiableProperties:72)
[2018-04-12 22:02:37,199] WARN Property host.name is not valid (kafka.utils.VerifiableProperties:87)
[2018-04-12 22:02:37,199] WARN Property id is not valid (kafka.utils.VerifiableProperties:87)
[2018-04-12 22:02:37,200] WARN Property listeners is not valid (kafka.utils.VerifiableProperties:87)
[2018-04-12 22:02:37,200] WARN Property schema.registry.url is not valid (kafka.utils.VerifiableProperties:87)
[2018-04-12 22:02:37,200] INFO Property zookeeper.connect is overridden to  (kafka.utils.VerifiableProperties:72)
[2018-04-12 22:02:37,450] INFO KafkaAvroDeserializerConfig values:
        schema.registry.url = [http://172.21.0.2:8081, http://172.21.0.4:8081, http://172.21.0.3:8081]
        auto.register.schemas = true
        max.schemas.per.subject = 1000
        specific.avro.reader = false
 (io.confluent.kafka.serializers.KafkaAvroDeserializerConfig:175)
[2018-04-12 22:02:37,466] INFO KafkaJsonDecoderConfig values:
        json.fail.unknown.properties = true
 (io.confluent.kafka.serializers.KafkaJsonDecoderConfig:175)
[2018-04-12 22:02:37,587] INFO Adding listener: http://0.0.0.0:8082 (io.confluent.rest.Application:182)
[2018-04-12 22:02:37,678] INFO jetty-9.2.22.v20170606 (org.eclipse.jetty.server.Server:327)
[2018-04-12 22:02:38,575] INFO HV000001: Hibernate Validator 5.1.3.Final (org.hibernate.validator.internal.util.Version:27)
[2018-04-12 22:02:38,816] INFO Started o.e.j.s.ServletContextHandler@172b013{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)

And here's the config file I'm using:

id=4e863eeb15357103c9d9a6eed271b622de15ed8baa7c34549bfaa9aff9dccd8a
host.name=kafka_colo_ubuntu_0
listeners=http://0.0.0.0:8082
schema.registry.url=http://172.21.0.2:8081,http://172.21.0.4:8081,http://172.21.0.3:8081
bootstrap.servers=PLAINTEXT://172.21.0.2:9092

For the bootstrap.servers property, I also tried PLAINTEXT://172.21.0.2:9092,PLAINTEXT://172.21.0.3:9092,PLAINTEXT://172.21.0.4:9092, but it has the same result (as I would expect).

But I'm confused by all the log messages that indicate properties are being overridden or that they're not valid.

As you can see in the logs, REST Proxy starts its listener successfully, but then when I attempt to do anything, I get these error messages:

[2018-04-12 21:57:34,802] WARN [Producer clientId=producer-2] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:241)
[2018-04-12 21:57:35,163] WARN [Producer clientId=producer-2] Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:241)
[2018-04-12 21:57:35,369] WARN [Producer clientId=producer-2] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:241)

I can fix the above issue by reverting back to using zookeeper.connect, so that my config file looks like this:

id=4e863eeb15357103c9d9a6eed271b622de15ed8baa7c34549bfaa9aff9dccd8a
host.name=kafka_colo_ubuntu_0
zookeeper.connect=172.21.0.2:2181,172.21.0.4:2181,172.21.0.3:2181
listeners=http://0.0.0.0:8082
schema.registry.url=http://172.21.0.2:8081,http://172.21.0.4:8081,http://172.21.0.3:8081

Then logs look fine:

[2018-04-12 22:09:44,781] INFO Logging initialized @953ms (org.eclipse.jetty.util.log:186)
[2018-04-12 22:09:45,403] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread:65)
[2018-04-12 22:09:45,417] INFO Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.ZooKeeper:100)
[2018-04-12 22:09:45,417] INFO Client environment:host.name=4e863eeb1535 (org.apache.zookeeper.ZooKeeper:100)
[2018-04-12 22:09:45,418] INFO Client environment:java.version=1.8.0_161 (org.apache.zookeeper.ZooKeeper:100)
[2018-04-12 22:09:45,418] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper:100)
[2018-04-12 22:09:45,418] INFO Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre (org.apache.zookeeper.ZooKeeper:100)

And sure enough REST Proxy works without issue. Can you help me understand what's going on here?

Update: Note that I also tried using bootstrap.servers and zookeeper.connect at the same time, and that had the same results as only using bootstrap.servers.

josh-padnick commented 6 years ago

A colleague asked what version of REST Proxy I was on, and while I was pretty sure I installed v4.0.0 and confirmed as such, I couldn't find a way to verify the version of REST Proxy that I'm actually running, even in log files. Is there an authoritative way to do this?

The best I could come up with so far is:

root@4e863eeb1535:/usr/share/java/kafka-rest# ls -al /usr/share/java/kafka-rest/ | grep kafka-rest
-rw-r--r--  1 root root  202273 Apr 11 21:32 kafka-rest-4.0.0.jar
-rw-r--r--  1 root root  202275 Mar 29 22:13 kafka-rest-4.0.1.jar
josh-padnick commented 6 years ago

I was experiencing the above issue while running a REST Proxy cluster in Docker, however I don't seem to have the same issues when running the cluster on EC2! I can't explain why this wouldn't work on Docker, since I use the same Packer image to build an AMI and a Docker Image, o I'll leave the ticket open for now.

sergeiwaigant commented 6 years ago

I have the same issue right now when installing Confluent Platform OSS 4.1.1 with Kafka REST 4.1.1.

This is my configuration for my 3 zk and 4 broker cluster with one vm for the confluent services I simplified the hostnames, in real this are the FQDN

id=1
host.name=cp3301
bootstrap.servers=cp1201:9093,cp1202:9093,cp2201:9093,cp2202:9093
zookeeper.connect=cp1101:2181,cp3101:2181,cp2101:2181/var/lib/zookeeper

listeners=http://cp3301:8082
schema.registry.url=http://cp3301:8081

#SASL additions
client.security.protocol=SASL_PLAINTEXT
client.sasl.mechanism=PLAIN

The full log of one start in the attachment.

[2018-07-04 09:44:13,139] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient:713)
[2018-07-04 09:44:13,146] INFO zookeeper state changed (SaslAuthenticated) (org.I0Itec.zkclient.ZkClient:713)
[2018-07-04 09:44:13,161] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$:31)
[2018-07-04 09:44:13,201] INFO Verifying properties (kafka.utils.VerifiableProperties:66)
[2018-07-04 09:44:13,217] WARN Property bootstrap.servers is not valid (kafka.utils.VerifiableProperties:70)
[2018-07-04 09:44:13,218] WARN Property client.sasl.mechanism is not valid (kafka.utils.VerifiableProperties:70)
[2018-07-04 09:44:13,218] WARN Property client.security.protocol is not valid (kafka.utils.VerifiableProperties:70)
[2018-07-04 09:44:13,218] INFO Property group.id is overridden to  (kafka.utils.VerifiableProperties:66)
[2018-07-04 09:44:13,219] WARN Property host.name is not valid (kafka.utils.VerifiableProperties:70)
[2018-07-04 09:44:13,219] WARN Property id is not valid (kafka.utils.VerifiableProperties:70)
[2018-07-04 09:44:13,219] WARN Property listeners is not valid (kafka.utils.VerifiableProperties:70)
[2018-07-04 09:44:13,219] WARN Property schema.registry.url is not valid (kafka.utils.VerifiableProperties:70)
[2018-07-04 09:44:13,219] INFO Property zookeeper.connect is overridden to  (kafka.utils.VerifiableProperties:66)

kafka-rest-warning.log

sibblegp commented 4 years ago

Did anyone figure this out?

lukasz-kastelik commented 4 years ago

Hi,

I'm facing the same issue. I get the messages saying that Property is not valid. How do I fix it? And why do I get this error? I'm using the newest version of the Kafka REST Proxy.

[2020-03-25 10:50:36,272] INFO Logging initialized @561ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:169) [2020-03-25 10:50:36,361] WARN DEPRECATION warning:listenersconfiguration is not configured. Falling back to the deprecatedportconfiguration. (io.confluent.rest.ApplicationServer:93) [2020-03-25 10:50:36,362] INFO Adding listener: http://0.0.0.0:8082 (io.confluent.rest.ApplicationServer:316) [2020-03-25 10:50:36,622] INFO Registered kafka:type=kafka.Log4jController MBean (io.confluent.kafka.utils.Log4jControllerRegistration$:31) [2020-03-25 10:50:36,906] INFO Verifying properties (io.confluent.kafka.utils.VerifiableProperties:66) [2020-03-25 10:50:36,955] WARN Property authentication.method is not valid (io.confluent.kafka.utils.VerifiableProperties:70) [2020-03-25 10:50:36,956] WARN Property authentication.realm is not valid (io.confluent.kafka.utils.VerifiableProperties:70) [2020-03-25 10:50:36,956] WARN Property authentication.roles is not valid (io.confluent.kafka.utils.VerifiableProperties:70) [2020-03-25 10:50:36,956] WARN Property bootstrap.servers is not valid (io.confluent.kafka.utils.VerifiableProperties:70) [2020-03-25 10:50:36,956] WARN Property debug is not valid (io.confluent.kafka.utils.VerifiableProperties:70) [2020-03-25 10:50:36,957] INFO Property group.id is overridden to (io.confluent.kafka.utils.VerifiableProperties:66) [2020-03-25 10:50:36,957] INFO Property zookeeper.connect is overridden to (io.confluent.kafka.utils.VerifiableProperties:66) [2020-03-25 10:50:36,975] INFO KafkaAvroDeserializerConfig values: bearer.auth.token = [hidden]

jpuris commented 4 years ago

Same exact happens when starting confluent-kafka-rest with following config

# Unique ID for the Confluent REST Proxy server instance
id=server-01

# The base URL for Schema Registry that should be used by the serializer
schema.registry.url=https://server-01:8081,https://server-02:8081,https://server-03.mnw.no:8081

# A list of Kafka brokers to connect to discover list of all brokers
bootstrap.servers=SSL://server-01:9093,SSL://server-02:9093,SSL://server-03:9093

# listeners
listeners=https://server-01:8082

# SSL
# REST API
ssl.enabled.protocols=TLSv1.2
ssl.keystore.location=/etc/ssl/kafka.server.keystore.jks
ssl.keystore.password=<redacted>
ssl.key.password=<redacted>

# Communication with Kafka Broker
client.security.protocol=SSL
client.ssl.enabled.protocols=TLSv1.2
client.ssl.truststore.location=/etc/ssl/kafka.server.truststore.jks
client.ssl.truststore.password=<redacted>

In the logs just after INFO Verifying properties when starting the component

[2020-09-25 15:37:39,340] INFO Verifying properties (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,401] WARN Property bootstrap.servers is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,402] WARN Property client.security.protocol is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,402] WARN Property client.ssl.enabled.protocols is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,402] WARN Property client.ssl.truststore.location is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,403] WARN Property client.ssl.truststore.password is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,404] INFO Property group.id is overridden to  (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,404] WARN Property id is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,404] WARN Property listeners is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,405] WARN Property schema.registry.url is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,405] WARN Property ssl.enabled.protocols is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,405] WARN Property ssl.key.password is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,407] WARN Property ssl.keystore.location is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,407] WARN Property ssl.keystore.password is not valid (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,407] INFO Property zookeeper.connect is overridden to  (io.confluent.kafka.utils.VerifiableProperties)
[2020-09-25 15:37:39,428] INFO KafkaAvroDeserializerConfig values:

EDIT: Confluent Community package version is5.5.1

htxryan commented 2 years ago

Any resolution to this?