search

confluentinc / schema-registry

Confluent Schema Registry for Kafka
https://docs.confluent.io/current/schema-registry/docs/index.html
Other
2.22k stars 1.11k forks source link

Command line producer/consumer should honor all fields in producer.config and consumer.config #1052

Open ybyzek opened 5 years ago

ybyzek commented 5 years ago

Configuration file specifies fields for:

basic.auth.credentials.source=
schema.registry.basic.auth.user.info=
schema.registry.url=

But even when passing in the file via producer.config , kafka-avro-console-producer does not honor these fields. It works only if they are overridden explicitly as property arguments.

This GH issue to enhance kafka-avro-console-producer and kafka-avro-console-consumer to honor all properties configured in the configuration file passed in via producer.config or consumer.config

ybyzek commented 5 years ago

A symptom of this issue may be the following error while running kafka-avro-console-producer. The user thinks they are passing in the SR security parameters but they are not being picked up.

org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"demo","fields":[{"name":"name","type":"string"}]}
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 2]; error code: 50005
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:226)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:252)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:352)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:344)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:330)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:142)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:196)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:172)
    at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:79)
    at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:181)
    at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:54)
    at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
ybyzek commented 5 years ago

Ideal usage:

kafka-avro-console-producer --topic test1 --producer.config=<config> --property value.schema=<schema> --broker-list <bootstrap-server>

Workaround due to this GH issue:

kafka-avro-console-producer --topic test1 --producer.config=<config> --property value.schema=<schema> --broker-list <bootstrap-server> --property schema.registry.basic.auth.user.info=<username:password> --property schema.registry.url=<SR endpoint> --property basic.auth.credentials.source=USER_INFO

ybyzek commented 4 years ago

Another user in the community hit this issue today trying to connect to Confluent Cloud Schema Registry.

kpmeen commented 4 years ago

Just encountered this myself. Spent the better part of an hour before finding this issue describing the workaround :-/

cbcoutinho commented 4 years ago

Issue is still present in current console tools available from confluent-platform v5.5.1 on Homebrew

yuranos commented 4 years ago

This issue is so annoying. And also misleading. In my case, I use kafka-avro-console-consumer and it was throwing unknown schema for id at me, not a word about creds. To be honest, I don't even see the point of having DEFAULT_SCHEMA_REGISTRY_URL="--property schema.registry.url=http://localhost:8081" specified. Without it I at least see the problem right away: Missing schema registry url! If I saw it right away I would suss out the issue with properties a lot facter.

ybyzek commented 3 years ago

In case this helps other users: here is a validated example of the workaround

https://github.com/confluentinc/examples/blob/latest/clients/cloud/kafka-commands/kafka-commands-ccsr.sh

rayokota commented 2 years ago

A fix depends on https://github.com/apache/kafka/pull/12175