Open ybyzek opened 5 years ago
A symptom of this issue may be the following error while running kafka-avro-console-producer
. The user thinks they are passing in the SR security parameters but they are not being picked up.
org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"demo","fields":[{"name":"name","type":"string"}]}
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 2]; error code: 50005
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:226)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:252)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:352)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:344)
at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:330)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:142)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:196)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:172)
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:79)
at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:181)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:54)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Ideal usage:
kafka-avro-console-producer --topic test1 --producer.config=<config> --property value.schema=<schema> --broker-list <bootstrap-server>
Workaround due to this GH issue:
kafka-avro-console-producer --topic test1 --producer.config=<config> --property value.schema=<schema> --broker-list <bootstrap-server> --property schema.registry.basic.auth.user.info=<username:password> --property schema.registry.url=<SR endpoint> --property basic.auth.credentials.source=USER_INFO
Another user in the community hit this issue today trying to connect to Confluent Cloud Schema Registry.
Just encountered this myself. Spent the better part of an hour before finding this issue describing the workaround :-/
Issue is still present in current console tools available from confluent-platform v5.5.1 on Homebrew
This issue is so annoying. And also misleading. In my case, I use kafka-avro-console-consumer
and it was throwing unknown schema for id at me, not a word about creds.
To be honest, I don't even see the point of having DEFAULT_SCHEMA_REGISTRY_URL="--property schema.registry.url=http://localhost:8081"
specified. Without it I at least see the problem right away: Missing schema registry url!
If I saw it right away I would suss out the issue with properties a lot facter.
In case this helps other users: here is a validated example of the workaround
A fix depends on https://github.com/apache/kafka/pull/12175
Configuration file specifies fields for:
But even when passing in the file via
producer.config
,kafka-avro-console-producer
does not honor these fields. It works only if they are overridden explicitly asproperty
arguments.This GH issue to enhance
kafka-avro-console-producer
andkafka-avro-console-consumer
to honor all properties configured in the configuration file passed in viaproducer.config
orconsumer.config