Open spenczar opened 2 years ago
I have a very similar issue where I have a JavaScript client that can successfully tail a topic while kcat
errors with:
Avro/Schema-registry key deserialization: Invalid CP1 magic byte 57, expected 0: message not produced with Schema-Registry Avro framing: terminating
I know there's a magic byte because I'm pulling it off the raw message and I know it is encoded in Avro.
Each message must be prefixed by the schema-registry specific framing, which is one byte for magic (value 0) and then the schemaId in big-endian (4 bytes), then followed by the serialized payload (e.g., serialized avro).
This message means there was no such framing.
You can verify what the framing looks like by running kcat without a deserializer and passing the key to hexdump
or similar.
oooh, my issue was staring me in the face and I didn't notice... "key deserialization". Only my values are serialized. Changing my config to -s value=avro
fixed it. This wasn't intuitive because other clients I've used and written search for the magic byte and only decode if it exists.
I also had to pass my schema registry password unencoded where I could have sworn I read that it should be url encoded.
@edenhill You can confirm that msg.avro
in the zip I provided uses the schema-registry specific framing ("Confluent wire format"):
-> % cat msg.avro | head -c 5 | hexdump
0000000 0000 0000 0001
0000005
This is saying that I have a message with schema ID of 1. I still see the above error from kcat.
Failed to encode Avro as JSON:
suggests that deserialization worked, but it failed to convert the avro objects to JSON.
It could be that your schema uses constructs that are not supported by the avro-c version that is used by kcat.
I am trying to use kcat to read Confluent Wire Format-encoded messages. They're Avro encoded, and I have a Schema Registry. This little Python script works with confluent_kafka_python v1.7.0:
But this kcat invocation fails, even though it should be identical, and even while running on the same messages:
None of this data is sensitive, so I can upload the schema and message here too: message_and_schema.zip
That zip has a 'msg.avro' which is the Kafka message (including the 5-byte Confluent Wire Format header) and a "schema.json" which is the Avro schema used.
Be warned - this is a big and complex schema, about 40KB, and the Avro message is 580KiB of scientific simulation data.