search

edenhill / kcat

Generic command line non-JVM Apache Kafka producer and consumer
Other
5.45k stars 484 forks source link

AVRO Record types without fields can't be consumed #305

Open PHameete opened 3 years ago

PHameete commented 3 years ago

When consuming a topic that contains an AVRO type that has no fields, the following error is produced:

% ERROR: Failed to format message in event-rule-engine-eventTriggerStateStore-changelog [2] at offset 18734304: Avro/Schema-registry message deserialization: Record type must have at least one field : terminating

The AVRO specification does not indicate that such types are not allowed. You can create them using the official Java AVRO library, and the Confluent Java AVRO serdes can also serialize/deserialize them without issues.

Is it possible to remove this check: https://github.com/confluentinc/avro-c-packaging/blob/master/src/schema.c#L922 so kafkacat can also consume this type of record?

OneCricketeer commented 3 years ago

Can you clarify the purpose of a record without fields? Perhaps you'd be better off just sending a boolean primitive instead if you simply want some message?

does not indicate that such types are not allowed

How so? The docs say "fields: a JSON array, listing fields (required)"

PHameete commented 3 years ago

Can you clarify the purpose of a record without fields? Perhaps you'd be better off just sending a boolean primitive instead if you simply want some message?

For forward compatibility I find it easier to have a record without fields and then add (optional) fields to it later than to replace a boolean with a record.

How so? The docs say "fields: a JSON array, listing fields (required)"

Yes the "fields" array must exist, but it is allowed to be empty. At least that is my interpretation, and the offical AVRO library will also accept and generate classes for such a schema.

bashtanov commented 3 years ago

I'd find it useful too, as we have empty record types as well. Other avro-aware software works well with it, so, in case one finds the spec ambiguous, it's allowed de-facto.

r-glyde commented 2 years ago

It looks like a fix for this was merged into avro-c a while ago (apache/avro#159) and is in the release-1.11.0 tag. https://github.com/confluentinc/avro-c-packaging doesn't have the same fix but I'm not really sure what the purpose of that repo is as it just looks like a copy and paste of an old version of the apache/avro c implementation?

Would we be able to just update the version being used here? https://github.com/edenhill/kcat/blob/7a61200247732ff057724cb7ed56411b6a5797e4/bootstrap.sh#L147

We generally build this using the bootstrap script but not sure if this would fix the problem for everyone (haven't tested for ourselves yet either) or if there is somewhere else that would need to be updated?