search

confluentinc / examples

Apache Kafka and Confluent Platform examples and demos
Apache License 2.0
1.91k stars 1.12k forks source link

Examples Should Avoid Using Schema Registry for Keys #817

Open chuck-confluent opened 4 years ago

chuck-confluent commented 4 years ago

I don't think it's commonly understood by new learners that using Schema Registry for keys is bad practice (see this internal confluent slack convo ). For example, if you reorder fields, it will break partitioning. Also protobuf and json schemas don't serialize deterministically, so using those formats for keys will also break partitioning.

This python example uses Avro for the key Name and it probably shouldn't.

Mcfloy commented 3 years ago

Hello Chuck,

We don't have access to the confluent slack convo, is there a public documentation about it to prevent the users to keep this bad practice ?

I'm also surprised that some tools like Conduktor don't make a friendly warning about this practice.

chuck-confluent commented 3 years ago

I'll make a forum thread on this over in forum.confluent.io as well as raise an issue with our docs team. I'll keep you posted!

chuck-confluent commented 3 years ago

Made forum post: https://forum.confluent.io/t/partitioning-gotchas-dont-use-avro-json-or-protobuf-for-keys-and-be-aware-of-client-hashing-differences/2718

I also filed an issue with the docs team to be more opinionated about these things