search

confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
78 stars 890 forks source link

Support for multiple schemas in one topic #477

Open golfpreben opened 5 years ago

golfpreben commented 5 years ago

Description

Support for multiple schemas in one topic for JAVA avro serializer was added in pull request: https://github.com/confluentinc/schema-registry/pull/680 Any chance this could be supported from python too?

How to reproduce

In tried to add value.subject.name.strategy to the config of the avro producer

avroProducer = AvroProducer({
'bootstrap.servers': '127.0.0.1:9092',
'schema.registry.url': 'http://127.0.0.1:8081',
'value.subject.name.strategy': 'io.confluent.kafka.serializers.subject.TopicNameStrategy'
}, default_key_schema=key_schema, default_value_schema=value_schema)

and got the following errror:

cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="No such configuration property: "value.subject.name.strategy""}

Checklist

Please provide the following information:

OneCricketeer commented 5 years ago

See #442 & #401

cschaible commented 5 years ago

Are there any news regarding this issue? Both referenced issues (#442, #401) are closed. The issue #401 references #502, where nothing happens since months. As written in #560 the issue #502 might be included in the next release but (when) will this happen? Maybe @mhowlett can provide information, since he's reviewing the pull-request #502. Thanks in advance!

enima2684 commented 4 years ago

Is there any workaround we can use while waiting for the new release ? Many thanks

mhowlett commented 4 years ago

AvroProducer is being replaced by the new avro serdes which support naming strategies: https://github.com/confluentinc/confluent-kafka-python/blob/master/confluent_kafka/schema_registry/avro.py

we don't yet support this: https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/

enima2684 commented 4 years ago

Thanks for the quick reply and good to know this functionality is already implemented.

However, it is not yet clear for me where/how to enter this configuration parameter ?

For the moment, I create a Producer object as follows : 

        producer = AvroProducer(
            {
              'bootstrap.servers': "myBootstrapServers",
              'schema.registry.url': "mySchemaRegistryUrl",
              'enable.idempotence': 'true'
            },
            default_key_schema=self.get_key_schema(),
            default_value_schema=self.get_value_schema()     
 )

Is it something to add in the producer config ?

amineouba commented 2 years ago

did you find a solution ?