confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
110 stars 895 forks source link

Python Dictionary to Class Handling (How/Why) #1100

Open slominskir opened 3 years ago

slominskir commented 3 years ago

Description

Is there any discussion on how to handle serde transformation from a Python Dictionary to a Class? The Consumer API for example currently accepts a callable function from_dict and is demonstrated in the examples.

This appears to preclude using a generator such as avro-gen, since it apparently simply wraps the Dictionary instead of transform it and toss it out as is done with _fromdict. Maybe avro_gen wrapping is a bad approach? Maybe classes aren't "Pythonic" anyways? Just trying to figure out how best to use the confluent_kafka_python API in my own code and figure someone has already thought about this.

How to reproduce

This is an API / Documentation question.

Checklist

None

mhowlett commented 3 years ago

avro-gen was not on our radar (seems to be new). It may be a good approach, we haven't looked at it.

We need to re-visit all of this and get Python serdes in a better state. I plan on doing this, but have competing priorities. Feel free to leave any opinions you may have, we'll take note.

slominskir commented 3 years ago

I haven't formed an opinion yet, but digging a little I found some additional information to consider:

@scottbelden is probably more knowledgeable about this.

The advantages of a class over a dict include:

The disadvantages may include:

scottbelden commented 3 years ago

If I were to re-write fastavro from scratch I would probably have it use classes (or dataclasses) instead of regular dictionaries. It would solve some of the ambiguity problems that currently exist in fastavro and could theoretically allow for faster serialization/deserialization since the classes could be generated with custom, schema specific encoders/decoders.

However, that's all a large enough change that it would almost certainly have to be in a completely new version two library and so I don't imagine fastavro ever moving away from using dictionaries for things like records.

slominskir commented 3 years ago

So when does fasteravro (fastavro2) come out? 👍

scottbelden commented 3 years ago

Whenever someone decides to write it 😛