search

confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
81 stars 890 forks source link

Does python binding support exactly once semantics? #357

Closed hrchu closed 3 years ago

hrchu commented 6 years ago

Does python binding support exactly once semantics in producer/consumer side in Kafka 0.11.0.0? I don't see any document mentioned it.

edenhill commented 6 years ago

It does not, we're looking to add EoS support in Q3

hrchu commented 6 years ago

Related question: Is transactional producer/consumer available now or in Q3? ( I will like to do exactly-once delivery when processing data between Kafka topics)

edenhill commented 6 years ago

No, idempotent producer, transactions and EoS will be released together in Q3

lcary commented 6 years ago

@edenhill Do you have an update or an ETA for the above-mentioned release? v0.11.5 didn't appear to have any related changes. Thanks in advance.

edenhill commented 6 years ago

@lcary We're currently working on the idempotent producer in librdkafka, which will be released in a month or so, including support in the Python, Go and .NET clients. It provides exactly once and ordering producer guarantees.

Transaction support in the consumer (which boils down to filtering out uncommitted messages) will come soon after.

As for Producer Transaction support, we don't currently have a scheduled release date.

This issue tracks demand for the different EOS features, please cast your vote there: https://github.com/edenhill/librdkafka/issues/1308

lcary commented 6 years ago

@edenhill thanks so much for the link and update. I just cast my votes.

tarunagra commented 5 years ago

Wondering, if idempotent support for Python Producer is released?

edenhill commented 5 years ago

Target is within two weeks, all systems are go.

russellpierce commented 5 years ago

👋 Looking for an update on this issue.

mhowlett commented 5 years ago

idempotent producer is included in 1.0, which is now released. transactions are not here yet, but we have begun work on the first part of that (consumer side only) which will be done this quarter.

hrchu commented 5 years ago

@mhowlett any update on this?

mhowlett commented 5 years ago

support in the consumer to ignore aborted transactions is almost merged and a release containing this is currently on track for a few weeks. work on the producer side of transactions hasn't begun yet.

bretlowery commented 5 years ago

What's the current schedule for full transaction support? Will producer side be done this q?

mhowlett commented 5 years ago

this quarter is too optimistic, but it's actively being worked on.

alexpirine commented 4 years ago

@mhowlett I am sorry, I didn't comment on the right project (https://github.com/confluentinc/confluent-kafka-dotnet/issues/1091#issuecomment-590960136). I was wondering about the status of the transactions integration for the Python library…

There are some other implementations (kafka-python and aiokafka) and we wonder if we should wait for transactions support or switch to one of these.

Thank you

mhowlett commented 4 years ago

I don't think those libraries have support for either idempotent producer or transactions. v1.4.0 of this client is not far off and will include support for transactions (it already provides idempotent producer).

lsorber commented 4 years ago

Now that v1.4.0 has been released, is this issue resolved?

Would someone be so kind as to update the docs with details about EOS?

edenhill commented 3 years ago

EOS is now fully supported