search

redpanda-data / console

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
https://redpanda.com
3.81k stars 348 forks source link

Transaction support #122

Open larsroholm opened 4 years ago

larsroholm commented 4 years ago

It would be nice to have support for transactions, so that you can see if a given record was part of a transaction or not, and what the state of the transaction is. Being able to see the transaction id would be a bonus (not to be confused with the transactional id of the producer!).

The information is available in the __transaction_state topic, which is created the first time a producer starts using transactions.

As it stands now, we have no way of telling whether or not a record will be consumed by a consumer running with isolation level ReadComitted.

The information could be displayed in a new column, or we could get an option to filter uncomitted records, or both.

weeco commented 4 years ago

Do you happen to know whether Kafka has an API for this as well? I assume otherwise we'd need to consume the topic on each instance and keep a lot information in RAM (or another persistence) available.

Similar to consumer group offsets the Kafka brokers offer an API to get this information straight from the Brokers.

larsroholm commented 4 years ago

I am not aware of any API that allows you to read that information. We use confluent-kafka-dotnet, which in turn uses librdkafka, and I can't find anything there.

weeco commented 3 years ago

This will be partially fixed with https://github.com/cloudhut/kowl/pull/154 . At least the part that shows whether a message was part of a transaction or not. I'm not sure whether we could track all transaction states as this would make Kowl stateful no? As far as I got that would require consuming the whole transaction state topic?

larsroholm commented 3 years ago

Yup, you would need to consume that topic.

weeco commented 3 years ago

Seems like a DescribeTransactions API will soon be added: https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions . Worth to wait for I think.