search

pinterest / secor

Secor is a service implementing Kafka log persistence
Apache License 2.0
1.85k stars 540 forks source link

Unable To Process Transaction API Events #682

Closed kautshukla closed 4 years ago

kautshukla commented 5 years ago

I'm using kafak_2.0.0 profile with https://github.com/pinterest/secor/blob/master/src/main/config/kafka-2.0.0/kafka.properties properties to process the events which are pushed through Kafka producer using transaction API. While writing to an intermediate log file only captures the last event rest get skipped, found the same in the uploaded file as well.

Why it is happening?

Code base: 0.28-SNAPSHOT

I found the bug where the offset change isn't by +1 in case of read_commited isolation for Kafka consumer, whereas consuming the Kafka transaction API events pushed.

Block : https://github.com/pinterest/secor/blob/4068d9d3f6fa2bc2d30ad352973e044df4ccf4db/src/main/java/com/pinterest/secor/writer/MessageWriter.java#L78

There should be the check if new consumer which consumes {Kafka producer using transaction API} produced events which have offset difference of +2 as well.

kautshukla commented 5 years ago

@HenryCaiHaiying could you please have a look in this issue.

HenryCaiHaiying commented 4 years ago

Duplicate of https://github.com/pinterest/secor/pull/1017