Closed majidghgol closed 6 years ago
It should be Zookeeper & Kafka issue, because Kafka fails to commit offset to coordinator and consumers get the same data again and again. Have you solved this problem by freezing more memory or tuning Kafka config? There's also another way to commit offset (not implemented in myDIG): commit manually and yield consumers from getting message until coordinator receiving last commit.
@GreatYYX I added a line consumer.commit()
in run_core_kafka
after getting the message from the consumer, and turned off auto commit in ETL config. It solved this issue for me.
OK, thx.
Closing this issue. ETK consumer forced commits offset in the latest version.
@GreatYYX I have one document in my project in mydig, and looking at the ETK log, it seems that ETK is called multiple times (it is called in a loop over and over):
Note that in the ETK output above, "correct version2!" means that my local ETK is being run. Moreover, although nested documents are present, I have commented out the piece for nested docs in
run_core_kafka
, so the documents being processed are not nested docs (this can be also observed by the printed doc_id "xx" which I have assigned to the original document, the nested documents have different doc_id's).