Add idempotent consumer implementation to Kafka consumer

teragrep / cfe_39

HDFS Data Ingestion for PTH_06 use

GNU Affero General Public License v3.0

0 stars 3 forks source link

Add idempotent consumer implementation to Kafka consumer #10

Open Tiihott opened 6 months ago

Tiihott commented 6 months ago

Description Without implementing idempotent Kafka consumer the database is fully reconstructed every time cfe_39 is started. By implementing idempotent Kafka consumer the consumer will skip processing all records that are already present in the HDFS database.

Tiihott commented 3 months ago

Solved in PR #11