De-duplication flow - Githubissues

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.

7 stars 4 forks source link

Closed olessia closed 6 years ago

olessia commented 6 years ago

Add deduplicator class.
Create a non-scheduled deduplication task: it will need to be modified manually in Azkaban to take a file.

olessia commented 6 years ago

wip