Camus sweeper azkaban job

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.

7 stars 4 forks source link

Closed yagnik closed 10 years ago

yagnik commented 10 years ago

Add a public run method so Camus Sweeper can be started from Azkaban. cc @drdee

drdee commented 10 years ago

:gem:

yagnik commented 10 years ago

Things to talk about:

[ ] Late arriving events and appending to file
[ ] Deletion of hourly data from camus
[ ] Max deadline to drop events if not received within that time frame ~ 7days (same as kafka logs)

yagnik commented 10 years ago

We have moved to using the other branch. closing this