search

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

Increase frequency in which data is moved to HDFS #33

Closed bani closed 9 years ago

bani commented 9 years ago

We'd like to have the data from Kafka updated more often so that we could use it in incremental jobs more frequently

@yagnik

drdee commented 9 years ago

@bani data is dropped at an hourly interval, is that not frequent enough?