search

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

Log delay from camus #38

Closed yagnik closed 9 years ago

yagnik commented 9 years ago

This will log delay to :dog: if it is over 1 hour from load time. It will allow us to see which topics are regularly late and we can then dig into why.

I'm already re-writing reporting module for camus for confluent so I don't plan to move this upstream to linkedin.

@drdee @kmtaylor-github cc @Shopify/data-engineers

kmtaylor-github commented 9 years ago

:+1:

yagnik commented 9 years ago

@drdee thoughts ?

drdee commented 9 years ago

On what?

yagnik commented 9 years ago

On the using instance approach...