search

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

Add custom entry point to Camus that can be called by Azkaban2 #16

Closed drdee closed 10 years ago

drdee commented 10 years ago

Add custom entry point to Camus that can be called by Azkaban2

Azkaban2 using the HadoopJava plugin cannot launch a Camus job, see bug report https://github.com/azkaban/azkaban-plugins/issues/132

This PR adds a method runFromAzkaban() to the CamusJob class that needs to be specified in the method.run argument in Azkaban job properties file. The runFromAzkaban() will load the Camus specific properties from a file and then invoke the regular Camus run() method.

This is not something we will upstream for obvious reasons but it does fix the problem.

@yagnik @wvanbergen @airhorns

yagnik commented 10 years ago

lgtm

airhorns commented 10 years ago

Cewl

wvanbergen commented 10 years ago

:+1: nice find!