search

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

Deploy to oozie #46

Closed orenmazor closed 9 years ago

orenmazor commented 9 years ago

What

this moves camus over to azkaban. once you ship this, the camus bundle will immediately start running. be aware, unlike me (sorry again, @yagnik and @drdee)

How

using oozie.py, the same way that starscream runs it. this ssh's into hadoop-misc4 and executes the same command that azkaban does.

Where

example bundle: https://hue.data.shopify.com/oozie/list_oozie_bundle/0000285-150806185006694-oozie-oozi-B

orenmazor commented 9 years ago

We'll need to manually disable the azkaban job when this ships… there isnt a mechanism to automatically do that here.

orenmazor commented 9 years ago

@yagnik I also removed shipping to azkaban here.

orenmazor commented 9 years ago

ping

drdee commented 9 years ago

:ship: but with the caveat of removing Camus from azkaban first :)

orenmazor commented 9 years ago

Haha okay. On Fri, Aug 21, 2015 at 4:33 PM Diederik van Liere notifications@github.com wrote:

[image: :ship:] but with the caveat of removing Camus from azkaban first :)

— Reply to this email directly or view it on GitHub https://github.com/Shopify/camus/pull/46#issuecomment-133552046.

orenmazor commented 9 years ago

\O/

On Tue, Aug 25, 2015 at 12:08 PM Yagnik notifications@github.com wrote:

Merged #46 https://github.com/Shopify/camus/pull/46.

— Reply to this email directly or view it on GitHub https://github.com/Shopify/camus/pull/46#event-391854653.