search

confluentinc / camus

Mirror of Linkedin's Camus
Apache License 2.0
0 stars 62 forks source link

Switch to thin jars for packaging, adding bin scripts to make running the hadoop job simpler #10

Closed ewencp closed 9 years ago

ewencp commented 9 years ago

@Ishiihara Probably makes most sense for you to review this. We need to completely remove any shaded jars since they get published to Maven if they get built at all and we can't do that for licensing reasons. The bulk of the patch is just the license files, which you can ignore unless you feel like double checking that I correctly covered all the jars we put into the package.

What most needs review and testing is that the bin scripts work properly and that the quickstart instructions are still ok.

Ishiihara commented 9 years ago

@ewencp Thanks a lot for working on this. Another side effect of this PR is that Camus run on HDP smoothly with the mapreduce.job.user.classpath.first configuration. I figured this out and was planning to add it to mapred-site.xml but your solution is definitely better.

Tested together with https://github.com/confluentinc/ducttape/issues/21 both on CDH and HDP and Camus runs smoothly on all the deployments. Left some minor comments and otherwise LGTM.

ewencp commented 9 years ago

@Ishiihara Ok, so I think this is good to merge except that I want to verify that libjars should always be required (I'm assuming no local mode since I don't want to address a simple testing setup in detail in the instructions).

If libjars isn't always required, can you explain the exact setup where it isn't? I'd like to verify that with some testing since my understanding is that it should be required unless the only jar you need is the one you run hadoop jar on (i.e. it should only work if you're using uber jars).

Ishiihara commented 9 years ago

@ewencp I think you are right on the libjars. I was under the impression of the Camus uber jar was used.