search

linkedin / linkedin-gradle-plugin-for-apache-hadoop

Apache License 2.0
117 stars 76 forks source link

Minor build setup cleanups and improvements - Apply eclipse and idea … #123

Closed convexquad closed 7 years ago

convexquad commented 7 years ago

…plugins to all projects

pranayhasan commented 7 years ago

Don't include jline in azkaban-client build script as it overrides older version of jline included in hadoop-plugin. It caused hadoop validator to not work previously.

convexquad commented 7 years ago

Whoops, I had stashed these changes from an earlier point in the commit history and forgot to rebase. I will remove jline from azkaban-client.

pranayhasan commented 7 years ago

Oh, sorry. You have already taken care of it. I would want to test if Hadoop Validator works appropriately before pushing this version.

convexquad commented 7 years ago

What do you use for testing the Hadoop Validator tasks? Can I just run the hadoopValidate task in the Hadoop Starter Kit?

pranayhasan commented 7 years ago

Yes, you can. Follow go/hadoopvalidator to setup hadoop-starter-kit to be able to run it. I'm really excited to use this enhancement soon. I would use it once you push this. 💯

pranayhasan commented 7 years ago

Never mind, as you removed jline dependency for azkaban-client it would anyway work.

convexquad commented 7 years ago

In the hadoop-plugin project I moved jline:jline:1.0 to a runtime dependency, since it isn't actually used for compilation (but is indeed necessary at runtime).

I tested all the tasks with the hello-pig-project in the HSK and they all work correctly. The pigDataExists task fails since one of the datasets used in the HSK is no longer available (the old MEMBER2 database).

I am going to merge this PR since my SNAPSHOT build works correctly with the HSK. This PR will help me complete my work on the Hadoop DSL Intellij syntax completion.

@pranayhasan The new pigValidate tasks are totally awesome! I am super impressed with them. It is almost Thanksgiving here in the US, but after I come back I will help promote these new tasks, and also I want to do a bit more refactoring with the code base so that we can keep the tasks in good shape moving forward.