Work with Hadoop 2.x branch

[mrflip]

I am running CDH5.2.0, Hadoop version 2.5.0, and get

2014-11-19 10:11:14,405 [main] WARN  org.apache.pig.backend.hadoop20.PigJobControl - falling back to default JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
    at java.lang.Class.getDeclaredField(Class.java:1953)
    at org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
    at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:98)

Basically this is caused by building against the wrong set of Hadoop libs.

For building Pig, this would be corrected by running

  ant jar-withouthadoop -Dhadoopversion=23

What is the equivalent for Lipstick? Will it build against Hadoop 2.x?

[alienrobotwizard]

The simplest way, if you want the new features of the "lipstick 2" server, would be to build console from the pig0.13 branch (uses pig 0.13 and hadoop 2.3.0). Then just run the master branch's lipstick-server. They're independent and the master branch's server will consume plans constructed from any branch's console.

• Long term roadmap is to port all the pig specific code (lipstick-console) into pig itself and make the logical plan accessible to ppnl's. This way all the lipstick project needs to do is maintain a simple ppnl that is registered by pig and constructs the appropriate json.

[mrflip]

In more detail:

1. Checkout Pig from apache's git repo, change to branch-0.13
2. Build pig against your Hadoop
3. Mavenize that jar
4. Clone lipstick's repo into a directory called lipstick-con
5. Switch to lipstick's pig0.13 branch.
6. alter the build scripts to source the custom jar you built
7. build the console only.
8. Clone the repo at master branch into a directory called lipstick-svr
9. build the server only.

My scripts:

pig setup:

PIG_VERSION=0.13
PIG_HOME=/usr/local/pig

apt-get install -y ant maven2

git clone https://github.com/apache/pig.git $PIG_HOME-$PIG_VERSION
ln  -s $PIG_HOME-$PIG_VERSION $PIG_HOME

cd $PIG_HOME
git checkout --track -b branch-$PIG_VERSION origin/branch-$PIG_VERSION

perl -pi -e 's/hadoop-(.*)=2.0.3-alpha/hadoop-\1=2.5.0/g' ivy/libraries.properties

ant -Dhadoopversion=23 jar jar-withouthadoop

mvn org.apache.maven.plugins:maven-install-plugin:2.5:install-file \
    -Dfile=build/pig-0.13.1-SNAPSHOT.jar \
    -DgroupId=org.apache.pig \
    -DartifactId=pig \
    -Dversion=0.13.1-h2 \
    -Dpackaging=jar

LIPSTICK_CON_DIR=/usr/local/lipstick-con
LIPSTICK_CONSOLE_LIBS=$LIPSTICK_CON_DIR/lipstick-console/build/libs

apt-get install -y graphviz

# Clone repo into a directory with version `dev`, link it to the canonical named location
git clone https://github.com/Netflix/Lipstick.git $LIPSTICK_CON_DIR-dev
ln -s $LIPSTICK_CON_DIR-dev $LIPSTICK_CON_DIR
cd $LIPSTICK_CON_DIR

# Work with the pig0.13 branch for the console
git checkout --track -b pig0.13 origin/pig0.13
# Fix gitignore
echo -e "lipstick-server/config\nlipstick-server/examples\n*.war\n.yardoc\n.gem\nlipstick-server/lib/*.jar\nlipstick-server/app/public/doc" >> .gitignore
# Change to use our pig and hadoop
perl -pi -e 's/(hadoop-.*):2\.3\.0/\1:2.5.0/g' build.gradle 
perl -pi -e 's/(:pig.*):0\.13\.0/\1:0.13.1-h2/g' build.gradle 
git commit -m "use our pig and our hadoop; and gitignore more" .
# Build
echo -e "\n♫ gradle gradle gradle I made you out of clay ♪\n"
./gradlew :lipstick-console:allJars
echo -e "\n♪ And when we are done gradling with lipstick I shall play ♬\n"

Server:

LIPSTICK_SVR_DIR=/usr/local/lipstick-svr
# Clone repo into `-dev`, link it to the canonical named location 
git clone https://github.com/Netflix/Lipstick.git $LIPSTICK_SVR_DIR-dev && \
  ln -s $LIPSTICK_SVR_DIR-dev $LIPSTICK_SVR_DIR                               
git checkout master

echo -e "\n♫ gradle gradle gradle I made you out of clay ♪\n"

./gradlew :lipstick-server:war

grep "task('run-app'" $LIPSTICK_SVR_DIR/build.gradle    && \
  perl -pi -e "s/task\\(.run-app..*/task\\(\'run-app\', type:JavaExec\\) \\{/" $LIPSTICK_SVR_DIR/build.gradle  &&  \
  grep "task('run-app'" $LIPSTICK_SVR_DIR/build.gradle
git commit -m "Don't rebuild universe to run app" .