Test standalone install of trunk

[milandesai]

We need to test both a standalone install and a distributed install; this issue tracks the former. I have hit some configuration and HBase related problems while attempting to format and run Giraffa; will post them and their fixes soon.

[milandesai]

Posting the bugs I have found so far:

1. fs.defaultFS should be moved to core-site.xml
2. The HBase DynamicClassLoader, called when we initialize GiraffaFileSystem, lets you load jars dynamically from a specific folder. This folder can be located in Hadoop. So it tries to load GiraffaFileSystem again to access this directory. But the DynamicClassLoader was loaded in a static initialization block of class ProtobufUtil, so the other static variables of ProtobufUtil have not been loaded yet, causing a "ghost" NullPointerException for ProtobufUtil.PB_MAGIC, even though it's a static final byte[] of length 4. The fix is to prevent the DynamicClassLoader from looking for a remote directory by setting "hbase.dynamic.jars.dir" in the configuration equal to the local jars dir, "${hbase.local.dir}/jars/". But this has to be done in hbase-site.xml, or else it will be overriden by hbase-defaults.xml. In fact, all of our hbase.* configs should be moved to hbase-site.xml.
3. Not particularly important, but the hadoop and hbase version numbers in hadoop-env.sh should be updated to prevent confusion.

Will create a pull request momentarily, but still testing, there may be other issues.

[milandesai]

1. GIRAFFA_OPTS not sent to Java command for giraffa format, causing logging and debugging to not work.

[milandesai]

Ls works, but mkdirs and other commands do not. Investigating.

[milandesai]

Problem was that after moving hbase configurations to hbase-site.xml, GiraffaConfiguration needed to be updated to include HBase resources. That also necessitated moving GiraffaConfiguration to the hbase package. Now I am getting a NullPointerException on create because the leaseManager is null (false alarm, forgot to update the giraffa jar in hbase).

[milandesai]

Client commands and MapReduce over Yarn work for me with the latest commit. This issue is ready for review. Please note that configuration.xsl, capacity-scheduler.xml, container-executer.cfg, yarn-env.sh, and mapred-sh were all copied exactly as is from Hadoop 2.5.1, so excuse the unnecessary changes and whitespace errors in these; I want to keep these files as exact copies so users can easily compare if they modify any values.

[milandesai]

@zero45, @shvachko, @octo47 - if you could all check out the branch and make sure the standalone works for you, that would be great. Here are the instructions (after this is pushed, I'll also update the Wiki):

Part 1: Giraffa Setup

1. Download and extract the distributions for Hadoop 2.5.1, HBase 1.0.1, and Giraffa 0.2 (from giraffa-standalone/target/giraffa-standalone/giraffa). Place them in an accessible location.
2. Configure Hadoop:
   • Set fs.defaultFS to hdfs://localhost:9000 [etc/hadoop/core-site.xml]
   • Set hadoop.tmp.dir to a permanent location [etc/hadoop/core-site.xml]
3. Configure HBase:
   • Set hbase.rootdir to hdfs://localhost:9000/hbase [conf/hbase-site.xml]
   • Set hbase.tmp.dir to a permanent location [conf/hbase-site.xml]
   • Set hbase.cluster.distributed to true [conf/hbase-site.xml]
   • Create core-site.xml and hdfs-site.xml with no properties set in conf/ (include the “configuration” start and end tags as in hbase-site.xml).
   • Uncomment the JAVA_HOME line in conf/hbase-env.sh and set the variable to /usr/libexec/java_home. This must be done even if you already have a global environment for JAVA_HOME configured.
   • Copy $GIRAFFA_HOME/lib/giraffa-*.jar to lib/.
4. Configure Giraffa: Point HADOOP_HOME and HBASE_HOME to correct locations in conf/giraffa-env.sh.
5. Format HDFS NameNode: bin/giraffa namenode -format
6. Start Giraffa in pseudo-distributed mode: bin/start-giraffa.sh. You should have the following processes running: NameNode, DataNode, HQuorumPeer, HMaster, HRegionServer.
7. Format Giraffa: bin/giraffa format.
8. Run Giraffa client: bin/giraffa fs …

Part 2: Yarn Setup

1. Launch the ResourceManager: bin/yarn-giraffa-daemon.sh start resourcemanager
2. Launch the NodeManager: bin/yarn-giraffa-daemon.sh start nodemanager. You should have the following processes running: NameNode, DataNode, HQuorumPeer, HMaster, HRegionServer, ResourceManager, NodeManager.
3. Run Yarn client: bin/yarn-daemon jar …

Part 3: TeraSort

1. Run TeraGen: bin/yarn-giraffa jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar teragen 10000000 /teragen
2. Run TeraSort: bin/yarn-giraffa jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar terasort /teragen /terasort
3. Run TeraValidate: bin/yarn-giraffa jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar teravalidate /terasort /teravalidate

[milandesai]

@shvachko, the pull request here is hard to decipher because it fixes multiple different issues that should probably have their own Jiras, and there are also some style issues that were fixed. So I'm proposing that we keep this Jira an investigative task and open four new issues:

1. Move fs.defaultFS configuration into core-site
2. Introduce hbase-site.xml for HBase configurations
3. Set HBase dynamic jars directory to local path
4. Pull updated Yarn configurations

Those four will make the standalone build work. Then later we can commit some "cleanup" issues:

1. Update giraffa-env.sh with correct Hadoop/HBase versions
2. Fix license and style issues on configurations

[shvachko]

Milan this is great progress! I like the patch and especially that it makes terasor pass:

1. Let's just remove blank space changes in quite a few places.
2. Do you need to add allowed.system.users? It is empty anyways.
3. There is a lot of comments and spaces added in yarn-env.sh. We should remove those unless it is an exact copy of something from yarn.
4. I would keep the rest as-is in one patch. Only would create an issue to update the documentation on cluster setup. There are some changes, right?

[milandesai]

Thanks Konstantin. The files capacity-scheduler.xml, configuration.xsl, container-executer.cfg, mapred-env.sh, and yarn-env.sh were copied exactly as is from Hadoop 2.5.1 configurations directory. That way were are consistent and any changes the user makes to these files can easily be compared with their counterparts in Hadoop. There are some other whitespace or unnecessary changes that I can remove though.

[milandesai]

Non-essential changes have been removed, except for the files listed above which remain exact copies.

[shvachko]

WooHoo

[milandesai]

Committed to trunk as 868b7230fb74361a6c00e2c3265b06bf98d80b9f