Multinode test on AWS with HDFS

[Klevmarken]

• Setup a hadoop cluster consisting of 1 namenode and 3 datanodes on amazone aws using whirr.
• Test shuttling SPLUNK_BUCKET and CSV data to HDFS in the cluster with bucket sizes of 100MB, 1GB, 10GB.
• Test thawing SPLUNK_BUCKET and CSV data from HDFS to the local file system.

[Klevmarken]

• Configured whirr to set up a hadoop cluster cluster with 1 namenode and 3 datanodes. Pre-requisites:

  • Java 6
  • An account with a cloud provider, such as Amazon EC2, or Rackspace Cloud Servers
  • An SSH client

  Installation:

  • Update brew
  • Brew install maven
  • Download and install the latest version of hadoop
  • Download latest tar of whirr
  • Untar whirr in folder of your choice
  • In the whirr home directory run:
    • mvn clean install Rerunning the command to solve initial errors.
    • mvn package -Ppackage Due to errors I had to run: mvn package -Pdist -DskipTests=true -Dtar
    • bin/whirr version now printed "Apache Whirr ${project.version}" and the version of jclouds.
  • Create a file called hadoop.properties which works as the configuration file for building the cluster.
    • For most current documentation of how to configure this file see http://whirr.apache.org/docs/0.7.1/configuration-guide.html
    • When specifying which hadoop version to run, make sure it is the same as on your local system.
  • Launch the cluster:
    • bin/whirr launch-cluster --config hadoop.properties
  • With the launch of the cluster whirr creates two files
    • ~/.whirr/myhadoopcluster/hadoop-proxy.sh
    • ~/.whirr/myhadoopcluster/hadoop-site.xml
    • The first is used to start a proxy tunnel to the cluster.
    • The second should be cp'd into /usr/local/hadoop/conf to override the default config files.
      • Side note: hadoop-site.xml is depreciated but still used by whirr in communicating between the local hadoop instance and the namenode in the cluster.
        • After all that one should be good to go.

  Test:

  • On hold since shuttl cannot shuttle directly to a hadoop cluster from outside of amazon aws.

[Klevmarken]

Persisting 1GB SPLUNK_BUCKET with >10GB of Data: Transfer time for 1 bucket: ~25 sec for ~1GB Data and ~500MB rawdata. Errors: None

Number of buckets transferred: 13 Thawing time for all buckets: ~5 min Errors: None

Number of buckets transferred: 13 Persisting 10GB SPLUNK_BUCKET buckets with >100GB of Data: Transfer time for 1 bucket: ~6 min Errors: None

Number of buckets transferred: 4 Thawing time for all buckets: ~18 min Errors: None

Number of buckets transferred: 4 Persisting 1 GB CSV buckets with >10GB of Data: Transfer time for 1 bucket: ~8 min Errors: None

Number of buckets transferred: 5 Thawing time for all buckets: N/A Errors: No errors but had to spam on the thaw button in order to get all buckets to thaw.

Number of buckets transferred: 5 Persisting 10GB CSV buckets with >100GB of Data: Transfer time for 1 bucket: ~2-3 hours at least Errors: None

Number of buckets transferred: 3 Thawing time for 1 bucket: ~2-3 hours Errors: None Warnings: 2012-11-28 19:49:18,612 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ^C ec2-user@ip-10-144-10-97:~$ echo $HADOOP_HOME /usr/local/hadoop-1.1.0 ec2-user@ip-10-144-10-97:~$ Number of buckets transferred: 3

[Klevmarken]

Persisting 1GB SPLUNK_BUCKET/CSV with >10GB of Data: Transfer time for 1 bucket: ~11min Errors: None Number of buckets transferred: 7

Thawing 1GB SPLUNK_BUCKET/CSV with >10GB of Data: Transfer time for all buckets: ~2:30min Errors: None Number of buckets transferred: 7

[Klevmarken]

Persisting 10GB SPLUNK_BUCKET/CSV with >100GB of Data: Transfer time for 1 bucket: Unknown Errors: None Number of buckets transferred: 2

Thawing 10GB SPLUNK_BUCKET/CSV with >100GB of Data: Transfer time for all buckets: ~10 min Errors:

Several - 2012-12-10 16:40:33,938 ERROR ShuttlServer: Error during startup

Due to the above error I shutdown shuttl in order for it to reboot and start properly. This was after thawing had commenced. The following errors were recorded. In the end everything thawed properly.

2012-12-10 16:40:41,502 ERROR com.splunk.shuttl.archiver.filesystem.transaction.AbstractTransaction: did="Transferred data with transaction: Transaction [data=Bucket [format=SPLUNK_BUCKET, indexName=cluster_shuttl_test, bucketName=db_1355085971_1343952000_60 bucketSize=null, path=/user/ec2-user/KlevisTestArea/archive_data/hdfs/HDP-nn-SPL-master/cluster_shuttl_test/db_1355085971_1343952000_60/SPLUNK_BUCKET], remoteTemp=/mnt/KlevisTestArea/shuttl_archiver/data/thaw-transfers-dir/cluster_shuttl_test/db_1355085971_1343952000_60/SPLUNK_BUCKET/db_1355085971_1343952000_60, dst=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/thaweddb/db_1355085971_1343952000_60]" happened="java.io.IOException: Filesystem closed" expected="To transfer file to remote file system via temp." from="Bucket [format=SPLUNK_BUCKET, indexName=cluster_shuttl_test, bucketName=db_1355085971_1343952000_60 bucketSize=null, path=/user/ec2-user/KlevisTestArea/archive_data/hdfs/HDP-nn-SPL-master/cluster_shuttl_test/db_1355085971_1343952000_60/SPLUNK_BUCKET]" to="/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/thaweddb/db_1355085971_1343952000_60" temp="/mnt/KlevisTestArea/shuttl_archiver/data/thaw-transfers-dir/cluster_shuttl_test/db_1355085971_1343952000_60/SPLUNK_BUCKET/db_1355085971_1343952000_60" 2012-12-10 16:40:41,502 ERROR com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: did="Executed transaction" happened="com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: java.io.IOException: Filesystem closed" expected="Transaction to prepare and commit" transaction="Transaction [data=Bucket [format=SPLUNK_BUCKET, indexName=cluster_shuttl_test, bucketName=db_1355085971_1343952000_60 bucketSize=null, path=/user/ec2-user/KlevisTestArea/archive_data/hdfs/HDP-nn-SPL-master/cluster_shuttl_test/db_1355085971_1343952000_60/SPLUNK_BUCKET], remoteTemp=/mnt/KlevisTestArea/shuttl_archiver/data/thaw-transfers-dir/cluster_shuttl_test/db_1355085971_1343952000_60/SPLUNK_BUCKET/db_1355085971_1343952000_60, dst=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/thaweddb/db_1355085971_1343952000_60]" 2012-12-10 16:40:41,502 ERROR com.splunk.shuttl.archiver.thaw.GetsBucketsFromArchive: did="Tried to thaw bucket" happened="java.lang.RuntimeException: java.io.IOException: Filesystem closed" expected="Place the bucket in thaw" bucket="Bucket [format=SPLUNK_BUCKET, indexName=cluster_shuttl_test, bucketName=db_1355085971_1343952000_60 bucketSize=null, path=/user/ec2-user/KlevisTestArea/archive_data/hdfs/HDP-nn-SPL-master/cluster_shuttl_test/db_1355085971_1343952000_60/SPLUNK_BUCKET]" exception="java.lang.RuntimeException: java.io.IOException: Filesystem closed" 2012-12-10 16:40:41,683 ERROR com.splunk.shuttl.archiver.thaw.GetsBucketsFromArchive: did="Tried to thaw bucket" happened="java.lang.IllegalStateException: Shutdown in progress" expected="Place the bucket in thaw" bucket="Bucket [format=SPLUNK_BUCKET, indexName=cluster_shuttl_test, bucketName=db_1355086118_1343433524_63 bucketSize=null, path=/user/ec2-user/KlevisTestArea/archive_data/hdfs/HDP-nn-SPL-master/cluster_shuttl_test/db_1355086118_1343433524_63/SPLUNK_BUCKET]" exception="java.lang.IllegalStateException: Shutdown in progress" 2012-12-10 16:40:41,916 ERROR com.splunk.shuttl.archiver.thaw.GetsBucketsFromArchive: did="Tried to thaw bucket" happened="java.lang.IllegalStateException: Shutdown in progress" expected="Place the bucket in thaw" bucket="Bucket [format=SPLUNK_BUCKET, indexName=cluster_shuttl_test, bucketName=db_1355098496_1355085203_61 bucketSize=null, path=/user/ec2-user/KlevisTestArea/archive_data/hdfs/HDP-nn-SPL-master/cluster_shuttl_test/db_1355098496_1355085203_61/SPLUNK_BUCKET]" exception="java.lang.IllegalStateException: Shutdown in progress"

Number of buckets transferred: 2