Splunk 5.0 clustering QA

Setup:

Set up a cluster of Splunk 5.0 instances.
Make one the Cluster Master.
Configure at least 2 other instances to be cluster slaves, connected to the master.
Configure an index that is replicated across the cluster.
Roll data on all indexers.

Assert:

Make sure there's no buckets called rb_...
Make sure there are no duplicated buckets.
Make sure that data is not stored in temp, forever.

Setup:

Cluster of Splunk 5.0 consisting of 1 Cluster Master and 4 Slaves
Hadoop 1.1.0 was configured on the CM and 3 of the Slaves (the slave without hadoop will be referred to has the "lone node")
Index with a 1GB bucket size was configured for all the nodes
Rolled data over the indexers by feeding the lone node with multiple 1GB large CSV files.

Asserted:

No buckets are called rb_...
No duplicate buckets exist in archive_data/ but duplicate map structures seem to persist under temp.
The data is not stored for forever in temp but the map structure after the buckets seem to persist.

Asserted errors on Splunk Slave/Hadoop Data Node:

2012-11-24 13:00:45,186 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/HDP-dn-SPL-sl-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-11-24 13:00:45,228 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/HDP-dn-SPL-sl-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-11-24 13:00:45,228 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/HDP-dn-SPL-sl-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-11-24 13:00:45,242 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/HDP-dn-SPL-sl-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-11-24 13:00:45,329 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Thu Aug 30 00:00:00 UTC 2012, getLatest()=Wed Nov 21 14:36:11 UTC 2012, getSize()=529888359]" 2012-11-24 13:00:45,330 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="rb_1353508571_1346284800_9_91C4875E-8017-4E9D-9004-5C3223F1042A" entity="" 2012-11-24 13:00:45,331 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-8-17-168.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-11-24 13:00:45,442 ERROR com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint: did="Tried archiving a bucket" happened="java.lang.NullPointerException" expected="To archive the bucket" index="cluster_shuttl_test" bucket_path="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1353736131_1334188800_0_CAD3840E-411D-42C2-AFF1-758FD30C8D7C" 2012-11-24 13:00:45,466 ERROR com.splunk.shuttl.archiver.archive.ArchiveRestHandler: did="Sent an archive bucket reuqest" happened="Got non ok http_status" expected="expected HttpStatus.SC_OK or SC_NO_CONTENT" http_status="500" bucket_name="rb_1353736131_1334188800_0_CAD3840E-411D-42C2-AFF1-758FD30C8D7C" 2012-11-24 13:00:45,467 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-8-17-168.ec2.internal:9090/shuttl/rest/archiver/bucket/archive"

Asserted errors on lone Splunk Slave:

2012-11-24 13:00:11,777 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Tue Oct 09 13:04:37 UTC 2012, getLatest()=Wed Nov 21 14:35:09 UTC 2012, getSize()=924692345], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/hdp-SPL-CP-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET]" 2012-11-24 13:00:11,781 ERROR com.splunk.shuttl.archiver.filesystem.transaction.AbstractTransaction: did="Tried making directories up to: /user/ec2-user/KlevisTestArea/temporary_data/hdp-SPL-CP-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET" happened="org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="temporary_data":ec2-user:supergroup:rwxr-xr-x" expected="To make directories." path="/user/ec2-user/KlevisTestArea/temporary_data/hdp-SPL-CP-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET" 2012-11-24 13:00:11,782 ERROR com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: did="Executed transaction" happened="com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="temporary_data":ec2-user:supergroup:rwxr-xr-x" expected="Transaction to prepare and commit" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Tue Oct 09 13:04:37 UTC 2012, getLatest()=Wed Nov 21 14:35:09 UTC 2012, getSize()=924692345], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/hdp-SPL-CP-1/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hadoop/hdp-SPL-CP-1/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET]" 2012-11-24 13:00:11,784 ERROR com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: did="Executed a bucket transaction." happened="com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="temporary_data":ec2-user:supergroup:rwxr-xr-x" expected="To transfer the bucket to the archive." bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Tue Oct 09 13:04:37 UTC 2012, getLatest()=Wed Nov 21 14:35:09 UTC 2012, getSize()=924692345]" 2012-11-24 13:00:11,784 DEBUG com.splunk.shuttl.archiver.archive.BucketCopier: did="Copied bucket" happened="com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="temporary_data":ec2-user:supergroup:rwxr-xr-x" result="Will eventually throw this exception" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Tue Oct 09 13:04:37 UTC 2012, getLatest()=Wed Nov 21 14:35:09 UTC 2012, getSize()=924692345]" 2012-11-24 13:00:11,785 ERROR com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint: did="Tried archiving a bucket" happened="java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=ubuntu, access=WRITE, inode="temporary_data":ec2-user:supergroup:rwxr-xr-x]" expected="To archive the bucket" index="cluster_shuttl_test" bucket_path="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A" 2012-11-24 13:00:11,799 ERROR com.splunk.shuttl.archiver.archive.ArchiveRestHandler: did="Sent an archive bucket reuqest" happened="Got non ok http_status" expected="expected HttpStatus.SC_OK or SC_NO_CONTENT" http_status="500" bucket_name="db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A" 2012-11-24 13:00:11,850 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508509_1349787877_26_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Tue Oct 09 13:04:37 UTC 2012, getLatest()=Wed Nov 21 14:35:09 UTC 2012, getSize()=924692345], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353750313_1347666192_16_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353750313_1347666192_16_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353750313_1347666192_16_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Fri Sep 14 23:43:12 UTC 2012, getLatest()=Sat Nov 24 09:45:13 UTC 2012, getSize()=1000410407], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353761640_1350025218_25_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353761640_1350025218_25_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353761640_1350025218_25_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Fri Oct 12 07:00:18 UTC 2012, getLatest()=Sat Nov 24 12:54:00 UTC 2012, getSize()=865116962], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353759962_1349363724_20_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353759962_1349363724_20_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353759962_1349363724_20_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Thu Oct 04 15:15:24 UTC 2012, getLatest()=Sat Nov 24 12:26:02 UTC 2012, getSize()=1077790757], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353750303_1347641804_15_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353750303_1347641804_15_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353750303_1347641804_15_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Fri Sep 14 16:56:44 UTC 2012, getLatest()=Sat Nov 24 09:45:03 UTC 2012, getSize()=981199209], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508538_1349308800_21_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508538_1349308800_21_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508538_1349308800_21_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Thu Oct 04 00:00:00 UTC 2012, getLatest()=Wed Nov 21 14:35:38 UTC 2012, getSize()=866783157], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353757174_1348963200_19_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353757174_1348963200_19_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353757174_1348963200_19_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Sun Sep 30 00:00:00 UTC 2012, getLatest()=Sat Nov 24 11:39:34 UTC 2012, getSize()=1060521346], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353753548_1347666088_17_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353753548_1347666088_17_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353753548_1347666088_17_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Fri Sep 14 23:41:28 UTC 2012, getLatest()=Sat Nov 24 10:39:08 UTC 2012, getSize()=823519370], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353760341_1349913600_24_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353760341_1349913600_24_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353760341_1349913600_24_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Thu Oct 11 00:00:00 UTC 2012, getLatest()=Sat Nov 24 12:32:21 UTC 2012, getSize()=843870691], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353753558_1349302218_18_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353753558_1349302218_18_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353753558_1349302218_18_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Wed Oct 03 22:10:18 UTC 2012, getLatest()=Sat Nov 24 10:39:18 UTC 2012, getSize()=1073415387], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353760340_1350085993_23_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353760340_1350085993_23_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353760340_1350085993_23_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Fri Oct 12 23:53:13 UTC 2012, getLatest()=Sat Nov 24 12:32:20 UTC 2012, getSize()=837663975], LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508553_1349395200_22_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353508553_1349395200_22_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353508553_1349395200_22_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Fri Oct 05 00:00:00 UTC 2012, getLatest()=Wed Nov 21 14:35:53 UTC 2012, getSize()=953548030]]" 2012-11-24 13:00:11,887 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-40-218-240.ec2.internal:9090/shuttl/rest/archiver/bucket/archive"

Persisting data with 1GB buckets to HDFS.

The following errors preceded the test in the log and are due to shuttl not rebooting with splunk on ubuntu:

2012-12-11 21:09:56,116 ERROR ShuttlServer: Error during startup java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:173) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:311) at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:251) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) at org.eclipse.jetty.server.Server.doStart(Server.java:274) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) at com.splunk.shuttl.server.ShuttlJettyServer.main(ShuttlJettyServer.java:58)

Log from Splunk slave / HDP Datanode:

2012-12-11 21:24:35,188 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="cluster_shuttl_test" path="/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6" 2012-12-11 21:24:35,297 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: will="moving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=995720970]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:24:35,299 DEBUG com.splunk.shuttl.archiver.model.MovesBuckets: will="Attempting to move bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=995720970]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test" 2012-12-11 21:24:35,303 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: did="moved bucket" happened="success" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=995720970]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:24:35,332 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-68-121-205.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-11 21:24:37,800 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=995720970]" 2012-12-11 21:24:38,088 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=995720970]" 2012-12-11 21:24:38,103 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="rb_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-11 21:24:38,104 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[]" 2012-12-11 21:29:35,788 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="cluster_shuttl_test" path="/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6" 2012-12-11 21:29:35,942 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: will="moving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:29:35,944 DEBUG com.splunk.shuttl.archiver.model.MovesBuckets: will="Attempting to move bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test" 2012-12-11 21:29:35,946 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: did="moved bucket" happened="success" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:29:35,970 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-68-121-205.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-11 21:29:37,080 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302]" 2012-12-11 21:29:37,096 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302]" destination="/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET" 2012-12-11 21:29:37,099 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:29,943 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:29,944 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:29,954 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:29,967 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:30:29,973 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:30:29,973 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:30:29,983 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-68-121-205/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:30:30,100 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=451595302]" 2012-12-11 21:30:30,107 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="rb_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-11 21:30:30,109 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[]"

Log from the lone splunk slave:

2012-12-11 21:18:05,541 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="cluster_shuttl_test" path="/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6" 2012-12-11 21:18:05,660 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: will="moving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:18:05,662 DEBUG com.splunk.shuttl.archiver.model.MovesBuckets: will="Attempting to move bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test" 2012-12-11 21:18:05,668 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: did="moved bucket" happened="success" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:18:05,680 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-40-218-240.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-11 21:18:06,645 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588]" 2012-12-11 21:18:06,758 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588]" destination="/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET" 2012-12-11 21:18:06,762 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:19:28,761 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:19:28,762 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:19:28,765 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:19:28,788 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:28,793 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:28,793 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:28,797 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:29,127 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Thu Aug 09 00:00:00 UTC 2012, getLatest()=Tue Dec 11 21:13:25 UTC 2012, getSize()=882093588]" 2012-12-11 21:19:29,135 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="db_1355260405_1344470400_0_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-11 21:19:29,139 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393]]" 2012-12-11 21:19:29,142 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-40-218-240.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-11 21:19:29,173 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393]" 2012-12-11 21:19:29,177 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393]" destination="/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET" 2012-12-11 21:19:29,179 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET]" 2012-12-11 21:19:29,672 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET]" 2012-12-11 21:19:29,673 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET]" 2012-12-11 21:19:29,690 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET]" 2012-12-11 21:19:29,693 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:29,747 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:29,747 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:29,761 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:19:29,766 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getName()=db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A, getEarliest()=Mon Nov 26 19:04:50 UTC 2012, getLatest()=Mon Nov 26 19:04:50 UTC 2012, getSize()=1393]" 2012-12-11 21:19:29,767 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="db_1353956690_1353956690_28_91C4875E-8017-4E9D-9004-5C3223F1042A" entity="" 2012-12-11 21:24:20,851 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="cluster_shuttl_test" path="/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6" 2012-12-11 21:24:20,969 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: will="moving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:24:20,972 DEBUG com.splunk.shuttl.archiver.model.MovesBuckets: will="Attempting to move bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test" 2012-12-11 21:24:20,976 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: did="moved bucket" happened="success" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:24:20,989 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-40-218-240.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-11 21:24:21,376 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486]" 2012-12-11 21:24:21,424 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486]" destination="/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET" 2012-12-11 21:24:21,425 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:24:51,528 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:24:51,528 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:24:51,531 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:24:51,551 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:24:51,553 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:24:51,553 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:24:51,555 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:24:51,934 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:36:46 UTC 2012, getSize()=1046557486]" 2012-12-11 21:24:51,943 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="db_1355085406_1343952000_3_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-11 21:24:51,944 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[]" 2012-12-11 21:29:22,625 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="cluster_shuttl_test" path="/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6" 2012-12-11 21:29:22,745 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: will="moving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:29:22,747 DEBUG com.splunk.shuttl.archiver.model.MovesBuckets: will="Attempting to move bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test" 2012-12-11 21:29:22,753 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: did="moved bucket" happened="success" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/db/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-11 21:29:22,765 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-40-218-240.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-11 21:29:23,189 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068]" 2012-12-11 21:29:23,217 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068]" destination="/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET" 2012-12-11 21:29:23,218 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:36,976 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:36,977 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:36,980 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Fri Aug 03 00:05:45 UTC 2012, getLatest()=Sun Dec 09 20:38:55 UTC 2012, getSize()=923748068], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-11 21:30:36,990 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-11 21:30:36,992 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085535_1343952345_4_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-sla

Retested persisting looking at the log and size of the bucket at different stages of the process.

Some statistics regarding the size: Bucket nr:..................lf safe_buckets:.........Web:........................hdfs: 7..................................968MB.....................966MB.....................490MB 10................................871MB.....................868MB.....................438MB 5......................................-..........................449MB.....................471MB* 1...................................809MB....................807MB....................847MB 12...............................1005MB...................1002MB....................504MB

_The buckets marked with a star only had sizeManifest, archive_meta, and rawdata on hdfs. *_I have omitted bucket nr 5 because it was not logged in all of the splunk-shuttl-slave logs. It also only contains sizeManifest, archive_meta, and rawdata. The node I was currently monotering during the test seems to have gone down temporarily since it didn't log anything for a period of time. Then all of a sudden it started logging again. No hint that it ever stopped logging.

Retested persisting looking at the log and size of the buckets at different stages of the transfer process.

Some numeric results: Bucket nr:.......lf safe_buckets:.......Web:..................hdfs: 14..................999/514 (raw)...........512->997............~512 17...................817/395 (raw)..........393->815............~393 18..................988/491 (raw)...........unknown->986.....~489

The buckets only contained sizeManifest, archive_meta, and rawdata on hdfs.

The following log message is the final entry in shuttl.log and existed before the start of this test, but I found it strange that nothing was logged at all in the master's shuttl.log after starting the test.

Extract from Master node: 2012-12-12 01:41:48,431 FATAL org.apache.jasper.EmbeddedServletOptions: The scratchDir you specified: /tmp/jetty-0.0.0.0-9090-shuttl-_shuttl-any-/jsp is unusable.

I think the logs look normal except for that when persisting to hdfs only a few things are mentions in the prepare sections. I dont know if that is supposed to look like that but I thought more should be logged?

Extract from splunk-lone-slave log: 2012-12-12 02:04:10,829 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Tue Sep 11 15:56:13 UTC 2012, getLatest()=Wed Dec 12 01:47:03 UTC 2012, getSize()=1034011136], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-12 02:04:40,720 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Tue Sep 11 15:56:13 UTC 2012, getLatest()=Wed Dec 12 01:47:03 UTC 2012, getSize()=1034011136], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-12 02:04:40,721 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Tue Sep 11 15:56:13 UTC 2012, getLatest()=Wed Dec 12 01:47:03 UTC 2012, getSize()=1034011136], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-12 02:04:40,723 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Tue Sep 11 15:56:13 UTC 2012, getLatest()=Wed Dec 12 01:47:03 UTC 2012, getSize()=1034011136], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-12 02:04:40,732 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:40,734 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:40,734 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:40,737 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:41,132 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Tue Sep 11 15:56:13 UTC 2012, getLatest()=Wed Dec 12 01:47:03 UTC 2012, getSize()=1034011136]" 2012-12-12 02:04:41,140 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-12 02:04:41,142 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[]" 2012-12-12 02:04:42,218 INFO com.splunk.shuttl.server.mbeans.rest.ListBucketsEndpoint: happened="Received REST request to list indexes" endpoint="/index/list" 2012-12-12 02:04:42,226 INFO com.splunk.shuttl.server.mbeans.rest.ListBucketsEndpoint: happened="Received REST request to list buckets" endpoint="/bucket/list" index="null" from="null" to="null" 2012-12-12 02:04:42,227 INFO com.splunk.shuttl.server.mbeans.rest.RestUtil: No from time provided - defaulting to 0001-01-01 2012-12-12 02:04:42,227 INFO com.splunk.shuttl.server.mbeans.rest.RestUtil: No to time provided - defaulting to 9999-12-31 2012-12-12 02:04:46,268 INFO com.splunk.shuttl.server.mbeans.rest.ListBucketsEndpoint: happened="Received REST request to list indexes" endpoint="/index/list" 2012-12-12 02:04:46,274 INFO com.splunk.shuttl.server.mbeans.rest.ListBucketsEndpoint: happened="Received REST request to list buckets" endpoint="/bucket/list" index="null" from="null" to="null" 2012-12-12 02:04:46,274 INFO com.splunk.shuttl.server.mbeans.rest.RestUtil: No from time provided - defaulting to 0001-01-01 2012-12-12 02:04:46,274 INFO com.splunk.shuttl.server.mbeans.rest.RestUtil: No to time provided - defaulting to 9999-12-31

Extract from splunk-slave:

2012-12-12 02:04:42,136 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Tue Sep 11 15:56:13 UTC 2012, getLatest()=Wed Dec 12 01:47:03 UTC 2012, getSize()=1001109834], remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET]" 2012-12-12 02:04:42,142 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:42,143 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:42,143 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:42,144 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-12 02:04:42,404 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Tue Sep 11 15:56:13 UTC 2012, getLatest()=Wed Dec 12 01:47:03 UTC 2012, getSize()=1001109834]" 2012-12-12 02:04:42,410 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="rb_1355276823_1347378973_18_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-12 02:04:42,411 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[]"

Same test with new version of shuttl.

Extract from lone-slave:

2012-12-13 20:17:06,260 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:17:06,263 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:17:06,263 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:17:06,265 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/the-lone-slave/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:17:06,635 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Sat Aug 04 00:00:00 UTC 2012, getLatest()=Sun Dec 09 20:42:48 UTC 2012, getSize()=936792105]" 2012-12-13 20:17:06,643 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="db_1355085768_1344038400_24_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-13 20:17:06,644 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[]"

Extract from splunk-slave:

2012-12-13 20:10:50,972 DEBUG com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover: did="moved bucket" happened="success" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/splunk/var/lib/splunk/cluster_shuttl_test/colddb/rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Sat Jul 07 00:00:00 UTC 2012, getLatest()=Mon Dec 10 04:37:20 UTC 2012, getSize()=530845121]" destination="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets" 2012-12-13 20:10:50,983 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-190-170-133.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-13 20:10:51,334 WARN com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint: warning="Tried shuttling a replicted bucket that only contained rawdata" happened="Will not Shuttl this bucket, because Shuttl does not support rawdata only buckets." result="Deleted bucket. But don't worry, searchable copies and the original bucket will still be Shuttled, by the other indexers. You have Shuttl installed at all indexers, don't you?" bucket_path="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6" index="cluster_shuttl_test" 2012-12-13 20:10:51,340 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-13 20:10:51,343 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6, getName()=rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Sat Jul 07 00:00:00 UTC 2012, getLatest()=Mon Dec 10 04:37:20 UTC 2012, getSize()=530845121]]" 2012-12-13 20:10:51,344 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: will="Send an archive bucket request" request_uri="http://ip-10-190-170-133.ec2.internal:9090/shuttl/rest/archiver/bucket/archive" 2012-12-13 20:10:51,353 WARN com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint: warning="Tried shuttling a replicted bucket that only contained rawdata" happened="Will not Shuttl this bucket, because Shuttl does not support rawdata only buckets." result="Deleted bucket. But don't worry, searchable copies and the original bucket will still be Shuttled, by the other indexers. You have Shuttl installed at all indexers, don't you?" bucket_path="/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6" index="cluster_shuttl_test" 2012-12-13 20:10:51,354 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="rb_1355114240_1341619200_23_61E70347-200D-476F-A710-8E74F97527E6" entity=""

Thawing looks ok; right size, correct content.

2012-12-13 20:05:11,858 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:05:11,894 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Preparing transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:05:11,894 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:05:11,904 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: done="Commit transaction" transaction="Transaction [data=/mnt/KlevisTestArea/shuttl_archiver/data/metadata-dir/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/bucket.size, remoteTemp=/user/ec2-user/KlevisTestArea/temporary_data/ip-10-190-170-133/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size, dst=/user/ec2-user/KlevisTestArea/archive_data/hdfs/the-lone-slave/cluster_shuttl_test/db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6/SPLUNK_BUCKET/archive_meta/bucket.size]" 2012-12-13 20:05:12,152 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: done="Archived bucket" bucket="LocalBucket [getDirectory()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6, getName()=db_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6, getIndex()=cluster_shuttl_test, getFormat()=SPLUNK_BUCKET, getPath()=/mnt/KlevisTestArea/shuttl_archiver/data/safe-buckets/cluster_shuttl_test/rb_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6, getEarliest()=Sat Jul 07 22:16:26 UTC 2012, getLatest()=Thu Dec 13 19:59:10 UTC 2012, getSize()=933024481]" 2012-12-13 20:05:12,162 DEBUG com.splunk.shuttl.archiver.archive.ArchiveRestHandler: done="Got http response from archiveBucketRequest" status_code="204" bucket_name="rb_1355428750_1341699386_20_61E70347-200D-476F-A710-8E74F97527E6" entity="" 2012-12-13 20:05:12,164 DEBUG com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[]"

Verified all behaviours

splunk / splunk-shuttl

Splunk 5.0 clustering QA #102