search

splunk / splunk-shuttl

Splunk app for archive management, including HDFS support.
Apache License 2.0
36 stars 19 forks source link

Local Storage ERRORS in logs but reports success in Splunk App #132

Closed camAtGitHub closed 11 years ago

camAtGitHub commented 11 years ago

Hello,

I have installed Shuttle 0.8.4 with Java JDK 7u25. I have configured my main index for testing with local storage (Eventually to use with Gluster locally mounted)

While the Shuttle app reports x amount of buckets moved and the is data in the destination directory, shuttl.log and splunkd.log are getting flooded with ERRORS.

Eg. shuttl.log

2013-08-05 22:37:15,125 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="main" path="/opt/splunk/var/lib/splunk/main/colddb/db_1275609461_1275609461_2" 2013-08-05 22:37:15,891 ERROR com.splunk.shuttl.archiver.model.MovesBuckets: did="Attempted to move bucket" happened="move failed" bucket="LocalBucket [getDirectory()=/opt/splunk/var/lib/splunk/main/colddb/db_1375753993_1375753993_3, getName()=db_1375753993_1375753993_3, getIndex()=main, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/var/lib/splunk/main/colddb/db_1375753993_1375753993_3, getEarliest()=Mon Aug 05 21:53:13 EDT 2013, getLatest()=Mon Aug 05 21:53:13 EDT 2013, getSize()=6283]" destination="/root/shuttl_archiver/data/safe-buckets/main" 2013-08-05 22:37:16,075 ERROR com.splunk.shuttl.archiver.model.MovesBuckets: did="Attempted to move bucket" happened="move failed" bucket="LocalBucket [getDirectory()=/opt/splunk/var/lib/splunk/main/db/db_1375754810_1375754081_4, getName()=db_1375754810_1375754081_4, getIndex()=main, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/var/lib/splunk/main/db/db_1375754810_1375754081_4, getEarliest()=Mon Aug 05 21:54:41 EDT 2013, getLatest()=Mon Aug 05 22:06:50 EDT 2013, getSize()=43327]" destination="/root/shuttl_archiver/data/safe-buckets/main"

splunkd.log

08-05-2013 22:35:35.975 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.freezeBucket(BucketFreezer.java:76) 08-05-2013 22:35:35.975 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.runMainWithDependencies(BucketFreezer.java:124) 08-05-2013 22:35:35.977 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.BucketFreezer.main(BucketFreezer.java:106) 08-05-2013 22:35:35.981 -0400 ERROR BucketMover - coldToFrozenScript cmd='/opt/splunk/etc/apps/shuttl/bin/coldToFrozenScript.sh /opt/splunk/var/lib/splunk/main/db/db_1375754810_1375754081_4' exited with non-zero status='exited with code 1' 08-05-2013 22:35:35.984 -0400 ERROR BucketMover - coldToFrozenScript cmd='/opt/splunk/etc/apps/shuttl/bin/coldToFrozenScript.sh /opt/splunk/var/lib/splunk/main/colddb/db_1357753335_1354321811_1' exited with non-zero status='exited with code 1' 08-05-2013 22:35:35.984 -0400 ERROR BucketMover - coldToFrozenScript Exception in thread "main" java.lang.RuntimeException: Couldn't move bucket to destination: /root/shuttl_archiver/data/safe-buckets/main 08-05-2013 22:35:35.984 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.model.MovesBuckets.logMoveFailureAndThrowException(MovesBuckets.java:69) 08-05-2013 22:35:35.984 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.model.MovesBuckets.moveBucket(MovesBuckets.java:49) 08-05-2013 22:35:35.985 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover.moveBucketToMovedBucketsLocationAndPerserveItsIndex(IndexPreservingBucketMover.java:85) 08-05-2013 22:35:35.985 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.recovery.IndexPreservingBucketMover.moveBucket(IndexPreservingBucketMover.java:75) 08-05-2013 22:35:35.985 -0400 ERROR BucketMover - coldToFrozenScript at com.splunk.shuttl.archiver.archive.MoveAndArchiveBucketUnderLock.moveThenArchiveBucket(MoveAndArchiveBucketUnderLock.java:61)

I dont believe it to be file permission related because most of the folder are chmod 777 (as part of my troubleshooting process)

This is my archiver.xml :

`<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

file:/~/shuttl_archiver local /mnt/drvol02/FrozenData cluster1 testidx01 SPLUNK_BUCKET ` Any help is appreciated
camAtGitHub commented 11 years ago

Hello, Sorry this issue has been resolved. The issues were that parts of /opt/splunk resided on BTRFS filesystem. Once /opt/splunk got moved to a fully EXT4 filesystem that resolved the issues.