Problems with archive path on S3

[clagese]

I configured my cluster to archive frozen splunk data on s3 with shuttl. I installed Shuttl on all nodes. My data are really archived on s3 bucket but there is a problem with archiviation path. Any path values I set as "archivePath" in archiver.xml( /myrootpath or myrootpath or or /), Shuttl store my data in a root path "//" within my s3 bucket. The result is like a directory without name, in the root of s3 bucket. For example if I set /myrootpath in archiver.xml, I find on s3 splunk buckets like "s3://my_s3_bucket//myrootpath/archive_data/my_cluster/splunkIndex02/mytestdb/db_1397659802_1397655448_19_B95021DE-89AB-4A9D-B924-575736C54B81"

In the shuttl log I found these warning: 2014-04-16 18:56:30,700 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397659802_1397655448_19_B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Unexpected response code 404, expected 200 2014-04-16 18:56:30,701 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397659802_1397655448_19_B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Received error response with XML message

In spite of this problem the Thaw and flush process of splunk buckets from search head interface works well.

I red issues: http://answers.splunk.com/answers/85635/shuttl-archiving-errors https://github.com/splunk/splunk-shuttl/issues/131 where you say that this behavior is due to an Hadoop's old version of the S3 library.

Are there new shuttl release that fix this problem or are there some planned soon?

[clagese]

I found another problem with storage in S3. If I save frozen bucket in CSV format in S3 with Shuttl , stored files are not really in csv format and I cannot read them. I can only restore frozen bucket through shuttl and read data with splunk. I don't understand if this is a normal behavior. I documented this problem in the splunk issue: http://answers.splunk.com/answers/131297/shuttl-development-stopped I tested the develop branch of shuttl with the hope that this version fix these problems, but it not work. I get this errors in shuttl.log: 2014-05-14 18:10:30,721 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 23 16:55:24 CEST 2014, getLatest()=Sun Apr 27 02:17:01 CEST 2014, getSize()=132526]" 2014-05-14 18:10:30,821 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fdevelop_branch%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200 2014-05-14 18:10:30,822 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fdevelop_branch%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message 2014-05-14 18:10:30,905 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 23 16:55:24 CEST 2014, getLatest()=Sun Apr 27 02:17:01 CEST 2014, getSize()=132526]" destination="/develop_branch/archive_data/my_cluster/splunkIndex02/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/CSV" 2014-05-14 18:10:30,905 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 23 16:55:24 CEST 2014, getLatest()=Sun Apr 27 02:17:01 CEST 2014, getSize()=132526], remoteTemp=/develop_branch/temporary_data/splunkIndex02/develop_branch/archive_data/my_cluster/splunkIndex02/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/CSV, dst=/develop_branch/archive_data/my_cluster/splunkIndex02/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/CSV]" 2014-05-14 18:10:30,995 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fdevelop_branch%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200 2014-05-14 18:10:30,995 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fdevelop_branch%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message 2014-05-14 18:10:31,086 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2F' - Unexpected response code 403, expected 200 2014-05-14 18:10:31,086 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2F' - Received error response with XML message 2014-05-14 18:10:31,087 ERROR com.splunk.shuttl.archiver.filesystem.transaction.AbstractTransaction: did="Tried making directories up to: /develop_branch/temporary_data/splunkIndex02/develop_branch/archive_data/my_cluster/splunkIndex02/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/CSV" happened="org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY" expected="To make directories." path="/develop_branch/temporary_data/splunkIndex02/develop_branch/archive_data/my_cluster/splunkIndex02/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/CSV" 2014-05-14 18:10:31,087 ERROR com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: did="Executed transaction" happened="com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY" expected="Transaction to prepare and commit" transaction="Transaction [data=LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 23 16:55:24 CEST 2014, getLatest()=Sun Apr 27 02:17:01 CEST 2014, getSize()=132526], remoteTemp=/develop_branch/temporary_data/splunkIndex02/develop_branch/archive_data/my_cluster/splunkIndex02/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/CSV, dst=/develop_branch/archive_data/my_cluster/splunkIndex02/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/CSV]" 2014-05-14 18:10:31,179 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fdevelop_branch%2Ftemporary_data%2FsplunkIndex02%2Fdevelop_branch%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200 2014-05-14 18:10:31,179 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fdevelop_branch%2Ftemporary_data%2FsplunkIndex02%2Fdevelop_branch%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message 2014-05-14 18:10:31,180 ERROR com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: did="Executed a bucket transaction." happened="com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY" expected="To transfer the bucket to the archive." bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 23 16:55:24 CEST 2014, getLatest()=Sun Apr 27 02:17:01 CEST 2014, getSize()=132526]" 2014-05-14 18:10:31,181 ERROR com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint: did="Tried archiving a bucket" happened="java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY]" expected="To archive the bucket" index="mytestdb" bucket_path="/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82" 2014-05-14 18:10:31,181 ERROR com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint: did="Tried archiving bucket" happened="java.lang.RuntimeException: java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY]" expected="to archive the bucket" path="/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1398557821_1398264924_27_B95021DE-89AB-4A9D-B924-575736C54B82" index="mytestdb" 2014-05-14 18:10:31,182 WARN org.eclipse.jetty.servlet.ServletHandler: /shuttl/rest/archiver/bucket/archive java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY] at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:73) at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:167) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:594) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:485) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:412) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:451) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:931) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY] at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.shuttlBucket(ShuttlBucketEndpoint.java:76) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpointHelper.shuttlBucket(ShuttlBucketEndpointHelper.java:33) at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:64) ... 40 more Caused by: java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY] at com.splunk.shuttl.archiver.archive.BucketCopier.copyBucket(BucketCopier.java:55) at com.splunk.shuttl.archiver.archive.BucketArchiver.archiveBucket(BucketArchiver.java:34) at com.splunk.shuttl.archiver.archive.BucketArchiver.shuttlBucket(BucketArchiver.java:40) at com.splunk.shuttl.archiver.archive.BucketShuttlerRunner.archiveBucket(BucketShuttlerRunner.java:101) at com.splunk.shuttl.archiver.archive.BucketShuttlerRunner.archiveBucketsIfLocksAreAcquired(BucketShuttlerRunner.java:94) at com.splunk.shuttl.archiver.archive.BucketShuttlerRunner.run(BucketShuttlerRunner.java:71) at java.lang.Thread.run(Thread.java:744) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.runShuttlerOnASeparateThread(ShuttlBucketEndpoint.java:111) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.shuttlWithLocks(ShuttlBucketEndpoint.java:102) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.createAndRunBucketShuttling(ShuttlBucketEndpoint.java:94) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.shuttlBucket(ShuttlBucketEndpoint.java:72) ... 42 more 2014-05-14 18:10:31,182 WARN org.eclipse.jetty.servlet.ServletHandler: /shuttl/rest/archiver/bucket/archive java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY] at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:73) at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:167) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:594) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:485) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:412) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:451) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:931) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY] at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.shuttlBucket(ShuttlBucketEndpoint.java:76) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpointHelper.shuttlBucket(ShuttlBucketEndpointHelper.java:33) at com.splunk.shuttl.server.mbeans.rest.ArchiveBucketEndpoint.archiveBucket(ArchiveBucketEndpoint.java:64) ... 40 more Caused by: java.lang.RuntimeException: Got some exceptions when copying bucket: [com.splunk.shuttl.archiver.archive.FailedToArchiveBucketException: com.splunk.shuttl.archiver.filesystem.transaction.TransactionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidObjectStateThe operation is not valid for the object's storage class3A4D763EE819CFE4nOWlHTwtcDeaTylnGvoMYq5FC9C96hHMEM17CL0IC7GOX15xy05AmD6xoddM/2cY] at com.splunk.shuttl.archiver.archive.BucketCopier.copyBucket(BucketCopier.java:55) at com.splunk.shuttl.archiver.archive.BucketArchiver.archiveBucket(BucketArchiver.java:34) at com.splunk.shuttl.archiver.archive.BucketArchiver.shuttlBucket(BucketArchiver.java:40) at com.splunk.shuttl.archiver.archive.BucketShuttlerRunner.archiveBucket(BucketShuttlerRunner.java:101) at com.splunk.shuttl.archiver.archive.BucketShuttlerRunner.archiveBucketsIfLocksAreAcquired(BucketShuttlerRunner.java:94) at com.splunk.shuttl.archiver.archive.BucketShuttlerRunner.run(BucketShuttlerRunner.java:71) at java.lang.Thread.run(Thread.java:744) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.runShuttlerOnASeparateThread(ShuttlBucketEndpoint.java:111) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.shuttlWithLocks(ShuttlBucketEndpoint.java:102) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.createAndRunBucketShuttling(ShuttlBucketEndpoint.java:94) at com.splunk.shuttl.server.mbeans.rest.ShuttlBucketEndpoint.shuttlBucket(ShuttlBucketEndpoint.java:72) ... 42 more