search

gaul / s3proxy

Access other storage backends via the S3 API
Apache License 2.0
1.79k stars 232 forks source link

MultipartPart upload failed #722

Closed singhinderjeet0708 closed 4 days ago

singhinderjeet0708 commented 5 days ago

Hi, i've this error message while trying to backup minio files using mc mirror.

But some files failed to be uploaded with this error:

[s3proxy] W 11-20 12:48:21.222 S3Proxy-Jetty-15 o.g.s.o.e.j.server.HttpChannel:797 |::] /miniobackup/test.csv
java.lang.IllegalStateException: Duplicate key 10000 (attempted merging values MultipartPart{partNumber=10000, partSize=16777216, partETag=, lastModified=null} and MultipartPart{partNumber=10000, partSize=16777216, partETag=, lastModified=null})
    at java.base/java.util.stream.Collectors.duplicateKeyException(Unknown Source)
    at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Unknown Source)
    at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source)
    at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
    at org.gaul.s3proxy.S3ProxyHandler.handleCompleteMultipartUpload(S3ProxyHandler.java:2280)
    at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:729)
    at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:80)
    at org.gaul.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.gaul.shaded.org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.gaul.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.gaul.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.gaul.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.gaul.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.gaul.shaded.org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.gaul.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base/java.lang.Thread.run(Unknown Source)

I've tried to upload with azureblob-sdk, but in that case I've this error:

mc: <ERROR> Failed to copy `http://minio:9000/test.csv`. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 501 Not Implemented</title>
</head>
<body><h2>HTTP ERROR 501 Not Implemented</h2>
<table>
<tr><th>URI:</th><td>/miniobackup/test.csv</td></tr>
<tr><th>STATUS:</th><td>501</td></tr>
<tr><th>MESSAGE:</th><td>Not Implemented</td></tr>
<tr><th>SERVLET:</th><td>-</td></tr>
</table>
<hr/><a href="https://jetty.org">Powered by Jetty:// 11.0.23</a><hr/>

</body>
</html>
mc: <ERROR> Unable to list comparison retrying.. context canceled

Do you have any suggestions that I can try to make this work? Thank you

singhinderjeet0708 commented 4 days ago

Anyone facing the same issue when trying to backup with minio, can try with: mc mirror --disable-multipart

For the moment this workaround works. Thanks

gaul commented 4 days ago

Fixed by #727.