search

cryptomator / android

Cryptomator for Android
https://cryptomator.org
GNU General Public License v3.0
690 stars 114 forks source link

Error "No Network Connection" on android 11 while uploading multiple files #370

Open pawanrai9999 opened 2 years ago

pawanrai9999 commented 2 years ago

Please agree to the following

Summary

I was trying to upload multiple .jpg image to my S3 Vault. Then it gave me error "No Network Connection" but when I try to upload a single file it uploads successfully. I am confirmed there is not issue with network.

System Setup

- Android: 11
- Cryptomator: 1.6.1

Cloud Type

S3

Steps to Reproduce

  1. Unlock a vault
  2. Click on upload file option
  3. Select multiple files and wait

Expected Behavior

Cryptomator should upload file and after uploading the file it should update the available file in folder.

Actual Behavior

Cryptomator will show a toast "No Network Connection".

Reproducibility

Intermittent

Relevant Log Output

D   20211028221002.916  ExceptionHandler    Unexpected error
org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
    at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
    at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14935)
    at io.reactivex.Flowable.subscribe(Flowable.java:14882)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
    ... 22 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
    at okio.OutputStreamSink.write(JvmOkio.kt:53)
    at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
    at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
    at okio.ForwardingSink.write(ForwardingSink.kt:29)
    at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
    at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
    at io.minio.S3Base.execute(S3Base.java:532)
    at io.minio.S3Base.uploadPart(S3Base.java:2159)
    at io.minio.S3Base.putObject(S3Base.java:1175)
    at io.minio.MinioClient.putObject(MinioClient.java:1666)
    at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
    ... 24 more

ErrorCode: 10UH:OG5O
D   20211028221002.919  ExceptionHandler    org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
    at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
    at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14935)
    at io.reactivex.Flowable.subscribe(Flowable.java:14882)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
    ... 22 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
    at okio.OutputStreamSink.write(JvmOkio.kt:53)
    at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
    at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
    at okio.ForwardingSink.write(ForwardingSink.kt:29)
    at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
    at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
    at io.minio.S3Base.execute(S3Base.java:532)
    at io.minio.S3Base.uploadPart(S3Base.java:2159)
    at io.minio.S3Base.putObject(S3Base.java:1175)
    at io.minio.MinioClient.putObject(MinioClient.java:1666)
    at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
    ... 24 more

ErrorCode: 10UH:OG5O
W   20211028221002.926  Message No network connection

Anything else?

I am using Backblaze. Automatic photo upload is active. Cache size is 100 MB Keep vaults unlocked while editing files : ON

pawanrai9999 commented 2 years ago 
D   20211029091725.286  UploadFilesUseCase  failed 4a158cd9
D   20211029091725.313  ExceptionHandler    Unexpected error
org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
    at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
    at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14935)
    at io.reactivex.Flowable.subscribe(Flowable.java:14882)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
    ... 22 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
    at okio.OutputStreamSink.write(JvmOkio.kt:53)
    at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
    at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
    at okio.ForwardingSink.write(ForwardingSink.kt:29)
    at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
    at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.provideOfflineCacheInterceptor$lambda-1(S3ClientFactory.kt:79)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.$r8$lambda$ZlZNFkypEyH2ozaExGLRHlQReU8(Unknown Source:0)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion$$ExternalSyntheticLambda0.intercept(Unknown Source:2)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
    at io.minio.S3Base.execute(S3Base.java:532)
    at io.minio.S3Base.uploadPart(S3Base.java:2159)
    at io.minio.S3Base.putObject(S3Base.java:1175)
    at io.minio.MinioClient.putObject(MinioClient.java:1666)
    at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
    ... 24 more

ErrorCode: 10UH:16N9
D   20211029091725.315  ExceptionHandler    org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
    at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
    at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14935)
    at io.reactivex.Flowable.subscribe(Flowable.java:14882)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
    ... 22 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeToSocket(ConscryptEngineSocket.java:715)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.writeInternal(ConscryptEngineSocket.java:689)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLOutputStream.write(ConscryptEngineSocket.java:652)
    at okio.OutputStreamSink.write(JvmOkio.kt:53)
    at okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:103)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:146)
    at okhttp3.internal.http1.Http1ExchangeCodec$KnownLengthSink.write(Http1ExchangeCodec.kt:271)
    at okio.ForwardingSink.write(ForwardingSink.kt:29)
    at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:203)
    at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:57)
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.provideOfflineCacheInterceptor$lambda-1(S3ClientFactory.kt:79)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.$r8$lambda$ZlZNFkypEyH2ozaExGLRHlQReU8(Unknown Source:0)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion$$ExternalSyntheticLambda0.intercept(Unknown Source:2)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
    at io.minio.S3Base.execute(S3Base.java:532)
    at io.minio.S3Base.uploadPart(S3Base.java:2159)
    at io.minio.S3Base.putObject(S3Base.java:1175)
    at io.minio.MinioClient.putObject(MinioClient.java:1666)
    at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
    ... 24 more

ErrorCode: 10UH:16N9
W   20211029091725.328  Message No network connection
D   20211029091725.363  AssertUnlockedUseCase   started 4a158cf4
D   20211029091725.364  AssertUnlockedUseCase   finished 4a158cf4
pawanrai9999 commented 2 years ago

After digging some more I found this is happening with file sizes larger than 200 MB.

SailReal commented 2 years ago

Can you please enable debug mode and reproduce the problem? Then we also see each request and response, this should bring some light into this.

Is it possible that you set e.g. an upload limit to files?

Can you please try to upload a file of this size using a different S3 client. Does the problem also occur with it?

I just tested it myself and unfortunately can not reproduce it.

pawanrai9999 commented 2 years ago

1.txt 0.txt

  1. Above are the links to my log file.
  2. There is no upload limit set on files. I tried smaller files and they are uploaded anything less than 200 MB.
  3. Yes I was successfully able to upload files of much larger size from different clients (BucketAnywhere for S3)

P.S. Error is in 1.txt at timestamp 20211029192152.373

SailReal commented 2 years ago

Hmm there are two pretty strange response types in the log file. There is a possibility that they may or may not be the same or similar problems.

One problem is that it looks like the server is under heavy load and responds then with a 503 (ServiceUnavailable) HTTP status code while uploading a chunk. While recovery we delete the file again which works but we could also handle those 503 status codes using a exponential backoff algorithm and retry those requests and only give up when reaching a retry counter limit.

D   20211029100110.982  OkHttp  --> POST https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploads= http/1.1 (0-byte body)
D   20211029100110.982  OkHttp  Content-Type: application/octet-stream
V   20211029100110.982  Progress    UPLOAD 11%
D   20211029100110.982  OkHttp  Content-Length: 0
D   20211029100110.982  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029100110.982  OkHttp  Accept-Encoding: identity
D   20211029100110.983  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029100110.983  OkHttp  Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D   20211029100110.983  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029100110.983  OkHttp  x-amz-date: 20211029T043110Z
D   20211029100110.983  OkHttp  --> END POST
D   20211029100111.326  OkHttp  <-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploads= (343ms)
D   20211029100111.327  OkHttp  x-amz-request-id: 884b41516d2721ac
D   20211029100111.328  OkHttp  x-amz-id-2: aOJJibTcgMEg3jWELZak5eWSfNHg3jzmY
D   20211029100111.328  OkHttp  x-amz-version-id: 4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029
D   20211029100111.328  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029100111.328  OkHttp  Date: Fri, 29 Oct 2021 04:31:11 GMT
D   20211029100111.328  OkHttp  Keep-Alive: timeout=5
D   20211029100111.328  OkHttp  Connection: keep-alive
D   20211029100111.328  OkHttp  <-- END HTTP
D   20211029100111.341  OkHttp  --> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1 http/1.1 (5242880-byte body)
D   20211029100111.342  OkHttp  Content-Type: application/octet-stream
D   20211029100111.342  OkHttp  Content-Length: 5242880
D   20211029100111.342  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029100111.342  OkHttp  Accept-Encoding: identity
D   20211029100111.342  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029100111.342  OkHttp  Content-MD5: 8YloTdnfiR5kF45tHoUKLQ==
D   20211029100111.342  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029100111.342  OkHttp  x-amz-date: 20211029T043111Z
D   20211029100111.343  OkHttp  --> END PUT
D   20211029100114.436  OkHttp  <-- 503  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1 (3093ms)
D   20211029100114.438  OkHttp  x-amz-request-id: fd487dd29df7b560
D   20211029100114.439  OkHttp  x-amz-id-2: aOHFiRjdzMNM3nGGqZck5I2S+NNc3oTkf
D   20211029100114.439  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029100114.439  OkHttp  Date: Fri, 29 Oct 2021 04:31:12 GMT
D   20211029100114.439  OkHttp  Connection: close
D   20211029100114.439  OkHttp  <-- END HTTP
D   20211029100114.463  OkHttp  --> DELETE https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029 http/1.1 (unknown length)
D   20211029100114.465  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029100114.465  OkHttp  Accept-Encoding: identity
D   20211029100114.465  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029100114.465  OkHttp  Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D   20211029100114.465  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029100114.466  OkHttp  x-amz-date: 20211029T043114Z
D   20211029100114.466  OkHttp  --> END DELETE
D   20211029100116.296  OkHttp  <-- 204  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029 (1830ms)
D   20211029100116.298  OkHttp  x-amz-request-id: 365d72af6a5c723b
D   20211029100116.299  OkHttp  x-amz-id-2: aOOtiLzfzMGM34WGbZeg5ZmQUNB83kjnZ
D   20211029100116.299  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029100116.299  OkHttp  Date: Fri, 29 Oct 2021 04:31:16 GMT
D   20211029100116.299  OkHttp  Keep-Alive: timeout=5
D   20211029100116.299  OkHttp  Connection: keep-alive
D   20211029100116.300  OkHttp  <-- END HTTP
D   20211029100116.335  UploadFilesUseCase  failed 4a505e27
D   20211029100116.363  ExceptionHandler    Unexpected error
org.cryptomator.domain.exception.FatalBackendException: error occurred
ErrorResponse(code = ServiceUnavailable, message = no tomes available, bucketName = null, objectName = null, resource = null, requestId = null, hostId = null)
request={method=PUT, url=https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/5N/QMPGGS66NU7IXVU5F3HUKCO5D6Z2NI/Z7bEcdrWQBenXtg26YdNNFp2h15cjvpPTssTJcP3bWsSB4DP5ZI7.c9r?uploadId=4_z884b4720f79abe797dc40719_f202b5a9a01489e91_d20211029_m043112_c000_v0001082_t0029&partNumber=1, headers=Host: s3.us-west-000.backblazeb2.com
Accept-Encoding: identity
User-Agent: MinIO (Linux; aarch64) minio-java/dev
Content-MD5: 8YloTdnfiR5kF45tHoUKLQ==
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20211029T043111Z
Authorization: ██
Cache-Control: max-age=0
}
response={code=503, headers=x-amz-request-id: fd487dd29df7b560
x-amz-id-2: aOHFiRjdzMNM3nGGqZck5I2S+NNc3oTkf
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Content-Length: 151
Date: Fri, 29 Oct 2021 04:31:12 GMT
Connection: close
}

    at io.minio.S3Base.execute(S3Base.java:670)
    at io.minio.S3Base.uploadPart(S3Base.java:2159)
    at io.minio.S3Base.putObject(S3Base.java:1175)
    at io.minio.MinioClient.putObject(MinioClient.java:1666)
    at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
    ... 25 more

ErrorCode: 2B6R:5GJ3

The other problem is that sometimes the server completely terminates the connection, I might be wrong but probably for the same reason (heavy load). Perhaps the exponential backoff for the 503 status codes is already enough for the server to no longer disconnect.

D   20211029192138.138  OkHttp  --> HEAD https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r http/1.1 (unknown length)
D   20211029192138.139  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192138.139  OkHttp  Accept-Encoding: identity
D   20211029192138.139  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192138.140  OkHttp  Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D   20211029192138.141  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192138.141  OkHttp  x-amz-date: 20211029T135138Z
D   20211029192138.141  OkHttp  --> END HEAD
D   20211029192138.501  OkHttp  <-- 404  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r (359ms)
D   20211029192138.502  OkHttp  x-amz-request-id: c436ebaa73234416
D   20211029192138.503  OkHttp  x-amz-id-2: aODJiajcaMCg36mHsZcs5iWQ1NAw32znD
D   20211029192138.504  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029192138.505  OkHttp  Date: Fri, 29 Oct 2021 13:51:38 GMT
D   20211029192138.505  OkHttp  Keep-Alive: timeout=5
D   20211029192138.505  OkHttp  Connection: keep-alive
D   20211029192138.505  OkHttp  <-- END HTTP
D   20211029192138.690  OkHttp  --> POST https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploads= http/1.1 (0-byte body)
D   20211029192138.690  OkHttp  Content-Type: application/octet-stream
D   20211029192138.690  OkHttp  Content-Length: 0
D   20211029192138.690  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192138.690  OkHttp  Accept-Encoding: identity
D   20211029192138.690  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192138.690  OkHttp  Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D   20211029192138.690  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192138.690  OkHttp  x-amz-date: 20211029T135138Z
D   20211029192138.691  OkHttp  --> END POST
D   20211029192139.057  OkHttp  <-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploads= (366ms)
D   20211029192139.058  OkHttp  x-amz-request-id: ca8ea16fe380bca1
D   20211029192139.059  OkHttp  x-amz-id-2: aOD5ijDckMEc3x2FMZSE5Q2SUNGA3MzkO
D   20211029192139.059  OkHttp  x-amz-version-id: 4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001
D   20211029192139.059  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029192139.059  OkHttp  Date: Fri, 29 Oct 2021 13:51:38 GMT
D   20211029192139.059  OkHttp  Keep-Alive: timeout=5
D   20211029192139.059  OkHttp  Connection: keep-alive
D   20211029192139.059  OkHttp  <-- END HTTP
D   20211029192139.073  OkHttp  --> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=1 http/1.1 (5242880-byte body)
D   20211029192139.074  OkHttp  Content-Type: application/octet-stream
D   20211029192139.074  OkHttp  Content-Length: 5242880
D   20211029192139.074  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192139.074  OkHttp  Accept-Encoding: identity
D   20211029192139.074  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192139.075  OkHttp  Content-MD5: fq1azAcy9MADGOuVPeQ28g==
D   20211029192139.075  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192139.075  OkHttp  x-amz-date: 20211029T135139Z
D   20211029192139.075  OkHttp  --> END PUT
D   20211029192142.404  OkHttp  <-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=1 (3328ms)
D   20211029192142.405  OkHttp  x-amz-request-id: d3313547033b755a
D   20211029192142.406  OkHttp  x-amz-id-2: aOMhiUzfyMJA3OWHwZf05I2QRNEs3czkj
D   20211029192142.406  OkHttp  ETag: "7ead5acc0732f4c00318eb953de436f2"
D   20211029192142.407  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029192142.407  OkHttp  Date: Fri, 29 Oct 2021 13:51:42 GMT
D   20211029192142.407  OkHttp  Keep-Alive: timeout=5
D   20211029192142.408  OkHttp  Connection: keep-alive
D   20211029192142.408  OkHttp  <-- END HTTP
D   20211029192142.551  OkHttp  --> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=2 http/1.1 (5242880-byte body)
D   20211029192142.551  OkHttp  Content-Type: application/octet-stream
D   20211029192142.551  OkHttp  Content-Length: 5242880
D   20211029192142.551  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192142.551  OkHttp  Accept-Encoding: identity
D   20211029192142.551  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192142.552  OkHttp  Content-MD5: 75SEsyqJUSyZAPkO+laAsQ==
D   20211029192142.552  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192142.552  OkHttp  x-amz-date: 20211029T135142Z
D   20211029192142.552  OkHttp  --> END PUT
D   20211029192145.270  OkHttp  <-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=2 (2717ms)
D   20211029192145.272  OkHttp  x-amz-request-id: 7eae625a095d2537
D   20211029192145.272  OkHttp  x-amz-id-2: aOFZiAzdNMFA3dGHwZWk5aWQvNFM3wjnm
D   20211029192145.272  OkHttp  ETag: "ef9484b32a89512c9900f90efa5680b1"
D   20211029192145.273  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029192145.273  OkHttp  Date: Fri, 29 Oct 2021 13:51:44 GMT
D   20211029192145.273  OkHttp  Keep-Alive: timeout=5
D   20211029192145.273  OkHttp  Connection: keep-alive
D   20211029192145.274  OkHttp  <-- END HTTP
D   20211029192145.413  OkHttp  --> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=3 http/1.1 (5242880-byte body)
D   20211029192145.413  OkHttp  Content-Type: application/octet-stream
D   20211029192145.413  OkHttp  Content-Length: 5242880
D   20211029192145.413  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192145.413  OkHttp  Accept-Encoding: identity
D   20211029192145.413  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192145.414  OkHttp  Content-MD5: bmkA7uZvXTiyUMYLqtuEuA==
D   20211029192145.414  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192145.414  OkHttp  x-amz-date: 20211029T135145Z
D   20211029192145.414  OkHttp  --> END PUT
D   20211029192148.650  OkHttp  <-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=3 (3235ms)
D   20211029192148.651  OkHttp  x-amz-request-id: 2268e2fd86dddbd7
D   20211029192148.652  OkHttp  x-amz-id-2: aOOZiDjcyMNg3FWFoZeQ5eWRTNGg3QTmE
D   20211029192148.653  OkHttp  ETag: "6e6900eee66f5d38b250c60baadb84b8"
D   20211029192148.653  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029192148.654  OkHttp  Date: Fri, 29 Oct 2021 13:51:47 GMT
D   20211029192148.654  OkHttp  Keep-Alive: timeout=5
D   20211029192148.654  OkHttp  Connection: keep-alive
D   20211029192148.654  OkHttp  <-- END HTTP
D   20211029192148.760  OkHttp  --> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=4 http/1.1 (5242880-byte body)
D   20211029192148.760  OkHttp  Content-Type: application/octet-stream
D   20211029192148.761  OkHttp  Content-Length: 5242880
D   20211029192148.761  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192148.761  OkHttp  Accept-Encoding: identity
D   20211029192148.761  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192148.761  OkHttp  Content-MD5: BgOtBD9qdhqzsy91HhksyA==
D   20211029192148.761  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192148.761  OkHttp  x-amz-date: 20211029T135148Z
D   20211029192148.761  OkHttp  --> END PUT
D   20211029192151.109  OkHttp  <-- 200  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=4 (2348ms)
D   20211029192151.111  OkHttp  x-amz-request-id: ad2d614a1c2b8ce2
D   20211029192151.112  OkHttp  x-amz-id-2: aOG1i7TeNMEs3jWHRZUM5wmT1NJs3WTmc
D   20211029192151.112  OkHttp  ETag: "0603ad043f6a761ab3b32f751e192cc8"
D   20211029192151.112  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029192151.112  OkHttp  Date: Fri, 29 Oct 2021 13:51:50 GMT
D   20211029192151.113  OkHttp  Keep-Alive: timeout=5
D   20211029192151.113  OkHttp  Connection: keep-alive
D   20211029192151.113  OkHttp  <-- END HTTP
D   20211029192151.258  OkHttp  --> PUT https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001&partNumber=5 http/1.1 (5242880-byte body)
D   20211029192151.259  OkHttp  Content-Type: application/octet-stream
D   20211029192151.259  OkHttp  Content-Length: 5242880
D   20211029192151.259  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192151.259  OkHttp  Accept-Encoding: identity
D   20211029192151.259  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192151.259  OkHttp  Content-MD5: 92J8ozZqevkBmadcXQ1YwA==
D   20211029192151.259  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192151.259  OkHttp  x-amz-date: 20211029T135151Z
D   20211029192151.259  OkHttp  --> END PUT
D   20211029192152.373  OkHttp  <-- HTTP FAILED: java.net.SocketException: Connection reset
D   20211029192152.386  OkHttp  --> DELETE https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001 http/1.1 (unknown length)
D   20211029192152.387  OkHttp  Host: s3.us-west-000.backblazeb2.com
D   20211029192152.388  OkHttp  Accept-Encoding: identity
D   20211029192152.389  OkHttp  User-Agent: MinIO (Linux; aarch64) minio-java/dev
D   20211029192152.389  OkHttp  Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
D   20211029192152.389  OkHttp  x-amz-content-sha256: UNSIGNED-PAYLOAD
D   20211029192152.389  OkHttp  x-amz-date: 20211029T135152Z
D   20211029192152.389  OkHttp  --> END DELETE
D   20211029192154.795  OkHttp  <-- 204  https://s3.us-west-000.backblazeb2.com/crypt9999/cryptomator/crypt9999/d/LQ/FYIZBVQKETLS2DCNRM6PAGT5W2US2F/4HOzOJgrlrWRRSFst0EaL6KUeFMXapOScnUdFMBeZGCk39eB3Qpq.c9r?uploadId=4_z884b4720f79abe797dc40719_f2050cf70a6bc162b_d20211029_m135139_c000_v0001088_t0001 (2405ms)
D   20211029192154.797  OkHttp  x-amz-request-id: dfd418d42dc54fa5
D   20211029192154.798  OkHttp  x-amz-id-2: aOJFiIjfcMCY3f2HwZRU5jGQ+NB83BDlA
D   20211029192154.798  OkHttp  Cache-Control: max-age=0, no-cache, no-store
D   20211029192154.798  OkHttp  Date: Fri, 29 Oct 2021 13:51:55 GMT
D   20211029192154.799  OkHttp  Keep-Alive: timeout=5
D   20211029192154.799  OkHttp  Connection: keep-alive
D   20211029192154.799  OkHttp  <-- END HTTP
D   20211029192154.919  UploadFilesUseCase  failed 4c519f94
D   20211029192154.950  ExceptionHandler    Unexpected error
org.cryptomator.domain.exception.NetworkConnectionException: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwConnectionErrorIfRequired(S3CloudContentRepository.kt:47)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository.throwWrappedIfRequired(S3CloudContentRepository.kt:30)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:160)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeFromTmpFile(CryptoImplDecorator.kt:257)
    at org.cryptomator.data.cloud.crypto.CryptoImplDecorator.writeShortNameFile(CryptoImplDecorator.kt:443)
    at org.cryptomator.data.cloud.crypto.CryptoImplVaultFormat7.write(CryptoImplVaultFormat7.kt:439)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:90)
    at org.cryptomator.data.cloud.crypto.CryptoCloudContentRepository.write(CryptoCloudContentRepository.kt:21)
    at org.cryptomator.data.repository.DispatchingCloudContentRepository.write(DispatchingCloudContentRepository.kt:149)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.writeCloudFile(UploadFiles.java:117)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:94)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:82)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.upload(UploadFiles.java:74)
    at org.cryptomator.domain.usecases.cloud.UploadFiles.execute(UploadFiles.java:61)
    at org.cryptomator.domain.usecases.cloud.UploadFilesUseCase$Launcher$2.subscribe(UploadFilesUseCase.java:119)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14935)
    at io.reactivex.Flowable.subscribe(Flowable.java:14882)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:923)
Caused by: org.cryptomator.domain.exception.FatalBackendException: java.net.SocketException: Connection reset
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:143)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:62)
    at org.cryptomator.data.cloud.InterceptingCloudContentRepository.write(InterceptingCloudContentRepository.kt:155)
    ... 22 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:215)
    at java.net.SocketInputStream.read(SocketInputStream.java:144)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readFromSocket(ConscryptEngineSocket.java:936)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:900)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readUntilDataAvailable(ConscryptEngineSocket.java:815)
    at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.read(ConscryptEngineSocket.java:788)
    at okio.InputStreamSource.read(JvmOkio.kt:90)
    at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
    at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:427)
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:320)
    at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
    at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178)
    at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106)
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.provideOfflineCacheInterceptor$lambda-1(S3ClientFactory.kt:79)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion.$r8$lambda$ZlZNFkypEyH2ozaExGLRHlQReU8(Unknown Source:0)
    at org.cryptomator.data.cloud.s3.S3ClientFactory$Companion$$ExternalSyntheticLambda0.intercept(Unknown Source:2)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getResponseLoggingExceptions(HttpLoggingInterceptor.kt:51)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.getAndLogResponse(HttpLoggingInterceptor.kt:42)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.proceedWithLogging(HttpLoggingInterceptor.kt:28)
    at org.cryptomator.data.cloud.okhttplogging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:18)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
    at io.minio.S3Base.execute(S3Base.java:532)
    at io.minio.S3Base.uploadPart(S3Base.java:2159)
    at io.minio.S3Base.putObject(S3Base.java:1175)
    at io.minio.MinioClient.putObject(MinioClient.java:1666)
    at org.cryptomator.data.cloud.s3.S3Impl.write(S3Impl.kt:247)
    at org.cryptomator.data.cloud.s3.S3CloudContentRepository$Intercepted.write(S3CloudContentRepository.kt:141)
    ... 24 more

ErrorCode: 10UH:3SG0
W   20211029192154.965  Message No network connection
pawanrai9999 commented 2 years ago

While I was reading the documentation of Backblaze I found this:-

Some errors returned from b2 upload file mean that you should get a new upload URL and try again, while others mean that there is a problem with your request and trying again will not help.

These indicate that you should get a new upload URL and try again:

The last one, "broken pipe", happens when you are sending a file big enough that the buffers in the HTTP connection won't hold it. HTTP client libraries send the entire request before looking for a response, and if the B2 server has already replied with an error, you'll be unable to send the entire file and will get a "broken pipe" error.

SailReal commented 2 years ago

Thanks for your amazing research regarding this topic. Can you please also post the source for this because we need to create an issue in the minio-java repository as we use this as the S3 client and it would be great if this error handling would be implemented directly in it.

pawanrai9999 commented 2 years ago

Integration Checklist of backblaze

SailReal commented 2 years ago

Thanks for the source, just created this issue https://github.com/minio/minio-java/issues/1255

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.