search

Backblaze / b2-sdk-java

The official Java SDK for using Backblaze's B2 Storage APIs
Other
93 stars 26 forks source link

B2Exception 500 Connection pool shut down #146

Closed NikitaZukowski closed 3 years ago

NikitaZukowski commented 3 years ago

Hello there

I get a strange error when I attempt to download multiple files at once:

<B2Exception 500 unexpected: unexpected: java.lang.IllegalStateException: Connection pool shut down>

The problem with this exception is that it appears occasionally when I call the download method.

What I do is following:

import java.util.concurrent.*;

...

ExecutorService executorService = Executors.newFixedThreadPool(b2DownloadFiles.size()); // I create a thread for every file to download, to do it in parallel
CompletionService<B2DownloadedFile> executorCompletionService = new ExecutorCompletionService<>(executorService);
try {
    List<Future<B2DownloadedFile>> futureList = b2DownloadFiles.stream()
            .map(b2DownloadFile -> new DownloadFileCallable(fileDownloader, client, b2DownloadFile))
            .map(downloadFileCallable -> executorCompletionService.submit(downloadFileCallable))
            .collect(Collectors.toList());

    for (int i = 0; i < futureList.size(); i++) {
        B2DownloadedFile b2DownloadedFile = executorCompletionService.take().get();
        b2DownloadedFiles.add(b2DownloadedFile);
    }
} catch (InterruptedException | ExecutionException e) {
    throw new B2Exception(B2Exception.ERROR_DOWNLOADING_FILE, e);
} finally {
    executorService.shutdown();
}

The DownloadFileCallable wraps the call for the client:

public class DownloadFileCallable implements Callable<B2DownloadedFile> {
    private final FileDownloader fileDownloader;
    private final B2StorageClient client;
    private final B2DownloadFile b2DownloadFile;

    public DownloadFileCallable(FileDownloader fileDownloader, B2StorageClient client, B2DownloadFile b2DownloadFile) {
        this.fileDownloader = fileDownloader;
        this.client = client;
        this.b2DownloadFile = b2DownloadFile;
    }

    @Override
    public B2DownloadedFile call() throws B2Exception {
        return fileDownloader.downloadFileById(client, b2DownloadFile);
    }
}

The FileDownloader calls the client:

public class FileDownloader {
    public B2DownloadedFile downloadFileById(B2StorageClient client, B2DownloadFile b2DownloadFile) throws B2Exception {
        B2DownloadByIdRequest request = B2DownloadByIdRequest.builder(b2DownloadFile.getExternalFileId())
                .build();

        B2ContentMemoryWriter b2ContentMemoryWriter = B2ContentMemoryWriter.builder()
                .setVerifySha1ByRereadingFromDestination(true)
                .build();

        try {
            client.downloadById(request, b2ContentMemoryWriter);
        } catch (com.backblaze.b2.client.exceptions.B2Exception e) {
            String message = String.format(B2Exception.ERROR_DOWNLOADING_FILE, e.toString());
            throw new B2Exception(message, e);
        }

        return B2DownloadedFile.builder()
                .internalId(b2DownloadFile.getInternalId())
                .bytes(b2ContentMemoryWriter.getBytes())
                .externalFileId(b2DownloadFile.getExternalFileId())
                .externalFileName(b2DownloadFile.getExternalFileName())
                .build();
    }
}

After some debug sessions I found out that this is the actual error B2NetworkException STATUS 904:

<B2Exception 904 io_exception: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake talking to https://api.backblazeb2.com/b2api/v2/b2_authorize_account>
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.translateToB2Exception(B2WebApiHttpClientImpl.java:256)
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.postAndReturnString(B2WebApiHttpClientImpl.java:223)
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.postJsonAndReturnString(B2WebApiHttpClientImpl.java:185)
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.postJsonReturnJson(B2WebApiHttpClientImpl.java:71)
    at com.backblaze.b2.client.B2StorageClientWebifierImpl.authorizeAccount(B2StorageClientWebifierImpl.java:141)
    at com.backblaze.b2.client.B2AccountAuthorizerSimpleImpl.authorize(B2AccountAuthorizerSimpleImpl.java:38)
    at com.backblaze.b2.client.B2AccountAuthorizationCache.get(B2AccountAuthorizationCache.java:50)
    at com.backblaze.b2.client.B2StorageClientImpl.lambda$downloadById$6(B2StorageClientImpl.java:295)
    at com.backblaze.b2.client.B2Retryer.lambda$doRetry$0(B2Retryer.java:45)
    at com.backblaze.b2.client.B2Retryer.doRetry(B2Retryer.java:85)
    at com.backblaze.b2.client.B2Retryer.doRetry(B2Retryer.java:45)
    at com.backblaze.b2.client.B2StorageClientImpl.downloadById(B2StorageClientImpl.java:292)
    at les.external.modules.backblaze.b2.businesslogic.download.FileDownloader.downloadFileById(FileDownloader.java:22)
    at les.external.modules.backblaze.b2.businesslogic.workflow.DownloadFileCallable.call(DownloadFileCallable.java:24)
    at les.external.modules.backblaze.b2.businesslogic.workflow.DownloadFileCallable.call(DownloadFileCallable.java:11)
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
    at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1588)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1416)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.postAndReturnString(B2WebApiHttpClientImpl.java:211)
    ... 21 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483)
    at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
    at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
    ... 37 more

and this

<B2Exception 904 io_exception: javax.net.ssl.SSLException: Socket closed talking to https://f000.backblazeb2.com/b2api/v2/b2_download_file_by_id?fileId=4_zb195509e84a1cc896011061f_f10638790e039c88d_d20210305_m182042_c000_v0001078_t0024>
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.translateToB2Exception(B2WebApiHttpClientImpl.java:256)
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.getContent(B2WebApiHttpClientImpl.java:127)
    at com.backblaze.b2.client.B2StorageClientWebifierImpl.downloadGuts(B2StorageClientWebifierImpl.java:428)
    at com.backblaze.b2.client.B2StorageClientWebifierImpl.downloadById(B2StorageClientWebifierImpl.java:392)
    at com.backblaze.b2.client.B2StorageClientImpl.lambda$downloadById$6(B2StorageClientImpl.java:296)
    at com.backblaze.b2.client.B2Retryer.lambda$doRetry$0(B2Retryer.java:45)
    at com.backblaze.b2.client.B2Retryer.doRetry(B2Retryer.java:85)
    at com.backblaze.b2.client.B2Retryer.doRetry(B2Retryer.java:45)
    at com.backblaze.b2.client.B2StorageClientImpl.downloadById(B2StorageClientImpl.java:292)
    at les.external.modules.backblaze.b2.businesslogic.download.FileDownloader.downloadFileById(FileDownloader.java:22)
    at les.external.modules.backblaze.b2.businesslogic.workflow.DownloadFileCallable.call(DownloadFileCallable.java:24)
    at les.external.modules.backblaze.b2.businesslogic.workflow.DownloadFileCallable.call(DownloadFileCallable.java:11)
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLException: Socket closed
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:144)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at com.backblaze.b2.client.webApiHttpClient.B2WebApiHttpClientImpl.getContent(B2WebApiHttpClientImpl.java:105)
    ... 18 more
    Suppressed: java.net.SocketException: Socket closed
        at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
        at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:380)
        ... 38 more
Caused by: java.net.SocketException: Socket closed
    at java.base/java.net.SocketInputStream.socketRead0(Native Method)
    at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
    at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
    at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
    ... 35 more

Have you any idea how this can be fixed?

malaysf commented 3 years ago

Hi @NikitaZukowski thanks for filing this issue and providing the helpful details. I have a few questions that would help us understand the issue better.

Do you continue to run into this regularly? Could you share some timestamps from when you encountered this? Do you have an HTTP proxy configured?

This appears to be a network connectivity issue, however one that should have been retried by the SDK before throwing an exception.

NikitaZukowski commented 3 years ago

Yes, I run in it regularly. Not every time I call it. It is like -x--x-xx-x (where x is a failure). I do not have any proxies, I call the service from a localhost app. The retryAfterSecondsOrNull is always null.

NikitaZukowski commented 3 years ago

Can I provide more information for this issue? I just don't know where to start.

ericjding commented 3 years ago

Hi @NikitaZukowski, it would be helpful if you could give us some logs of when you encounter the issue, along with specific timestamps (in UTC or, if not, with your timezone noted so that we can compare with our internal logs).

Also, are the failures consistently repeatable with the same file over and over again? Or is it unpredictable?

NikitaZukowski commented 3 years ago

I wanted to create a github repository to present the error. Thanks to it I found out that the mistake was the wrong order of opening and closing clients. Now, each method accessing the store directly opens and closes its client object itself. So this problem is solved.

Could you give me an advice if it is a good idea to keep connection alive a long time, or close them as soon as possible? Is there a max connection setting?

ericjding commented 3 years ago

It depends on your situation and how your code is structured, but we encourage you to re-use the same client whenever possible. It is thread-safe, so you should be able to use the same one in multi-threaded situations as well. Reusing the same client does not mean it keeps an HTTPS connection alive indefinitely.

NikitaZukowski commented 3 years ago

Okay, thanks!