S3TransferManager: Could not initialize class software.amazon.awssdk.crt.io.ClientBootstrap

[TheKeveloper]

Describe the bug

When trying to create an S3TransferManager using the S3TransferManager builder, I get a NoClassDefFoundError in a specific runtime.

Expected Behavior

Create an S3TransferManager instance without any exceptions

Current Behavior

Throws a NoClassDefFoundError when trying to build the S3TransferManager.

The full stacktrace is

java.lang.NoClassDefFoundError: Could not initialize class software.amazon.awssdk.crt.io.ClientBootstrap
    at software.amazon.awssdk.transfer.s3.internal.S3NativeClientConfiguration.<init>(S3NativeClientConfiguration.java:60)
    at software.amazon.awssdk.transfer.s3.internal.S3NativeClientConfiguration$Builder.build(S3NativeClientConfiguration.java:208)
    at software.amazon.awssdk.transfer.s3.internal.S3CrtAsyncHttpClient.<init>(S3CrtAsyncHttpClient.java:63)
    at software.amazon.awssdk.transfer.s3.internal.S3CrtAsyncHttpClient.<init>(S3CrtAsyncHttpClient.java:49)
    at software.amazon.awssdk.transfer.s3.internal.S3CrtAsyncHttpClient$Builder.build(S3CrtAsyncHttpClient.java:244)
    at software.amazon.awssdk.transfer.s3.internal.DefaultS3CrtAsyncClient.<init>(DefaultS3CrtAsyncClient.java:66)
    at software.amazon.awssdk.transfer.s3.internal.DefaultS3CrtAsyncClient.<init>(DefaultS3CrtAsyncClient.java:54)
    at software.amazon.awssdk.transfer.s3.internal.DefaultS3CrtAsyncClient$DefaultS3CrtClientBuilder.build(DefaultS3CrtAsyncClient.java:223)
    at software.amazon.awssdk.transfer.s3.internal.DefaultS3TransferManager.initializeS3CrtClient(DefaultS3TransferManager.java:111)
    at software.amazon.awssdk.transfer.s3.internal.DefaultS3TransferManager.<init>(DefaultS3TransferManager.java:75)
    at software.amazon.awssdk.transfer.s3.internal.DefaultS3TransferManager$DefaultBuilder.build(DefaultS3TransferManager.java:425)

Reproduction Steps

I'm honestly not sure how to reproduce this. This doesn't happen with local testing, but only in a very specific production environment. The code for creating the transfer manager is pretty standard

S3TransferManager.builder()
                .s3ClientConfiguration(builder -> 
                    builder.credentialsProvider(...)
                            .maxConcurrency(...)
                            .region(Region.of(...));
                )
                .transferConfiguration(builder -> builder.executor(...))
                .build();

Possible Solution

The issue seems to be that the ClientBootstrap class cannot do all of its static initialization. ClientBootstrap only has two suspect fields which are the native methods: https://github.com/awslabs/aws-crt-java/blob/main/src/main/java/software/amazon/awssdk/crt/io/ClientBootstrap.java#L126-L127

However, the superclass, CrtResource does some static initialization that seems suspect: https://github.com/awslabs/aws-crt-java/blob/main/src/main/java/software/amazon/awssdk/crt/CrtResource.java#L102-L105

I suspect that in the runtime environment where this error occurs, there is something that is preventing the native code from running, possibly due to a lack of access to some dynamically linked libraries.

Creating a standard S3AsyncClient seems not have any issues, although those clients are created with a NettyNioAsyncHttpClient as the underlying http client, while the transfer manager seems to use a S3CrtAsyncHttpClient by default.

One potential solution (which may already be possible but I don't know how to do it) would be to allow us to provide a SdkAsyncHttpClient that the transfer manager should use, or even just an S3AsyncClient similar to how we could configure an S3 client to use in the v1 transfer manager. I don't know if there is some reason why the transfer manager depends on using the CRT HTTP client, but I suspect that if we were able to use the Netty http client this might resolve some of our issues issues.

Additional Information/Context

No response

AWS Java SDK version used

2.17.184-PREVIEW

JDK version used

11

Operating System and version

CentOS 7

[zoewangg]

Hi @TheKeveloper, thanks for reaching out. It seems there is some issue with CRT binary with that specific environment. Could you provide more information on it? Is CentOS 7 the OS?

I don't know if there is some reason why the transfer manager depends on using the CRT HTTP client

To answer your question, performance is one of the most critical features of S3 Transfer Manager, and CRT S3 client is designed to maximize throughput for S3 operations. In fact, a lot of features have been built in CRT to enhance performance, which may not be achieved with the existing Java SDK stack. We did benchmarking to compare the performance between v1 S3 Transfer Manager (Java stack) and v2 (CRT stack) and in our tests, v2 outperforms v1, mostly attributed to CRT. You can learn more about CRT in this doc https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html

One potential solution (which may already be possible but I don't know how to do it) would be to allow us to provide a SdkAsyncHttpClient that the transfer manager should use, or even just an S3AsyncClient similar to how we could configure an S3 client to use in the v1 transfer manager.

We do have plans to expose an API on S3 Transfer Manager builder to allow users to pass an S3AsyncClient. However, in the GA version, only S3 CRT Client will have parallel upload/download feature, which means if you pass an S3AsyncClient with NettyNioAsyncHttpClient, all objects would be uploaded/downloaded as a single object. There is a plan to add concurrent transfers support for Java S3AsyncClient as well, but it may not happen soon.

[TheKeveloper]

Hi @zoewangg, thanks for the quick response. Yes, CentOS 7 is the OS: https://www.centos.org/download/.

The performance considerations for CRT make sense. Is there some way we can check for CRT availability in a non-throwing manner? The only error we are encountering now is an uncaught throwable without any specific error message, which makes it difficult for us to diagnose the issue.

[zoewangg]

@TheKeveloper can you tell us what architecture is?

Is there some way we can check for CRT availability in a non-throwing manner?

@graebm, @TingDaoK, do either of you know the answer?

[TheKeveloper]

@zoewangg architecture is x86_64

[acrogecko]

We are having the same problem with an identical stack trace. Our configuration:

S3 Transfer Manager: software.amazon.awssdk/s3-transfer-manager/2.17.186-PREVIEW Java: version: 17.0.2, vendor: Eclipse Adoptium OS: Amazon AL2, version: "4.14.275-207.503.amzn2.x86_64", arch: "amd64"

Are we perhaps needing some setup of PATH or java.library.path to enable loading of the proper native library?

[TheKeveloper]

@zoewangg just wanted to check if there were any updates on how we might be able to address this issue?

[zoewangg]

Apologies for the delayed response.

Could you enable CRT logs? You can do so by adding the following code before transfer manager initialization.

// logging to a file

Log.initLoggingToFile(Log.LogLevel.Debug, "log.txt")

// logging to the console
Log.initLoggingToStdout(Log.LogLevel.Debug);

[jonathansloman]

Hi @zoewangg - I've just hit the same issue. Trying to use 2.17.243-PREVIEW.

I attempted to enable the CRT logs, but that also failed: java.lang.NoClassDefFoundError: Could not initialize class software.amazon.awssdk.crt.Log

I noticed that the dependency on aws-crt was quite old (0.15.2) so tried forcing that to the latest (0.18.1), but that also failed in the same way.

We're running in EKS using a base image of: eclipse-temurin:17-jre-alpine

On a hunch, I've tried using a non-alpline based image ( eclipse-temurin:17-jre ) and that seems to have fixed it. So, currently, it looks like the aws-crt native code won't work on alpine, presumably due to the use of the musl libc.

So given others on this thread don't mention alpine, might not be relevant to them, but posting here in case this is useful to others.

[ventureh]

@zoewangg I was able to resolve the issue above by overriding the dependency with a newer version of software.amazon.awssdk.crt:aws-crt. Sharing my POM dependency workaround:

<dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3-transfer-manager</artifactId>
            <version>2.17.175-PREVIEW</version>
            <exclusions>
                <exclusion>
                    <groupId>software.amazon.awssdk.crt</groupId>
                    <artifactId>aws-crt</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.19.3</version>
        </dependency>

I wasn't able to get you the logs when the exception above happened due to what @jonathansloman was saying above, but I am attaching the log with the fix proposed implemented.

ps. running on aarch64 architecture ps2. somehow the fix above won't work with the latest version of s3-transfer-manager. I tried upgrading to 2.17.245-PREVIEW and it fails again with the same exception

log.txt

[joquijada]

In my case, I was missing the CRT dependency in my pom.xml altogether. I followed this article for exact instruction on how to use S3/CRT:

<dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.21.12</version>
</dependency>

[amitkc00]

In my case, I was missing the CRT dependency in my pom.xml altogether. I followed this article for exact instruction on how to use S3/CRT:

<dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.21.12</version>
</dependency>

This solved my problem. Thank you.

[daisy-smith]

Hi @zoewangg - I've just hit the same issue. Trying to use 2.17.243-PREVIEW.

I attempted to enable the CRT logs, but that also failed: java.lang.NoClassDefFoundError: Could not initialize class software.amazon.awssdk.crt.Log

I noticed that the dependency on aws-crt was quite old (0.15.2) so tried forcing that to the latest (0.18.1), but that also failed in the same way.

We're running in EKS using a base image of: eclipse-temurin:17-jre-alpine

On a hunch, I've tried using a non-alpline based image ( eclipse-temurin:17-jre ) and that seems to have fixed it. So, currently, it looks like the aws-crt native code won't work on alpine, presumably due to the use of the musl libc.

So given others on this thread don't mention alpine, might not be relevant to them, but posting here in case this is useful to others.

Hi @zoewangg - I am facing the same issue

We're running in EKS using a base image of: zulu-openjdk-alpine

When tried using a non-alpine based image, issue gets fixed.

But we have a requirement of using alpine based image only. So can you please help with any fix which can be applied for alpine based images?

[mubasherhussain9d]

hi @daisy-smith i am also facing the same error, have solved it??