DynamoDb-enhanced NoClassDefFoundError with different ClassLoaders

[Nithanim]

The BeanTableSchema of dynamodb-enhanced does not work correctly when this library and the bean class reside in different classloaders. A NoClassDefFoundError for the bean class is thrown, when a call against a bean class/object is made (\<init>, getter, setter).

Describe the bug

Internally, the BeanTableSchema creates "Accessor"-Lambdas for performant creation of bean objects, as well as getters and setters for the properties. An example would be the software.amazon.awssdk.enhanced.dynamodb.mapper.BeanTableSchema#newObjectSupplierForClass (also #getterForProperty and #setterForProperty) responsible for creating a Supplier that returns a new bean instance.

Internally, the LambdaToMethodBridgeBuilder is then used to construct such lambda via the LambdaMetafactory. The creation of the lambda itself is successful.

However, when this generated lambda is invoked, it crashes with the mentioned NoClassDefFoundError. This is related to the fact that the Lambda (somehow) tries to resolve the bean class with the classloader of the SDK. But since the bean class is loaded in a different classloader, the classloader of the SDK cannot find the bean class. More specifically, the problem arises when the SDK classloader is either a parent of or entirely unrelated to the classloader of the bean.

This issue is most likely related to issue #2198 since the error messages are identical. However that issue is specifically about usage with the play framework (which I do not know anything about). Furthermore, a usage of Class.forName is mentioned but I did not come across usage of that. Therefore I decided to create a more focused issue for the core problem I (somewhat) have a solution for.

This issue should fix the NoClassDefFoundError of https://github.com/quarkusio/quarkus/issues/12168.

Expected Behavior

Normal execution and same behavior when AWS SDK and bean class are in the same classloader.

Current Behavior

When such generated lambda is invoked, the following (sample) stack trace is thrown:

Caused by: java.lang.NoClassDefFoundError: org/acme/dynamodb/Fruit
    at software.amazon.awssdk.enhanced.dynamodb.internal.mapper.ResolvedImmutableAttribute.lambda$create$0(ResolvedImmutableAttribute.java:48)
    at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema.lambda$itemToMap$5(StaticImmutableTableSchema.java:502)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085)
    at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema.itemToMap(StaticImmutableTableSchema.java:500)
    at software.amazon.awssdk.enhanced.dynamodb.mapper.WrappedTableSchema.itemToMap(WrappedTableSchema.java:64)
    at software.amazon.awssdk.enhanced.dynamodb.mapper.WrappedTableSchema.itemToMap(WrappedTableSchema.java:64)
    at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.generateRequest(PutItemOperation.java:71)
    at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.generateRequest(PutItemOperation.java:40)
    at software.amazon.awssdk.enhanced.dynamodb.internal.operations.CommonOperation.execute(CommonOperation.java:113)
    at software.amazon.awssdk.enhanced.dynamodb.internal.operations.TableOperation.executeOnPrimaryIndex(TableOperation.java:59)
    at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:180)
    at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:188)
    at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:193)
    at org.acme.dynamodb.FruitSyncService.add(FruitSyncService.java:49)
    at org.acme.dynamodb.FruitSyncService_ClientProxy.add(FruitSyncService_ClientProxy.zig:354)
    at org.acme.dynamodb.FruitResource.add(FruitResource.java:40)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
    at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524)
    at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474)
    at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:476)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:434)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:408)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:69)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
    ... 15 more
Caused by: java.lang.ClassNotFoundException: org.acme.dynamodb.Fruit
    at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:421)
    at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:397)
    ... 47 more

(where Fruit is the bean class.)

The code that creates the dynamodb client looks like this:

DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder().dynamoDbClient(ddb).build();
var table = enhancedClient.table("Fruits", TableSchema.fromBean(Fruit.class));

offending user-application code is:

table.putItem(fruit);

Steps to Reproduce

Loading a dynamodb-enhanced annotated bean in a child (or unrelated) classloader, pass it to TableSchema.fromBean(...) and use any action like getItem or putItem.

A self-contained runnable example can be found here: https://github.com/Nithanim/awssdk2-dynamodb-enhanced-NDFE/tree/master/src/main/java

You can find a "crashing" example specifically using the problematic AWS SDK class. Additionally there is an isolated case without the AWS SDK only with plain java.

Possible Solution

When creating the Lambdas via the LambdaMetafactory the Lookup of the AWS SDK is used. This is (as far as I understand) the cause that the wrong classloader for the lookup is used.

A working solution would be to use the Lookup of the bean class, which in turn will use the classloader of the bean class to resolve the bean class on invocation of the lambda.

In the same repository (https://github.com/Nithanim/awssdk2-dynamodb-enhanced-NDFE/tree/master/src/main/java) you can also find the "Fixed" classes, which have been patched to reslove the issue. However, I am completely unsure if Lookup.privateLookupIn(...) is the correct way.

Context

I wanted to use the dynamodb-enhanced in quarkus (and specifically later in native-image) but the Tests kept failing with the NoClassDefFoundError. After some digging I found out that quarkus uses two different classloaders for tests. One for all libraries and then a child classloader for the tests. A bit of insight can be found here: https://quarkus.io/guides/class-loading-reference

Thank you for your help!

Your Environment

• AWS Java SDK version used: 2.16.81
• JDK version used: build 11.0.11+9-Ubuntu-0ubuntu2.20.04
• Operating System and version: Ubuntu 20.04.2 LTS

[drissamri]

Sadly have been stuck on this and have to avoid enhanced dynamodb up to now in our native images. When this issue gets resolved it would simplify our business code a lot

[Nithanim]

I am not sure if you know but this issue does not fix the native-image problems, it only really fixes the execution of tests in quarkus. The problem preventing dynamodb-enhanced in native-image is a different one, though I have a solution for that too. But I decided to introduce fixes step-by-step.

As a sidenote and bit of self-advertising: I made a custom quarkus extension (in case you are using quarkus) if you are interested: https://github.com/Nithanim/quarkus-dynamodb-enhanced Not 100% sure if everything works but it is looking good in our staging env and our plan is to push that to production in the next couple of days.

Background:
We did not compile our lambdas to native-images prviously but recently we were force into it for response times. I hated the idea of using the StaticTableSchema because it would mean error-prone and manual getter/setter definitions. Though, this should work fine if you want to go that route. The only problem I see for native-image is with BeanTableSchema, everything es should work just fine.

[Nithanim]

I noticed that my proposed solution only works for Java 9+ but this library is 8+. Sadly I have not found any other good solution.

I have not been able to find a workaround just using the MethodHandles/Lookups provided by Java 8 alone. A workaround would be to create child ClassLoader (CL) of the bean CL in which we can load a class from which we can fetch a Lookup for that context.

However, this is rather ugly because we would need the bytecode of such class and then load it in a classloader as shown here. My proof-of-concept with bytebuddy is already convoluted enough considering this is all just a dance-around for the simple Lookup.privateLookupIn() of Java 9+.

In short:

  private static MethodHandles.Lookup getPrivateLookup(ClassLoader beanClassLoader) {
    // We use BB to be able to simply define the class below and load a copy in a child CL
    ByteBuddy bb = new ByteBuddy();
    DynamicType.Unloaded<GetLookup> clone =
        bb.rebase(GetLookup.class).name("JustRenameBecauseClassIsAlreadyLoadedInParentCL").make();
    Class<? extends Supplier<MethodHandles.Lookup>> loaded =
        clone.load(beanClassLoader).getLoaded(); // Creates child CL automatically
    try {
      Supplier<MethodHandles.Lookup> supplier = loaded.getConstructor().newInstance();
      return supplier.get();
    } catch (Exception ex) {
      throw new IllegalStateException(ex);
    }
  }

  public static class GetLookup implements Supplier<MethodHandles.Lookup> {
    public MethodHandles.Lookup get() {
      return MethodHandles.lookup();
    }
  }

[fastluca]

Is there any update on this?

[debora-ito]

No updates here, just wanted to confirm that DynamoDB Enhanced Client does not support this case currently.

Add a 👍 in the main description of the issue to show your support, it helps us with prioritization.

[AmrHassanien]

It does not work well with Spring Boot Devtools too.

Dev tools uses two class loaders, one for third party libraries, and the other for project classes.

So I get something like this :

com.example.myblog.model.dynamodb.User incompatible with com.example.myblog.model.dynamodb.User java.lang.ClassCastException: com.example.myblog.model.dynamodb.User incompatible with com.example.myblog.model.dynamodb.User at com.example.myblog.repository.dynamodb.UserDynamoDbRepository.findAll(UserDynamoDbRepository.java:96) at com.example.myblog.controller.MainController.index(MainController.java:29) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:853)

[vaibhav170]

%dev.quarkus.class-loading.reloadable-artifacts=software.amazon.awssdk:dynamodb-enhanced this application property solves quarkus dev mode issue (NoClassDefFoundError)

[Srinivasa-Cana]

Still not working for me above solution. I am using quarkus 2.7.2.Final version

[vaibhav170]

https://github.com/vaibhav170/dynamo u can refer this if u are still facing issues @Srinivasa-Cana

[faermanj]

Same issue here with 2.11.1, even setting quarkus.class-loading.reloadable-artifact

[hamburml]

@faermanj It works on my side with quarkus 2.11.1.Final. I set quarkus.class-loading.reloadable-artifacts=software.amazon.awssdk:dynamodb-enhanced

[hamburml]

@faermanj No, it looks like it does not work :( @debora-ito any news on the prio? @Nithanim If you like you could help here https://github.com/quarkiverse/quarkus-amazon-services/pull/255

[vamsisayimpu]

TODO: Quakrus Testing framework uses QuarkusClassLoader, for workaround use this below property in test/app.properties

quarkus.test.flat-class-path=true

[hamburml]

this would not solve the native problem in quarkus.

[vaibhav170]

You can use static dynamodb schema to fix native issue, or else if its aws lambda give a snapstart feature try

[hamburml]

or you simply use the quarkus extension and it works :) edit and it would not solve issues in quarkus dev mode. The quarkus extension also does that.

[vaibhav170]

Quarkus extension dont solve dynamodb enhanced client issues ( native issue, dev and test issue),

To solve native issue if someone need to build native - use static dynamodb table schema

To solve dev, test issue - use %dev.quarkus.class-loading.reloadable-artifacts=software.amazon.awssdk:dynamodb-enhanced

%test.quarkus.class-loading.reloadable-artifacts=software.amazon.awssdk:dynamodb-enhanced

edit Quarkus dynamodb extension is needed and on top of that we have to follow above mentioned steps to make dynamodb enhanced client work( just dynamodb client will work with just extension but to make enhanced client work we need to make some extra changes than just adding quarkus dynamodb extension)

[hamburml]

Hey @vaibhav170, I welcome you to just try it out. We added some workarounds so that native, dev and test issues do work (it is the PullRequest you even added your message, it has UnitTests and IntegrationTests which are run in jvm-mode and in native-mode...). If the aws-sdk-java-v2 did solve the issues mentioned in this issue the workarounds can be removed. Currently the company I work for has services in production which use the enhanced client with dynamic schema table as native in aws lambda - and i know that because we did that. Just please try it out before you say that it is not working :) Thanks - Nevertheless the core issues should be tackled.

[vaibhav170]

Yes right, let me try, my observation was couple of months old, just now i saw another thread where you guys are discussing about fix, will try that and also update on cold start with both the approach, thanks

[hamburml]

Thanks! Just be sure that you add quarkus-amazon-dynamodb and quarkus-amazon-dynamodb-enhanced as dependencies :) As reference example you can try this https://github.com/quarkusio/quarkus-quickstarts/pull/1186/files - sadly it is not merged

[vaibhav170]

@hamburml it worked without adding any extra cold start, so all things are happening in build step as expected, thanks for this extension.

[jpdev01]

Same issue here