ClassNotFoundException when testing lambda container based on Java

[sethstone]

I'm attempting to use the new lambda container base image for Java with the "java-basic" sample app from: https://github.com/awsdocs/aws-lambda-developer-guide/tree/master/sample-apps/java-basic

The only change I made to the project template was to build a Jar rather than a Zip in the build.gradle file.

task buildJar(type: Jar) {
    from compileJava
    from processResources
    into('lib') {
        from configurations.runtimeClasspath
    }
}

build.dependsOn buildJar

I'm not a Java developer so I'm uncertain about this change. 👆

After running gradle build a new JAR file was placed in build/libs/project.jar

I then used the example Dockerfile from: https://hub.docker.com/r/amazon/aws-lambda-java

FROM public.ecr.aws/lambda/java:11

# Copy function code
COPY target/* ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "com.example.LambdaHandler::handleRequest" ]

I changed the contents to:

FROM public.ecr.aws/lambda/java:11

# Copy function code
COPY build/libs/* ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "example.handler::handleRequest" ]

I was able to successfully build and run the container from docker CLI.

However, when I attempt to test the container using curl I get the "class not found exception"

❯ curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
{"errorMessage":"Class not found: example.Handler","errorType":"java.lang.ClassNotFoundException"}%

The docker logs show the following:

time="2020-12-14T06:25:22.389" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)"
time="2020-12-14T06:25:36.199" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory"
time="2020-12-14T06:25:36.2" level=warning msg="Cannot list external agents" error="open /opt/extensions: no such file or directory"
START RequestId: b97ba52e-1cf1-4ad5-a5f2-1d503a40bba8 Version: $LATEST
Class not found: example.Handler: java.lang.ClassNotFoundException
java.lang.ClassNotFoundException: example.Handler
        at java.base/java.net.URLClassLoader.findClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Unknown Source)

END RequestId: b97ba52e-1cf1-4ad5-a5f2-1d503a40bba8
REPORT RequestId: b97ba52e-1cf1-4ad5-a5f2-1d503a40bba8  Init Duration: 0.30 ms  Duration: 453.78 ms     Billed Duration: 500 ms      Memory Size: 3008 MB    Max Memory Used: 3008 MB

I've confirmed that the project.jar exists inside the container in /var/task/project.jar and that the JAR file contains the Handler class. I've experimented with various paths and naming conventions inside the container to no avail. I can't quite get my head wrapped around what initiates the Java process that can't find the JAR/class.

I did notice that the "runtime" that gets executed (/var/runtime/bootstrap) did not have a classpath that included "/var/task", so I was able to update that with the help of the AWS_LAMBDA_EXEC_WRAPPER hook:

java_class_path (initial): /var/runtime/lib/aws-lambda-java-core-1.2.0.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/task

This unfortunately did not solve the issue. I chalk this up to my inexperience with Java (and Lambda), but on the off-chance there was something misconfigured about this base image I wanted to post this as a potential issue.

[sethstone]

I did a little more experimenting and added the full path to the JAR to the runtime's classpath.

I added the following to my Dockerfile:

COPY wrapper.sh ${LAMBDA_TASK_ROOT}

ENV AWS_LAMBDA_EXEC_WRAPPER=/var/task/wrapper.sh

So the full Dockerfile is this:

FROM public.ecr.aws/lambda/java:11

# Copy function code
COPY build/libs/* ${LAMBDA_TASK_ROOT}
COPY wrapper.sh ${LAMBDA_TASK_ROOT}

ENV AWS_LAMBDA_EXEC_WRAPPER=/var/task/wrapper.sh

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "example.Handler::handleRequest" ]

My wrapper.sh looks like this:

#!/bin/bash

exec $(echo "${@}" | sed "s/serialization-0.2.0.jar/serialization-0.2.0.jar:\/var\/task\/project.jar/")

This actually allowed me to get a bit further, now when I call Curl I don't get any output, but I see this in the docker logs:

time="2020-12-14T15:19:22.578" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)"
time="2020-12-14T15:19:27.144" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory"
time="2020-12-14T15:19:27.144" level=warning msg="Cannot list external agents" error="open /opt/extensions: no such file or directory"                                                                                                    START RequestId: 953418da-0af1-48e6-aaf2-cf39b6b624c0 Version: $LATEST
com/google/gson/GsonBuilder: java.lang.NoClassDefFoundError
java.lang.NoClassDefFoundError: com/google/gson/GsonBuilder
        at example.Handler.<init>(Handler.java:14)                                                                           at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
        at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.google.gson.GsonBuilder
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)                                                         ... 5 more                                                                                                                                                                                                                        time="2020-12-14T15:19:27.537" level=panic msg="ReplyStream not available"
2020/12/14 15:19:27 http: panic serving 127.0.0.1:49100: &{0xc000134000 map[] 2020-12-14 15:19:27.5372827 +0000 UTC m=+4.959239901 panic <nil> ReplyStream not available <nil> <nil> }
goroutine 23 [running]:
net/http.(*conn).serve.func1(0xc00019e1e0)
        /usr/local/go/src/net/http/server.go:1800 +0x139
panic(0x866640, 0xc0001c6230)                                                                                                /usr/local/go/src/runtime/panic.go:975 +0x3e3 

I didn't post the full stack trace, but you can see that it appears to be getting further. Still not quite sure if this is a bug in this Java base image or a configuration issue on my part.

[smirnoal]

Hello,

currently, as a quick fix you could try

1. Remove buildJar task in build.gradle, add a task to copy dependencies

   task copyDependencies(type: Copy) {
   from configurations.runtimeClasspath
   into 'build/dependencies'
   }
   //... 
   build.dependsOn copyDependencies

2. modify your Dockerfile to copy the jars into the appropriate directory structure. Here's what I used:

   
   FROM public.ecr.aws/lambda/java:11

Copy function code

COPY build/classes/java/main ${LAMBDA_TASK_ROOT}

Copy dependencies

COPY build/dependencies/* ${LAMBDA_TASK_ROOT}/lib/

Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)

CMD [ "example.Handler::handleRequest" ]

I see you have already fixed it - the class name referred in CMD should start with capital 'H' to match class name: `example.Handler`

[sethstone]

Great, that worked with one modification. I had to change:

COPY build/classes/java/main ${LAMBDA_TASK_ROOT}

to

COPY build/classes/java/main/* ${LAMBDA_TASK_ROOT}/example/

(Maybe I can work out something more generic so this line doesn't have to be changed per project)

Thanks for your help! I think that solved the core issue.

[rieckpil]

In case someone tries the same with Maven, simply copy the final .jar (shaded) to ${LAMBDA_TASK_ROOT}/lib/ and NOT ${LAMBDA_TASK_ROOT} ...

FROM public.ecr.aws/lambda/java:11

COPY target/java-chromedriver-aws-lambda.jar ${LAMBDA_TASK_ROOT}/lib/

CMD ["de.rieckpil.blog.InvokeChrome::handleRequest"]

[smirnoal]

Hello @rieckpil

while copying the shaded jar to the lib directory may technically work (and a little bit easier to implement!), the guaranteed way is to copy your classes under ${LAMBDA_TASK_ROOT}.

There are instructions for Maven posted at https://gallery.ecr.aws/lambda/java (Java tab), could you please check if they work for you?

[rieckpil]

@smirnoal thanks for the swift response, after hours of debugging (trying to copy the shaded .jar to /var/task) I then found the instructions for Maven in the Gallery.

Using the maven-dependency-plugin I'm now copying the dependencies to /var/task/lib and the Lambda code itself is in its own .jar and part of /var/task