ClassNotFoundException for the handler class when using java11 base image

[spirosag]

I am trying to make a demo of deploying a java lambda using containers but even though I am following the official aws guides I can't seem to make it work. For starters, I am using the simplest basic java app from the aws example lambdas. I add the following dockerfile according to the usage instructions of the aws lambda base image:

FROM public.ecr.aws/lambda/java:11

# Copy function code and runtime dependencies from Gradle layout
COPY build/classes/java/main ${LAMBDA_TASK_ROOT}
COPY build/dependency/* ${LAMBDA_TASK_ROOT}/lib/

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "example.Handler::handleRequest" ]

I am also adding the following to the build.gradle file so that the correct files are built:

task copyRuntimeDependencies(type: Copy) {
    from configurations.runtimeClasspath
    into 'build/dependency'
}

build.dependsOn copyRuntimeDependencies

At the docker container that is build I verify that the /var/task folder contains the following:

var/task/
var/task/example/
var/task/example/Handler.class
var/task/example/HandlerDivide.class
var/task/example/HandlerInteger.class
var/task/example/HandlerList.class
var/task/example/HandlerStream.class
var/task/example/HandlerString.class
var/task/example/HandlerWeatherData.class
var/task/example/InputLengthException.class
var/task/example/WeatherData.class
var/task/lib/
var/task/lib/aws-lambda-java-core-1.2.1.jar
var/task/lib/gson-2.8.6.jar

However, when I try to deploy and test the lambda I am getting the following ClassNotFoundException error:

START RequestId: 144bc991-3a7b-439a-86ac-faa647560763 Version: $LATEST
Class not found: example.Handler: java.lang.ClassNotFoundException
java.lang.ClassNotFoundException: example.Handler. Current classpath: file:/var/task/:file:/var/task/lib/aws-lambda-java-core-1.2.1.jar:file:/var/task/lib/gson-2.8.6.jar
END RequestId: 144bc991-3a7b-439a-86ac-faa647560763
REPORT RequestId: 144bc991-3a7b-439a-86ac-faa647560763  Duration: 1313.87 ms    Billed Duration: 1658 ms    Memory Size: 128 MB Max Memory Used: 81 MB  Init Duration: 343.91 ms    
XRAY TraceId: 1-633bddb6-081491797aa78be22f1d2de1   SegmentId: 032f849e7a82cdb4 Sampled: true   

Do I have to add something to the classfile? The class files seem to be there. Am I missing some configuration?

[maschnetwork]

Hi,

could you please provide a example repository to reproduce? I just tried the java-basic example with the gradle build step + the Dockerfile you mentioned and it seems to be working fine on my end.

Best, Max

[spirosag]

@maschnetwork Thanks a lot for taking time to look into it. This is the repo with the app, gradlew and docker file. In the readme I have written how I am building the docker image and pushing it to ECR (basic stuff).

Maybe I'm doing something wrong in the lambda configuration? When I create the AWS lambda I don't do any extra configuration regarding the image regarding the ENTRYPOINT override, CMD override or WORKDIR override. In order to test the lambda I go to the Test tab, create a sample test event, send it to the lambda and get the error I mentioned as a result.

[eldritchideen]

@spirosag I took a look at the repo you linked above.

I cloned the repo, did a gradle build and the following commands to build the container and upload it to my ECR repo.

$ docker build -t java-basic-container .
Sending build context to Docker daemon  1.922MB
Step 1/4 : FROM public.ecr.aws/lambda/java:11
11: Pulling from lambda/java
69b97305933b: Pull complete 
888685b93766: Pull complete 
f11326706a0c: Pull complete 
9963a4231942: Pull complete 
92ad906a1cd2: Pull complete 
7695dd01f822: Pull complete 
Digest: sha256:f33a646ddb76df2068a1d8097ca18fd4d9d957865170e3b16d1716ada81a200a
Status: Downloaded newer image for public.ecr.aws/lambda/java:11
 ---> f2961025a04b
Step 2/4 : COPY build/classes/java/main ${LAMBDA_TASK_ROOT}
 ---> 271e00b666c3
Step 3/4 : COPY build/dependency/* ${LAMBDA_TASK_ROOT}/lib/
 ---> 0a4fbff4ad65
Step 4/4 : CMD [ "example.Handler::handleRequest" ]
 ---> Running in 53a560ee0158
Removing intermediate container 53a560ee0158
 ---> abd151b41fca
Successfully built abd151b41fca
Successfully tagged java-basic-container:latest

$ docker tag java-basic-container:latest xxx.dkr.ecr.ap-southeast-2.amazonaws.com/java-basic-container:latest
$ docker push xxx.dkr.ecr.ap-southeast-2.amazonaws.com/java-basic-container:latest
The push refers to repository [xxx.dkr.ecr.ap-southeast-2.amazonaws.com/java-basic-container]
8be68769f819: Pushed 
fb64a1de7e02: Pushed 
d772e6cc783e: Pushed 
7411c4b1ea53: Pushed 
27b6e81506f5: Pushed 
66e8eb9bb970: Pushed 
dea202043ef7: Pushed 
60d48413aa2b: Pushed 
latest: digest: sha256:37e7b8be06616fb34f301139e2e12e35e2ff5c1c4649dc6627425326af3b6f17 size: 2000

Created a Lambda function via the console. Referencing the ECR entry xxx.dkr.ecr.ap-southeast-2.amazonaws.com/java-basic-container@sha256:37e7b8be06616fb34f301139e2e12e35e2ff5c1c4649dc6627425326af3b6f17. The only changes from default was allocating 1024MB of RAM for the function.

I ran a test against the default handler specified in the Dockerfile CMD entry. This worked without any issues.

Trying one of the other handlers in the code base works, by setting the CMD in Image Configuration to example.HandlerDivide::handleRequest Passing the array [20,10] works as expected.

Note that I am using JDK17 and Gradle 7.5.1 in my environment. What versions are you using?

The only way I was able to get a similar class not found error was when I made a mistake overriding the handler in the CMD in the Image Configuration.

[spirosag]

@eldritchideen Thank you so much for your effort. I am literally doing exactly the same as you. Only difference is that I am using openjdk11 although I have tried the same with corretto17 with the same results. I uploaded my resulting dokcer image here. Would you mind checking if your lambda works with it? If it does then I'm doing something wrong with my lambda configuration but if you get the same error this means that I am somehow messing up the docker image creation.

[eldritchideen]

@spirosag I get the same error using your container image. I did some analysis of the differences between our two container images built from the same source and Dockerfile. I have found that the file permissions on the class files in the container images are different between our builds. Inspecting the two running images on my Ubuntu Linux VM I see the following:

spirosag/java-lambda-container-test:latest

sh-4.2# cd /var/task/
sh-4.2# ls -l
total 8
drwxr-x--- 2 root root 4096 Oct  3 15:48 example
drwxr-xr-x 2 root root 4096 Oct  4 08:22 lib
sh-4.2# cd example/
sh-4.2# ls -l
total 36
-rw-r----- 1 root root 2371 Oct  3 15:48 Handler.class
-rw-r----- 1 root root 2478 Oct  3 15:48 HandlerDivide.class
-rw-r----- 1 root root 1967 Oct  3 15:48 HandlerInteger.class
-rw-r----- 1 root root 2189 Oct  3 15:48 HandlerList.class
-rw-r----- 1 root root 2966 Oct  3 15:48 HandlerStream.class
-rw-r----- 1 root root 1983 Oct  3 15:48 HandlerString.class
-rw-r----- 1 root root 1895 Oct  3 15:48 HandlerWeatherData.class
-rw-r----- 1 root root  380 Oct  3 15:48 InputLengthException.class
-rw-r----- 1 root root 1216 Oct  3 15:48 WeatherData.class

My image

sh-4.2# cd /var/task/
sh-4.2# ls -l
total 8
drwxrwxr-x 2 root root 4096 Oct  6 10:00 example
drwxr-xr-x 2 root root 4096 Oct  6 10:04 lib
sh-4.2# cd example/
sh-4.2# ls -l
total 36
-rw-rw-r-- 1 root root 2371 Oct  6 10:00 Handler.class
-rw-rw-r-- 1 root root 2478 Oct  6 10:00 HandlerDivide.class
-rw-rw-r-- 1 root root 1967 Oct  6 10:00 HandlerInteger.class
-rw-rw-r-- 1 root root 2189 Oct  6 10:00 HandlerList.class
-rw-rw-r-- 1 root root 2966 Oct  6 10:00 HandlerStream.class
-rw-rw-r-- 1 root root 1983 Oct  6 10:00 HandlerString.class
-rw-rw-r-- 1 root root 1895 Oct  6 10:00 HandlerWeatherData.class
-rw-rw-r-- 1 root root  380 Oct  6 10:00 InputLengthException.class
-rw-rw-r-- 1 root root 1216 Oct  6 10:00 WeatherData.class

Note the difference in file permissions. I build the project and create the Docker image on an Ubuntu Linux VM as a non-root user with default umask of 0002.

What system are using using to build the project? Check the permissions on the files in your build/classes and build/dependency directories. That seems to be the main difference I can see.

I tested this by changing the file permissions on my class and jar files after building manually with chmod 640 then packaged up a new docker image. If I try to run this in Lambda it fails with the same error you are seeing:

START RequestId: 7fdb1a0d-dc1a-40c3-8795-eec144329e2a Version: $LATEST
Class not found: example.Handler: java.lang.ClassNotFoundException
java.lang.ClassNotFoundException: example.Handler. Current classpath: file:/var/task/:file:/var/task/lib/aws-lambda-java-core-1.2.1.jar:file:/var/task/lib/aws-lambda-java-events-3.10.0.jar:file:/var/task/lib/gson-2.8.6.jar:file:/var/task/lib/joda-time-2.6.jar
Caused by: java.io.FileNotFoundException: /var/task/example/Handler.class (Permission denied)
at java.base/java.io.FileInputStream.open0(Native Method)

From the Lambda docs:

The default Lambda user must be able to read all the files required to run your function code. Lambda follows security best practices by defining a default Linux user with least-privileged permissions. Verify that your application code does not rely on files that other Linux users are restricted from running.

I believe if you change the file permissions, your Java handlers and container image will work with Lambda. Hope this helps with your issue.

[spirosag]

@eldritchideen That was it! Thank you so much! In the error I was getting there was no mention of permissions to make me check them. But yeah, once I fixed the permissions for the other group in both the class files and the dependencies I managed to run the container as a lambda. Thank you again so much for your detailed investigation! 🙏