[4.3.4] Running the native build image throws "GLIBC_2.32 not found"-Error at runtime

[mhuebner]

Expected Behavior

No response

Actual Behaviour

Building the native image works as expected but when the app is deployed to lambda I get an error:

./func: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./func)

I think it might be a problem related to the new image amazonlinux:2023. Perhaps switching back to amazonlinux:2 would fix this issue (going to test later). The image building process itself is done in bitbucket pipelines with base image amazoncorretto:17.

In Quarkus, there is the option quarkus.native.container-build - however, it seems that currently there is nothing comparable for Graal.

Steps To Reproduce

Build your app with buildNativeLambda and run it in aws Lambda: ./gradlew clean buildNativeLambda --stacktrace

Environment Information

No response

Example Application

No response

Version

4.3.4

[mhuebner]

Going back to amazonlinux:2 resolves this issue (workaround).

[sdelamo]

do you have a sample app to verify the issue? We had some functional tests and I did not encounter this issue

https://github.com/micronaut-projects/micronaut-starter/blob/4.3.x/test-lambda-graalvm.sh#L18

[smirnoal]

can you try using amazonlinux:2023 with java-17-amazon-corretto-devel package instead of amazoncorretto:17? i.e.

FROM amazonlinux:2023
RUN dnf -y install java-17-amazon-corretto-devel

amazoncorretto:17 uses GLIBC2.26 while amazonlinux:2023 is on 2.34 - this might be the cause of the issue you are facing

[mhuebner]

can you try using amazonlinux:2023 with java-17-amazon-corretto-devel package instead of amazoncorretto:17? i.e.

Thanks for the hint @smirnoal - i'll give it a try and then report back here.

[mhuebner]

Ok, tried that but still get the error:

./func: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./func)
./func: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./func)
EXTENSION   Name: cloudwatch_lambda_agent   State: Ready    Events: [INVOKE, SHUTDOWN]
INIT_REPORT Init Duration: 104.86 ms    Phase: init Status: error   Error Type: Runtime.ExitError
LOGS    Name: cloudwatch_lambda_agent   State: Already subscribed   Types: [Platform]
./func: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./func)
./func: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./func)
EXTENSION   Name: cloudwatch_lambda_agent   State: Ready    Events: [INVOKE, SHUTDOWN]
INIT_REPORT Init Duration: 310.06 ms    Phase: invoke   Status: error   Error Type: Runtime.ExitError
START RequestId: 2637d5f9-7421-4fbc-88c4-239047410315 Version: $LATEST
RequestId: 2637d5f9-7421-4fbc-88c4-239047410315 Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: 2637d5f9-7421-4fbc-88c4-239047410315
REPORT RequestId: 2637d5f9-7421-4fbc-88c4-239047410315  Duration: 813.61 ms Billed Duration: 814 ms Memory Size: 512 MB Max Memory Used: 14 MB  

My bitbucket-pipelines.yml setup (snippet) is as follows:

  name: Deploy to staging environment
  deployment: staging
  steps:
    - step:
        image: amazonlinux:2023
        size: 2x
        name: Build native image
        caches:
          - gradle
          - gradlewrapper
        services:
          - docker-6gb
        script:
          - dnf -y install java-17-amazon-corretto-devel findutils
          - ./gradlew clean buildNativeLambda --stacktrace
        artifacts:
          - build/libs/**

Perhaps it might be good choice to modify the built dockerfile in the gradle task:

tasks.named("dockerfileNative") {
    baseImage = "amazonlinux:2023"
    jdkVersion = "17"
    args(
            "-XX:MaximumHeapSizePercent=80",
            "-Dio.netty.allocator.numDirectArenas=0",
            "-Dio.netty.noPreferDirect=true"
    )
}

see https://micronaut-projects.github.io/micronaut-gradle-plugin/snapshot/#_customized_docker_files

[sdelamo]

You could replace the contents of the docker file used to generate the native image with:

tasks.named<io.micronaut.gradle.docker.DockerBuildOptions>("dockerfileNative") {
    editDockerfile {
        replace(
            "RUN yum install -y gcc gcc-c++ glibc-devel curl-minimal bash zlib zlib-devel zlib-static zip tar gzip",
            "RUN yum install -y gcc gcc-c++ glibc-devel glibc-langpack-en curl-minimal bash zlib zlib-devel zlib-static zip tar gzip"
        )
    }
}

New lambdas applications are generated with:

tasks.named<io.micronaut.gradle.docker.NativeImageDockerfile>("dockerfileNative") {
    baseImage.set("amazonlinux:2023")
    jdkVersion.set("21")
    args(
        "-XX:MaximumHeapSizePercent=80",
        "-Dio.netty.allocator.numDirectArenas=0",
        "-Dio.netty.noPreferDirect=true"
    )
}

[mhuebner]

Thank you for your response and the nice hint @sdelamo.

What I can see is that you added glibc-langpack-en package.

I tested your suggestion and the Dockerfile now looks like this:

FROM amazonlinux:2023 AS graalvm
ENV LANG=en_US.UTF-8
RUN yum install -y gcc gcc-c++ glibc-devel glibc-langpack-en curl-minimal bash zlib zlib-devel zlib-static zip tar gzip
RUN curl -4 -L https://download.oracle.com/graalvm/21/latest/graalvm-jdk-21_linux-aarch64_bin.tar.gz -o /tmp/graalvm-jdk-21_linux-aarch64_bin.tar.gz
RUN tar -zxf /tmp/graalvm-jdk-21_linux-aarch64_bin.tar.gz -C /tmp && ls -d /tmp/graalvm-jdk-21* | grep -v "tar.gz" | xargs -I_ mv _ /usr/lib/graalvm
RUN rm -rf /tmp/*
CMD ["/usr/lib/graalvm/bin/native-image"]
ENV PATH=/usr/lib/graalvm/bin:${PATH}
FROM graalvm AS builder
WORKDIR /home/app
COPY --link layers/libs /home/app/libs
COPY --link layers/app /home/app/
COPY --link layers/resources /home/app/resources
RUN mkdir /home/app/config-dirs
RUN mkdir -p /home/app/config-dirs/generateResourcesConfigFile
RUN mkdir -p /home/app/config-dirs/com.fasterxml.jackson.core/jackson-databind/2.15.2
RUN mkdir -p /home/app/config-dirs/ch.qos.logback/logback-classic/1.4.9
COPY --link config-dirs/generateResourcesConfigFile /home/app/config-dirs/generateResourcesConfigFile
COPY --link config-dirs/com.fasterxml.jackson.core/jackson-databind/2.15.2 /home/app/config-dirs/com.fasterxml.jackson.core/jackson-databind/2.15.2
COPY --link config-dirs/ch.qos.logback/logback-classic/1.4.9 /home/app/config-dirs/ch.qos.logback/logback-classic/1.4.9
RUN native-image -cp /home/app/libs/*.jar:/home/app/resources:/home/app/application.jar --no-fallback -o application -H:ConfigurationFileDirectories=/home/app/config-dirs/generateResourcesConfigFile,/home/app/config-dirs/com.fasterxml.jackson.core/jackson-databind/2.15.2,/home/app/config-dirs/ch.qos.logback/logback-classic/1.4.9 io.conversionbuddy.firehose.transformer.EventTransformationRuntime
FROM amazonlinux:2023
WORKDIR /function
RUN yum install -y zip
COPY --link --from=builder /home/app/application /function/func
RUN echo "#!/bin/sh" >> bootstrap && echo "set -euo pipefail" >> bootstrap && echo "./func -XX:MaximumHeapSizePercent=80 -Dio.netty.allocator.numDirectArenas=0 -Dio.netty.noPreferDirect=true -Djava.library.path=$(pwd)" >> bootstrap
RUN chmod 777 bootstrap
RUN chmod 777 func
RUN zip -j function.zip bootstrap func
ENTRYPOINT ["/function/func"]

The replaced RUN command is now also included. But I am still getting the same error in Lamda:

./func: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./func)
./func: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./func)

[sdelamo]

do you have a sample code or build file which I can use to reproduce the issue?

[mhuebner]

I built a simple test app: https://github.com/mhuebner/mn-sample-959 To reproduce the error you would have to build the app, deploy it in aws Lambda and then run a simple Test in Lambda.

[sdelamo]

I am unable to verify your issue.

I created a PR:

https://github.com/mhuebner/mn-sample-959/pull/1

• I am locally running in MacOS Arm.

• I run Java 21 in a terminal.

• Generate Native Image

./gradlew buildNativeLambda

• Create a Lambda with Amazon Linux 2023 runtime. I selected arm.

• Upload code build/libs/mn-sample-959-unspecified-lambda.zip

• Run a Kinesis firehose test events:

• it works fine:

Am I missing anything?

[sdelamo]

make sure you are using the same platform architecture in the native image generation and the lambda function.

[mhuebner]

Thank you so much for your thorough investigation of the issue. It indeed was related to the runtime, which I had reverted back to amazonlinux:2 after the initial adjustments. To put it bluntly: I failed to set it back again. This ticket has been enormously helpful in troubleshooting the problem. Thank you all for your assistance @sdelamo @smirnoal