JDK 17 ARM64 doesnt work - Java cannot run in AWS Fargate

[carpnick]

Jenkins and plugins versions report

Environment

```text uname -a Linux ip-.x.x.x.x.compute.internal 4.14.291-218.527.amzn2.aarch64 #1 SMP Fri Aug 26 09:54:28 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux ```

What Operating System are you using (both controller, and any agents involved in the problem)?

uname -a
Linux ip-.x.x.x.x.compute.internal 4.14.291-218.527.amzn2.aarch64 #1 SMP Fri Aug 26 09:54:28 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Reproduction steps

Try 1

1. On an amazon linux 2 machine with docker installed on ARM64 in AWS (IE t4g.medium)
2. docker pull jenkins/agent:3071.v7e9b_0dc08466-1-jdk17
3. docker run -it --entrypoint /bin/bash jenkins/agent:3071.v7e9b_0dc08466-1-jdk17 a. Run java within the container, error happens below: bash: /opt/java/openjdk/bin/java: cannot execute binary file: Exec format error

Try 2

Also tried the ARM64 specific image: docker run -it --entrypoint /bin/bash jenkins/agent:3071.v7e9b_0dc08466-1-jdk17@sha256:bc386f73b3654d7ccaa73060ee66453be35795eb48d9d670d0eaeaa38e16688e

bash: /opt/java/openjdk/bin/java: cannot execute binary file: Exec format error

Try3

If I use the preview image it works... docker run -it --entrypoint /bin/bash jenkins/agent:latest-jdk17-preview

Command Output

``` jenkins@a44fdd3f66c0:~$ java Usage: java [options] [args...] (to execute a class) or java [options] -jar [args...] (to execute a jar file) or java [options] -m [/] [args...] java [options] --module [/] [args...] (to execute the main class in a module) or java [options] [args] (to execute a single source-file program) Arguments following the main class, source file, -jar , -m or --module / are passed as the arguments to main class. where options include: -cp -classpath --class-path A : separated list of directories, JAR archives, and ZIP archives to search for class files. -p --module-path ... A : separated list of directories, each directory is a directory of modules. --upgrade-module-path ... A : separated list of directories, each directory is a directory of modules that replace upgradeable modules in the runtime image --add-modules [,...] root modules to resolve in addition to the initial module. can also be ALL-DEFAULT, ALL-SYSTEM, ALL-MODULE-PATH. --enable-native-access [,...] modules that are permitted to perform restricted native operations. can also be ALL-UNNAMED. --list-modules list observable modules and exit -d --describe-module describe a module and exit --dry-run create VM and load main class but do not execute main method. The --dry-run option may be useful for validating the command-line options such as the module system configuration. --validate-modules validate all modules and exit The --validate-modules option may be useful for finding conflicts and other errors with modules on the module path. -D= set a system property -verbose:[class|module|gc|jni] enable verbose output for the given subsystem -version print product version to the error stream and exit --version print product version to the output stream and exit -showversion print product version to the error stream and continue --show-version print product version to the output stream and continue --show-module-resolution show module resolution output during startup -? -h -help print this help message to the error stream --help print this help message to the output stream -X print help on extra options to the error stream --help-extra print help on extra options to the output stream -ea[:...|:] -enableassertions[:...|:] enable assertions with specified granularity -da[:...|:] -disableassertions[:...|:] disable assertions with specified granularity -esa | -enablesystemassertions enable system assertions -dsa | -disablesystemassertions disable system assertions -agentlib:[=] load native agent library , e.g. -agentlib:jdwp see also -agentlib:jdwp=help -agentpath:[=] load native agent library by full pathname -javaagent:[=] load Java programming language agent, see java.lang.instrument -splash: show splash screen with specified image HiDPI scaled images are automatically supported and used if available. The unscaled image filename, e.g. image.ext, should always be passed as the argument to the -splash option. The most appropriate scaled image provided will be picked up automatically. See the SplashScreen API documentation for more information @argument files one or more argument files containing options -disable-@files prevent further argument file expansion --enable-preview allow classes to depend on preview features of this release To specify an argument for a long option, you can use --= or -- . ```

[args...] (to execute a class) or java [options] -jar [args...] (to execute a jar file) or java [options] -m [/] [args...] java [options] --module [/] [args...] (to execute the main class in a module) or java [options] [args] (to execute a single source-file program) Arguments following the main class, source file, -jar , -m or --module / are passed as the arguments to main class. where options include: -cp -classpath --class-path A : separated list of directories, JAR archives, and ZIP archives to search for class files. -p --module-path ... A : separated list of directories, each directory is a directory of modules. --upgrade-module-path ... A : separated list of directories, each directory is a directory of modules that replace upgradeable modules in the runtime image --add-modules [,...] root modules to resolve in addition to the initial module. can also be ALL-DEFAULT, ALL-SYSTEM, ALL-MODULE-PATH. --enable-native-access [,...] modules that are permitted to perform restricted native operations. can also be ALL-UNNAMED. --list-modules list observable modules and exit -d --describe-module describe a module and exit --dry-run create VM and load main class but do not execute main method. The --dry-run option may be useful for validating the command-line options such as the module system configuration. --validate-modules validate all modules and exit The --validate-modules option may be useful for finding conflicts and other errors with modules on the module path. -D= set a system property -verbose:[class|module|gc|jni] enable verbose output for the given subsystem -version print product version to the error stream and exit --version print product version to the output stream and exit -showversion print product version to the error stream and continue --show-version print product version to the output stream and continue --show-module-resolution show module resolution output during startup -? -h -help print this help message to the error stream --help print this help message to the output stream -X print help on extra options to the error stream --help-extra print help on extra options to the output stream -ea[:...|:] -enableassertions[:...|:] enable assertions with specified granularity -da[:...|:] -disableassertions[:...|:] disable assertions with specified granularity -esa | -enablesystemassertions enable system assertions -dsa | -disablesystemassertions disable system assertions -agentlib:[=] load native agent library , e.g. -agentlib:jdwp see also -agentlib:jdwp=help -agentpath:[=] load native agent library by full pathname -javaagent:[=] load Java programming language agent, see java.lang.instrument -splash: show splash screen with specified image HiDPI scaled images are automatically supported and used if available. The unscaled image filename, e.g. image.ext, should always be passed as the argument to the -splash option. The most appropriate scaled image provided will be picked up automatically. See the SplashScreen API documentation for more information @argument files one or more argument files containing options -disable-@files prevent further argument file expansion --enable-preview allow classes to depend on preview features of this release To specify an argument for a long option, you can use --= or -- . ``` ### Actual Results ``` jenkins@b5f5793e03a8:~$ java bash: /opt/java/openjdk/bin/java: cannot execute binary file: Exec format error ``` ### Anything else? What is the difference between preview builds and versioned builds? Overall I am trying to use the docker-inbound-agent on AWS Fargate ARM, but hitting issues with Java not being the correct runtime in the ARM image.

[carpnick]

Doing some initial research this line may be the culprit. Saw this issue in moby: https://github.com/moby/buildkit/issues/1241

[gounthar]

Doing some initial research this line may be the culprit. Saw this issue in moby: moby/buildkit#1241

Makes sense... But we also have

To avoid "jmods: Value too large for defined data type" error,

on top of the FROM. I guess I can try a proposal and build it on my machines, but let's hope an expert will chime in.

Thanks a lot for your report.

[dduportal]

Thanks a lot @carpnick for raising this issue and sharing the potential solution!

Looks like we forgot to remove the --platform flag for this image because https://github.com/jenkinsci/docker-agent/blob/master/11/bullseye/Dockerfile#L23 does not have the flag.

@carpnick are you ok to open a pull request, this is a nice contribution!

[gounthar]

As I suspected, we're back to the first error.

[carpnick]

I wonder if the first error is related to QEMU 32bit/64 problem : https://bugs.launchpad.net/qemu/+bug/1805913 which then gets redirected to https://gitlab.com/qemu-project/qemu/-/issues/263

Until QEMU resolves - supporting arm32 in this repo will block folks trying to use arm64. This PR it was introduced for ARM32

[gounthar]

I plead guilty.

[dduportal]

Would be worth using a multi-stage Dockerfile to fix this:

• A FROM eclipse-temurin:17.0.4.1_1-jdk-focal AS jre-build by default to fix the issue here
• A FROM --platform=linux/arm32 eclipse-temurin:17.0.4.1_1-jdk-focal AS arm32-jre-build

WDYT?

[gounthar]

Great!

[carpnick]

How would it work @dduportal ? We have 2 images and need to selectively copy based on a special platform. COPY --from doesnt support conditionals. We need a conditional somewhere that says "when this platform use image 1, when not use image 2`.

This is me thinking out loud. I have been trying this locally and can't think of a way to do this without separating the docker files.

The line that controls this assuming you get multistage working: COPY --from=jre-build /javaruntime $JAVA_HOME

[carpnick]

Looking into docker-bake to see if we can do anything there

[dduportal]

@carpnick your analysis is correct: docker bake would be where the "condtionnal" would be by defining 2 different "targets" pointing to a Dockerfile targets each on the same Dockerfile

[gounthar]

I would love to try something, but I won't be able to address that before Wednesday.

[carpnick]

@gounthar and @dduportal - Thoughts - https://github.com/jenkinsci/docker-agent/pull/310

I looked into this more and jlinks is not working for arm/v7. I tried manually and even loading in some config tools. I am not an expert on java compilation. So instead I took the approach that we just outright copy the java runtime as is in the arm32 case. I consider this the best answer of a bad situation due to the error we are running into. I think us playing with BUILDPLATFORM environment is the wrong choice as we will run into this same error with potentially a different platform. So the least risky way to go about it also increases docker image file size (at least with my knowledge).

If we think it is good - I will clean it up properly.

[timja]

Looks like a good a approach to me

[gounthar]

I haven't had a thorough look yet, but what do you mean by

jlinks is not working for arm/v7

and

the java runtime as is in the arm32 case

For me, linux/armv7 is a synonym for arm32 in the Docker world. Sorry if I muddy the water. 😊

[carpnick]

@gounthar - sorry for not being clear. The --platform has to match one of the OS/Arch of the image we are using and so the only option was to set like this. That yielded the same error result:

Jenkins Error

```text 22:00:51 #43 [debian_jdk17 arm32-jre-build 2/2] RUN jlink --add-modules ALL-MODULE-PATH --strip-java-debug-attributes --no-man-pages --no-header-files --compress=2 --output /javaruntime 22:00:51 #43 12.03 Error: java.nio.file.FileSystemException: /opt/java/openjdk/jmods: Value too large for defined data type 22:00:51 #43 12.03 java.nio.file.DirectoryIteratorException: java.nio.file.FileSystemException: /opt/java/openjdk/jmods: Value too large for defined data type 22:00:51 #43 12.04 at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java:169) 22:00:51 #43 12.04 at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(UnixDirectoryStream.java:198) 22:00:51 #43 12.04 at java.base/jdk.internal.module.ModulePath.scanDirectory(ModulePath.java:275) 22:00:51 #43 12.04 at java.base/jdk.internal.module.ModulePath.scan(ModulePath.java:232) 22:00:51 #43 12.04 at java.base/jdk.internal.module.ModulePath.scanNextEntry(ModulePath.java:190) 22:00:51 #43 12.04 at java.base/jdk.internal.module.ModulePath.find(ModulePath.java:154) 22:00:51 #43 12.04 at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.newModuleFinder(JlinkTask.java:468) 22:00:51 #43 12.04 at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.initJlinkConfig(JlinkTask.java:387) 22:00:51 #43 12.04 at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:271) 22:00:51 #43 12.04 at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:55) 22:00:51 #43 12.04 at jdk.jlink/jdk.tools.jlink.internal.Main.main(Main.java:33) 22:00:51 #43 12.04 Caused by: java.nio.file.FileSystemException: /opt/java/openjdk/jmods: Value too large for defined data type 22:00:51 #43 12.04 at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) 22:00:51 #43 12.04 at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115) 22:00:51 #43 12.04 at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java:168) 22:00:51 #43 12.04 ... 10 more 22:00:53 #43 ERROR: process "/bin/sh -c jlink --add-modules ALL-MODULE-PATH --strip-java-debug-attributes --no-man-pages --no-header-files --compress=2 --output /javaruntime" did not complete successfully: exit code: 4 22:00:53 22:00:53 #42 [archlinux_jdk11 stage-1 4/8] RUN pacman -Syu git git-lfs --noconfirm ```

[dduportal]

@gounthar and @dduportal - Thoughts - #310

I looked into this more and jlinks is not working for arm/v7. I tried manually and even loading in some config tools. I am not an expert on java compilation. So instead I took the approach that we just outright copy the java runtime as is in the arm32 case. I consider this the best answer of a bad situation due to the error we are running into. I think us playing with BUILDPLATFORM environment is the wrong choice as we will run into this same error with potentially a different platform. So the least risky way to go about it also increases docker image file size (at least with my knowledge).

If we think it is good - I will clean it up properly.

Make sense : the additional size is not a problem if it is only on the arm32 images.

[carpnick]

Thank you all for feedback. I think #310 is ready to go.