Package for Linux aarch64

[axiopaladin]

Can Linux aarch64 be added to the list for standard build targets?

I want to run JabRef on my ARM Linux devices. Unfortunately, the only JabRef builds released for Linux right now are for x86-based architectures.

I'll work on trying to compile a binary for my own use, but I figure it would be a nice addition to the CI pipeline too.

[Siedlerchr]

@axiopaladin We would offer an arm-based image; however, GitHub currently doesn't provide an ARM based hosted runner ,only for self-hosting.

ARM based runners are only in closed beta right now https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

For self compiling: You need to clone JabRef and init the submodules. If you want to compile it yourself, you basically need to have jdk21 + gradle Simply running ./gradlew jpackage will be sufficient to build the executables.

See also: https://devdocs.jabref.org/getting-into-the-code/guidelines-for-setting-up-a-local-workspace/

[axiopaladin]

Thanks for the tips on self-compiling :)

Unfortunately it seems some of the build dependencies (notably: openjfx) are not packaged for aarch64; it throws this error when partway through:

> Configure project :
Project : => 'org.jabref' Java module

> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find javafx-fxml-21.0.2-linux-aarch64.jar (org.openjfx:javafx-fxml:21.0.2).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/org/openjfx/javafx-fxml/21.0.2/javafx-fxml-21.0.2-linux-aarch64.jar
   > Could not find javafx-web-21.0.2-linux-aarch64.jar (org.openjfx:javafx-web:21.0.2).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/org/openjfx/javafx-web/21.0.2/javafx-web-21.0.2-linux-aarch64.jar
   > Could not find javafx-controls-21.0.2-linux-aarch64.jar (org.openjfx:javafx-controls:21.0.2).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/org/openjfx/javafx-controls/21.0.2/javafx-controls-21.0.2-linux-aarch64.jar
   > Could not find javafx-swing-21.0.2-linux-aarch64.jar (org.openjfx:javafx-swing:21.0.2).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/org/openjfx/javafx-swing/21.0.2/javafx-swing-21.0.2-linux-aarch64.jar
   > Could not find javafx-media-21.0.2-linux-aarch64.jar (org.openjfx:javafx-media:21.0.2).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/org/openjfx/javafx-media/21.0.2/javafx-media-21.0.2-linux-aarch64.jar
   > Could not find javafx-graphics-21.0.2-linux-aarch64.jar (org.openjfx:javafx-graphics:21.0.2).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/org/openjfx/javafx-graphics/21.0.2/javafx-graphics-21.0.2-linux-aarch64.jar
   > Could not find javafx-base-21.0.2-linux-aarch64.jar (org.openjfx:javafx-base:21.0.2).
     Searched in the following locations:
         https://repo.maven.apache.org/maven2/org/openjfx/javafx-base/21.0.2/javafx-base-21.0.2-linux-aarch64.jar

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 2m 41s
7 actionable tasks: 6 executed, 1 up-to-date

---

To replicate this error, use this docker-compose.yml file on your favorite aarch64 machine with docker-compose up -d:

version: "3.9"

services:
  gradle:
    image: arm64v8/gradle:jdk21
    network_mode: host
    restart: on-failure
    tty: true
    command:
      - bash
      - -ec
      - |
        echo "Container started"
        sleep infinity

Then use docker exec -it {container_name} bash to get inside, and then git clone https://github.com/JabRef/jabref.git && ./jabref/gradlew jpackage and it will eventually run into the error shown above.

[Siedlerchr]

Ah seems like they have not yet? released the aarch64 binaries for Linux in jfx 21.0.2, in 21.0.1. they are present e.g.

You could change that version in the build.gradle file https://repo.maven.apache.org/maven2/org/openjfx/javafx-fxml/21.0.1/

When cloning the repo you need to include the submodules for the csl styles and abbrevs git clone --recurse-submodules https://github.com/JabRef/jabref.git

[axiopaladin]

@Siedlerchr thanks for the tips. I apologize for my last post being a little unfocused, I'm glad you were able to decipher it! I was unfamiliar with gradle so I've been learning as I go here; I think I'm finally barely starting to understand how build.gradle works.

I was able to get something built! I had to do a few hacks along the way, though:

1. apt update; apt install fakeroot inside this Ubuntu-based arm64v8/gradle:jdk21 docker image
2. Add the following changes to build.gradle:

   javafx { // <-- line 103 of build.gradle
   version = "22-ea+27" // changed to most recent version with aarch64 binaries released
   modules = [ 'javafx.controls', 'javafx.fxml', 'javafx.web', 'javafx.swing' ]
   }

            installerOptions = [
                    '--verbose',
                    '--type', 'deb', // <-- add this here, at line 660
                    '--vendor', 'JabRef',
                    '--app-version', "${project.version}",
                    //'--temp', "$buildDir/installer",
                    '--resource-dir', "${projectDir}/buildres/linux",
                    '--linux-menu-group', 'Office;',
                    // '--linux-rpm-license-type', 'MIT',
                    // '--license-file', "${projectDir}/LICENSE.md",
                    '--description', 'JabRef is an open source bibliography reference manager. The native file format used by>
                    '--linux-shortcut',
                    '--file-associations', "${projectDir}/buildres/linux/bibtexAssociations.properties"
            ]

After all that, a ./gradlew jpackage ended in a successful build!

However, I'm not sure how to export a useful binary from this. When I try to use dpkg to install the .deb that is produced (in build/distribution/), inside the docker container, it complains about dependencies, so clearly that isn't self-contained. If I try to run the executable binary build/distribution/JabRef/bin/JabRef inside the docker container, it complains about not having a display (which is reasonable), and if I try it outside the container it throws this error:

Error occurred during initialization of VM
Unable to load jimage library: jabref/build/distribution/JabRef/lib/runtime/lib/libjimage.so

I apologize if this is a foolish question, as I am not even a novice in gradle-fu, but how difficult would it be to convince jpackage to spit out a static/portable binary for the project as a whole? (Or perhaps the more appropriate solution would be to build a flatpak or similar containerized solution instead?)

In the releases for this project, I suppose the closest equivalent to what I'm looking for would be the JabRef-X.xx-portable_linux.tar.gz option (but for aarch64 instead of x86_64, of course). I thought that was what build/distribution/JabRef contained, but apparently not? I can run the official release just fine (on an x86 machine) but my own compiled binary doesn't work at all on my aarch64 machine.

[Siedlerchr]

Hi, your approach looks already good, for the portable it would be sufficient to call ./gradlew jlinkzip The application should also be self-contained with the java runtime.
can you integrate a step in the docker file to print out the java version? Should be java -versionand verify it's 21.0.1 or 21.0.2 (important)

[axiopaladin]

# java -version
openjdk version "21.0.2" 2024-01-16 LTS
OpenJDK Runtime Environment Temurin-21.0.2+13 (build 21.0.2+13-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.2+13 (build 21.0.2+13-LTS, mixed mode, sharing)

I've rebuilt with ./gradlew jlinkzip and I've gotten (what I assume is) the output at build/image.zip. However, it throws that same Error occurred during initialization of VM Unable to load jimage library message when I tried unzipping it and running the image/bin/JabRef binary.

My primary ARM system (that I've been using to test these) is running an Alpine-based distribution, perhaps some glibc dependency has leaked into the JabRef output and that is the real issue? I don't have any other ARM systems handy right now (...maybe this is an excuse to order another raspberry pi 🤔) so I'll just upload my image.zip here. If someone else has a non-Alpine aarch64 machine and feels like testing it out, that would be helpful :) Otherwise I'll put together a live USB or something to test with eventually.

[axiopaladin]

I've tried building on an Alpine system with the following (and much more elegant) docker-compose.yml, but this one fails to even build successfully!

version: "3.9"

services:
  alpine:
    image: alpine:latest
    network_mode: host
    restart: on-failure
    tty: true
    volumes:
      - ./working:/working:rw
    command: >
      sh -c 'apk update &&
             apk add --no-interactive git openjdk21 &&
             cd /working &&
             git clone --recurse-submodules https://github.com/JabRef/jabref.git 2> /dev/null ||
             (cd jabref ; git pull) &&
             java -version &&
             cd /working/jabref &&
             sed -i "s/21.0.2/22-ea+27/g" build.gradle &&
             ./gradlew jlinkzip ;
             ash'

I've automated all the important bits in this one, so just cd into the same folder and docker-compose run -i alpine and it should go through the built-in script to grab everything necessary and build. (At the end, I have it leave you at a command prompt in case you want to do more before shutting down the container.)

Gradle throws the following strange error when I run it on my aarch64 machine:

> Task :processResources FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/working/jabref/build.gradle' line: 285

* What went wrong:
Execution failed for task ':processResources'.
> Could not list contents of directory '/working/jabref/src/main/resources'.

I've checked the file permissions of that src/main/resources folder and I can't figure out why it would be inaccessible to java. I'll have to give it a try on an x86_64 machine later and see if it can build successfully there...

[Siedlerchr]

Hm, what I found online is only rare, there someone was using alpine as docker base image and had the same image... Apparently, adding libc6-compat or something like gcompat helped. But I have no idea.

You. need binutils for jlink as well

Edit// gradle jdk 21 also offers alpine image 8.6.0-jdk21-alpine, 8.6-jdk21-alpine, 8-jdk21-alpine, jdk21-alpine

[axiopaladin]

• I had gcompat and libc6-compat on my base system already, so unfortunately it seems they can't solve the issue :(
• I've just now tried adding binutils to the Alpine-based docker build and it does not get me past that "can't access src/main/resources" problem, so I don't think it's that.
• Unfortunately the alpine-based Gradle docker images are only for x86_64, none for ARM.
• I get the same "unable to load jimage library" when I try running the build/distribution/lib/runtime/bin/JabRef binary

[koppor]

* I've just now tried adding binutils to the Alpine-based docker build and it does not get me past that "can't access `src/main/resources`" problem, so I don't think it's that.

How did you start the Docker image? I think, something is wrong with the -v command for docker.

* Unfortunately the alpine-based Gradle docker images are only for x86_64, none for ARM.

One can IMHO easily install a JDK on a ARM based alpine docker image.

[axiopaladin]

Alright, I've tried building with the exact same script on both aarch64 and x86_64. Here's my latest and greatest docker-compose.yml file:

version: "3.9"

services:
  alpine:
    image: alpine:latest
    network_mode: host
    restart: on-failure
    tty: true
    volumes:
      - ./working:/working:rw
    command: >
      sh -c 'apk update &&
             apk add --no-interactive binutils dpkg fakeroot git openjdk21 rpm &&
             cd /working &&
             git clone --recurse-submodules https://github.com/JabRef/jabref.git ||
             (cd jabref ; git pull) &&
             java -version &&
             cd /working/jabref &&
             sed -i "s/21.0.2/22-ea+27/g" build.gradle &&
             ./gradlew jlinkzip ;
             ash'

It automatically installs all the necessary prerequisites, pulls the JabRef sources from GitHub, and runs the build process. (For @koppor -- put the docker-compose.yml in an empty directory and docker-compose run -i alpine from there to start it up and attach your terminal to it.)

On my x86_64 machine, it builds just fine; on my ARM machine, it fails partway through with the following error:

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.For more details on the daemon, please refer to https://docs.gradle.org/8.5/userguide/gradle_daemon.html in the Gradle documentation.
Process command line: /usr/lib/jvm/java-21-openjdk/bin/java --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED -Xmx4096M -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -cp /root/.gradle/wrapper/dists/gradle-8.5-bin/5t9huq95ubn472n8rpzujfbqh/gradle-8.5/lib/gradle-launcher-8.5.jar -javaagent:/root/.gradle/wrapper/dists/gradle-8.5-bin/5t9huq95ubn472n8rpzujfbqh/gradle-8.5/lib/agents/gradle-instrumentation-agent-8.5.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 8.5
Please read the following process output to find out more:
-----------------------
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000000002b1c0, pid=162, tid=165
#
# JRE version: OpenJDK Runtime Environment (21.0.2+13) (build 21.0.2+13-alpine-r0)
# Java VM: OpenJDK 64-Bit Server VM (21.0.2+13-alpine-r0, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# C  [libnative-platform-file-events.so+0x2eb8c]  _init+0x39e4
#
# Core dump will be written. Default location: /root/.gradle/daemon/8.5/core
#
# An error report file with more information is saved as:
# /root/.gradle/daemon/8.5/hs_err_pid162.log

I'll attach that hs_err_pid162.log here, for more details. But TL;DR, it looks like it's an issue with the gradle daemon, not JabRef?

---

That said, this exercise wasn't completely fruitless. One interesting thing that I noticed was that there does seem to be a difference between the images built on Ubuntu-based toolchains and those built on Alpine-based toolchains. Running ldd {unzipped image.zip}/bin/java, for example, shows that the Alpine-created binary relies on dynamic links specifically to Alpine's musl libraries:

$ ldd bin/java
        /lib/ld-musl-x86_64.so.1 (0x7f72664f0000)
        libjli.so => bin/../lib/libjli.so (0x7f72664db000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f72664f0000)
        libz.so.1 => /lib/libz.so.1 (0x7f72664c1000)

I think this explains why the aarch64 image I was able to build with the (Ubuntu-based) arm64v8/gradle:jdk21 container wouldn't run on my Alpine-based main system.

[Siedlerchr]

Ah yes, I remember gradle has problems with the deamon in docker! try running with --no-deamon

or gradle.properties org.gradle.daemon=false

[axiopaladin]

Ah yes, I remember gradle has problems with the deamon in docker! try running with --no-deamon

I tried with --no-daemon (per documentation), but unfortunately it seems that a daemon is mandatory for this build, so it just throws this warning before having the same crash:

# ./gradlew --no-daemon jlinkzip
> To honour the JVM settings for this build a single-use Daemon process will be forked.
> For more on this, please refer to
> https://docs.gradle.org/8.6/userguide/gradle_daemon.html#sec:disabling_the_daemon 
> in the Gradle documentation.

FAILURE: Build failed with an exception.

{rest of the error is the same as before}

[koppor]

This refs https://github.com/actions/runner-images/issues/5631, where GitHub runner support for Linux ARM64 is requested.

[koppor]

@axiopaladin I think, the next step is to make a minimal working example to see which part of the whole toolchain is the issue. A good start is the https://github.com/Siedlerchr/javafxreproducer project. Can this run on your aarch64 machine?

[koppor]

@axiopaladin Please check the aarch64 build at https://builds.jabref.org/pull/10867/merge/.

[axiopaladin]

Simplest task first: The published JabRef-5.13-portable_linux-aarch64.tar.gz build doesn't run on my machine, but I think that's an Alpine problem rather than a problem with the image. Trying to run it gives the same "unable to load jimage library" error mentioned earlier in this thread. When I examine it with ldd, it shows a Error relocating JabRef: __strdup: symbol not found which suggests to me that it is an incompatibility between the image (built against glibc) and my system (which uses musl).

I've also tried the javafxreproducer project and it fails at the same "unable to start daemon process" point as mentioned above. After doing a bit more research, it looks like this might be due to a longstanding bug in Gradle on aarch64; see https://github.com/gradle/gradle/issues/24875. However, it seems that the Gradle bug has some candidate fixes in PR right now, so hopefully it will be resolved in the next release.

[koppor]

TBH, I don't think, a fix of gradle will fix the JabRef jimage issue. For instance, the error you reported is appearing in Google at other places, For instance at the comment section at https://mkyong.com/java/how-to-install-java-on-mac-osx/ and https://bugs.openjdk.org/browse/JDK-8017234?focusedId=14344518&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel.

[koppor]

@axiopaladin Could you try https://builds.jabref.org/jdk-ea/JabRef-5.13-portable_linux-arm64.tar-jdk23-javafx22-ea%2B28.gz (resides inside https://builds.jabref.org/jdk-ea/, where also other arm64 variants are available).

[axiopaladin]

Took another stab at this problem today. Unfortunately the arm64 builds published on builds.jabref.org do not work on my Alpine system (probably because they're built against glibc and not musl).

However, I was able to get further in compiling my own build! The daemon bug is fixed in the latest nightly builds for gradle (to be released in 8.8, whenever that is finished), so after a few edits to jabref/gradle/wrapper/gradle-wrapper.properties (to change the distribution URL and sha256 checksum) the script attempts to use the nightly version of gradle. However, once the daemon runs, the process stops at one of the same roadblocks I hit before: Task :processResources. It throws the same Could not list contents of directory 'src/main/resources' error as before, and I have no idea what causes this error.

What should I do in order to diagnose this further?

[Siedlerchr]

Do you have a symlink somewhere in the folder path or placed your project folder somewhere under a dir with bin? E.g /opt/bin/jabref?

Otherwise try with the gradle debug or stack trace to see if you get any meaningful hints

[koppor]

@axiopaladin What would help in debugging is a Dockerfile with the commands. With https://github.com/multiarch/qemu-user-static, other interested persons should be able to run the ARM image also on other architectures.

[koppor]

As of today: https://search.maven.org/artifact/org.openjfx/javafx/23-ea%2B22/pom seems to offer aarch64 for linux

                <property>
                    <name>javafx.monocle</name>
                    <value>true</value>
                </property>

https://wiki.openjdk.org/display/OpenJFX/Monocle