Cross-compiling for CI and release artifacts

[headius]

We need to set up some builds so we can regenerate the native binaries on the platforms we support. Recent changes for better BSD support have needed tweaks to the jffi native library, and the difficulty of getting new builds has basically killed our ability to evolve this small native part of JNR.

@smortex pointed out in #66 that Cirrus-CI has support for Windows, Linux, MacOS, and FreeBSD, which covers a lot of platforms in one place.

Travis CI obviously supports Linux and MacOS and early preview of Windows.

Azure only supports Windows and Linux, which I expect also applies to Azure DevOps and that whole pipelines thing.

Obviously we can set up VMs on any cloud service, or perhaps use e.g. Amazon AMIs for the x86-based operating systems we want, but this would be the most labor-intensive option.

If there's something virtual or Docker-like, or a comprehensive set of cross-compilation tools we can Docker down to a local system, that would be acceptable too.

cc @tuxillo @freaky @tduehr

[headius]

Related issues about missing, unbuildable, or out-of-date binaries: #49, #57, #56, #50, #49, #47, #38, #30, #9.

See also jnr/jnr-ffi#134 which is blocked in some cases by not having new builds of jffi for various BSDs.

[tuxillo]

Cirrus CI is the most realistic CI that will likely work for DragonFly BSD in the future, but for now we're stuck (see https://github.com/cirruslabs/cirrus-ci-docs/issues/101) without it either :-S

@headius would Vagrant be an acceptable solution? I think I can more or less automate the image generation.

[smortex]

@headius may you describe the manual steps involved in building native libraries and what happen to these binary blobs so that I undestand more precisely the big picture and what is needed?

As far as I am concerned, I can build binaries for supported FreeBSD releases (currently 11.2, 12.0) on amd64 and i386; each combination would be built in a jail on a FreeBSD 12.0 amd64 host. I can then do whatever is needed with these binaries. If needed, I can do this from time to time, but automating this sounds a good idea as @tuxillo suggests. In such a situation, I guess I can tweak a Vagrantfile to do the right thing with a FreeBSD node.

[vielmetti]

Drone Cloud has native x86, arm64, and 32-bit Arm support - https://cloud.drone.io/ - free for open source. No FreeBSD service at the moment but if you need other architectures.

[headius]

@vielmetti Excellent, thank you!

[vielmetti]

Ah, found some FreeBSD CI as well (but not arm64):

Sourcehut http://sr.ht Cirrus CI announcement https://forums.freebsd.org/threads/cirrus-ci-free-freebsd-ci-testing-for-open-source-projects.68560/

@headius I don't have direct experience with either of these two, but one or the other might be worth exploring.

[smortex]

AFAIK, FreeBSD arm64 packages are cross-build on amd64 with poudriere (the standard package building system for FreeBSD). So FreeBSD arm64 bare metal is probably not a strong requirement.

[james-crowley]

@headius I would be great if we could add cross-compiling for s390x. There is a couple options that I can think of right now, if neither work I can reach out and do more digging.

First option is to look at this, https://osuosl.org/services/ibm-z/. Looks like they have some sort of Jenkins setup that you could utilize. It looks like this all backed up the LinuxOne Community Cloud. Additionally, it seems a lot of their workload require some sort of containerized setup using docker.

Second option is to use the LinuxOne Community Cloud by itself. IBM has some resources for developers like the LinuxOne Community Cloud where you can develop against s390x for free. You would need to register and you get access to a s390x VM. Here is the link, https://developer.ibm.com/linuxone/ .

Let me know what you guys are interested in and we can would together to get something running.

[tduehr]

My original plan was to create some qemu VMs in linux. With the current Debian (and NetBSD) distribution platforms and improvements in qemu, this shouldn't be as difficult as when I was first looking at it a couple years ago.

[TobiX]

I'd chime in with suggesting something with crosscompilers (& maybe qemu-user). It's pretty simple to build cross-toolchains with Gentoo's crossdev tool (see for example, my Dockerfile for a PlayStation2 toolchain) - That would cover at least all different Linux platforms...

[tuxillo]

DragonFly BSD now can run in GCE, I just need to prepare an official image so that CirrusCI can use it.

[headius]

@TobiX That sounds great! I would really appreciate help getting at least a basic framework set up to build for Linux across platforms. I have never (successfully) set up a cross-compile build myself.

[TobiX]

So I have a quick&dirty Docker setup, which works impressively well.

Requirements: (Basically this article, but ignore the buildx stuff)

• qemu-user with working binfmt_misc setup
• recent docker
• "experimental": true in /etc/docker/daemon.json

Then do this:

docker run --rm -ti --platform linux/aarch64 -v $PWD:/work debian:9
apt-get update -y && apt-get install -y --no-install-recommends ant make gcc libc6-dev
cd /work
ant jar && ant archive-platform-jar

This is slow as molasses and burns CPU cycles like crazy (because it's running a compiler in qemu), but in the end you get a nice

build/jni/libjffi-1.2.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b066859494d47c5f2408bbca6fb45b94054a07e0, not stripped

This should work for all architectures for which qemu-user is available & which are supported by docker... You can probably get builds going for all architectures supported by qemu-user if you ditch docker & build chroots yourself...

PS: Does Debian support more architectures then jffi itself?

[headius]

@TobiX That's excellent, exactly what we've been looking for!

If I'm reading you right, this will at least give us Debian-shaped Linux builds, at least initially for Docker-supported platforms and possibly beyond that if we do a more elaborate construction of our own root environment. That is at least a good start.

PS: Does Debian support more architectures then jffi itself?

That does appear to be the case. Currently jffi ships with the following platform binaries:

• amd64
• i386
• arm64
• "arm" which is most likely armhf (ARMv7)
• mips64el
• "ppc" (ppc 32-bit)
• ppc64
• ppc64le (ppc64el)
• s390x
• sparcv9 (sparc64)

Of these, only ppc is unsupported. We could add a few platforms based on the Debian list, but they're pretty obscure.

Would you be interested in throwing together a quick example Github Actions workflow for arm64? I can extrapolate that to other platforms, and then maybe we can actually try to test on platforms that also have Java support. That would be huge.

[headius]

A general reminder that there's really two goals here:

1. To be able to build the binary for all supported jffi platforms (which could be done offline/locally or periodically as part of CI)
2. To be able to test jffi and the rest of the JNR libraries on all supported jffi platforms (which should be done with every push in CI)

Incrementally approaching either of these goals would be fantastic, and even just getting (2) working on all Linux environments would be a big step forward.

[splatch]

I can bring an example of (javacan) where author used https://github.com/dockcross/dockcross for cross compiling. This method does not depend on any specific build tool except docker..

[headius]

@splatch I would be very interested to see that! At this point I see no reason not to merge this in as is since we will at least have the Linux builds covered, but anything we can do to cross compile more easily will be a great help.

Ultimate goal is to get all supported platforms building automatically using whatever combination of tools (cross-compilers, emulation environments) so we can start making long-delayed improvements at the native level.

[headius]

At this point I see no reason not to merge this

Referring to the existing PR #90.

[TobiX]

@splatch I'm always a bit conflicted in the cross-compile vs. native compilation discussions. I still prefer native compilation (even inside emulators) over cross-compilers whenever possible (and not too slow). Especially here, where we are talking about a couple of thousand lines of code and the setup is as simple as running docker with some obscure options :wink:

[james-crowley]

@TobiX I agree that cross-compile is not ideal. In some case with low level code like C cross-compiling can lead to some bugs. I seen this happen on when porting amd64 applications to s390x and other platforms. This does not happen always, it really depends on which parts of the hardware you are touching. For example tasks like crypto, decompression and compression are areas you usually run into issues with.

@headius If you are looking for hardware to build on Travis CI offers most of the architectures you listed for free. This includes support for s390x and ppc64le. If you did not want to go down that route, I know other open source projects I work with get sponsors to support them on various platforms. AWS now offers support for arm64 and IBM has resources that you can use for free.

[headius]

@james-crowley @TobiX The other issue with cross-compiling is that we would ideally like to verify those binaries, which necessitates running the test suites in a fully functional environment. Building the binaries is just the first step.

@james-crowley Yeah no reason we could not hook up those envs on Travis. We used to test there but I moved to GHA for simplicity some weeks ago. I also have a standing invite from AWS folks to get an arm64 environment. My larger concern is all the weirder platforms we do not have any way to access, like specific sparc variants of Solaris. Baby steps, I guess.

[TobiX]

@headius It seems Travis-CI isn't a good idea nowadays: https://twitter.com/james_hilliard/status/1336081776691843072 & https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing :frowning_face:

[headius]

@TobiX Getting back to this... we have moved our CI to GHA and have a working cross-platform build (for all Linux we can support through qemu + docker) which should form a base for adding other platforms (qemu + something? Docker-based cross-compilers?). Now is a good time for us to look into this, since we will need rebuilds to support more modern ffi varargs logic (particularly on Apple M1).