ARM binaries

[j-cr]

It would be nice to have prebuilt ARM binaries too, so babashka could be run on a raspberry pi and the likes.

Related links: https://www.reddit.com/r/Clojure/comments/epowjk/clojure_for_iot_gpio_i2c/ https://github.com/dvlopt/clojure-raspberry-pi

[borkdude]

From the GraalVM slack, native-image channel:

[j-cr]

Also, from the reddit discussion: https://github.com/oracle/graal/issues/1329 so only newer 64-bit ARMs are supported, that means it would work only on raspberry 4 and future versions (but not on raspberry pi zero).

[borkdude]

More info:

https://github.com/rmh78/quarkus-simple

[borkdude]

@j-cr There is now a graalvm-ce-java8-linux-aarch64-20.0.0.tar.gz download available:

https://github.com/graalvm/graalvm-ce-builds/releases

[borkdude]

Waiting for someone stepping up to work on this. I'll re-open when there's any new activity.

[borkdude]

Which CIs can build for arm64?

• https://sourcehut.org/alpha-details/
• Travis?

[sogaiu]

I was able to build and run clj-kondo for aarch64, so I'm now trying babashka.

So far the following line needed to be commented out for the build to proceed:

https://github.com/borkdude/babashka/blob/548ffa851b37c9f0e28e4ae542b9fb72779128c5/script/compile#L20

[sogaiu]

[sogaiu]

The procedure basically involved:

• appropriate android device
• graalvm java 11 19.3.1 for aarch64
• termux
• https://wiki.termux.com/wiki/Arch
• installing git, gcc, wget, lein within the "prooted" arch distribution (except for lein, use pacman -S <package> to install)
• commenting out one line of script/compile
• usual invocation of script/compile

As you can see it takes over 10 minutes on this tablet.

[sogaiu]

Some tests fail and I noticed that (System/getenv "PATH") also doesn't succeed.

[sogaiu]

Here's a zip of the build.

bb.zip

[sogaiu]

Some additional notes:

• The bb binary would not execute properly outside of the specially prepared environment in which it was built. There seem to be a variety of restrictions in place on Android. It ran so-so inside the "proot", but not outside -- the same was true of the java binary that comes with graalvm.

• Currently trying to set up an aarch64 emulated environment (qemu) in which to test the built binary.

• I should be able to provide the emulated environment with more memory too. The original build was done on a Samsung Tab S4 with 6 GB of memory.

• I thought of trying a build with a Raspberry Pi 2, but a couple of things disuaded me. First, the default distributions for the RPi are not aarch64-based IIUC. Also, I think my RPi 2 only has 1 GB of RAM.

• IIUC, the Raspberry Pi 4 has a model that comes with 4 GB and apparently people have had luck making their own aarch64-based distributions, so may be that's another possible path?

[sogaiu]

Had some success setting up a qemu environment with aarch64. The following article was the best that I found on the topic: https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/

It was able to execute the bb binary built on the Android device with similar results to how it behaved within the "proot" environment.

It still showed similar errors, but borkdude mentioned that there was a known issue with jdk 11 and he has a fix:

https://github.com/borkdude/clj-reflector-graal-java11-fix

Am currently trying that out.

project.clj is being modified by:

• changing the version of [borkdude/sci.impl.reflector "0.0.1"] to [borkdude/sci.impl.reflector "0.0.1-jdk11"]
• adding the dependency [borkdude/clj-reflector-graal-java11-fix "0.0.1-graalvm-19.3.1-alpha.2"] -- this isn't supposed to really go in the value of the top-level :dependencies key, but for the sake of the test, it is being done that way

A complete build on the tablet takes around 20 min.

Building in the qemu environment is quite slow. Allocating 8 GB of memory doesn't seem to have helped with the speed, but perhaps that is not too surprising. For subsequent runs I will try increasing the number of cores as well as allocate some more disk space.

[sogaiu]

The resulting binary appears to behave better :)

Here it is:

bb.zip

[sogaiu]

Tangential, but here's clj-kondo:

clj-kondo.zip

[sogaiu]

It may be that building would be faster if one tweaks the -Xmx setting in the compile script.

[sogaiu]

It appears to help a lot (though also added disk space for more swap):

[bb:1935]      [total]: 7,017,894.94 ms

For comparison, the number for clj-kondo was:

[clj-kondo:1049]      [total]: 4,099,406.93 ms

These are long times, but the previous attempts did not complete.

So babashka is a bit under 2 hours and clj-kondo is more than 1 hour.

[sogaiu]

I tried tweaking -Xmx for the tablet but values of 4 and 5 gigs lead to termux being killed by Android if I'm interpreting things correctly.

So it looks like the default of 3g is somewhere close to the limit -- and that means taking in the neighborhood of 20 min.

[sogaiu]

@j-cr Regarding use on the RPi models, it does seem to be the case that typical distributions don't use aarch64 for RPi 2 and 3 (notably Raspbian).

It appears though that technically one can run aarch64 based distributions on RPi 2 and 3 (at least scanning some of the forum posts seem to suggest so). I haven't tried it, but FWIW.

Update: Some RPi 2 are 32-bit and some are 64-bit -- unfortunately, it appears mine is in the former category.

[sogaiu]

@borkdude I didn't manage to find a graalvm-ce-java8-linux-aarch64-20.0.0.tar.gz at: https://github.com/graalvm/graalvm-ce-builds/releases (checking via your comment: https://github.com/borkdude/babashka/issues/241#issuecomment-587925422)

Not sure if it was there at one point or not, but FWIW.

[sogaiu]

I was able to run an emulated rpi3 via qemu-system-aarch64 -- and the built bb seems to at least start there:

Last login: Thu Dec 14 22:10:24 UTC 2017 on ttyAMA0
Linux rpi3 4.14.0-3-arm64 #1 SMP Debian 4.14.12-2 (2018-01-06) aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Please change the root password by running passwd
root@rpi3:~# mv /boot/firmware/bb /root
root@rpi3:~# cd /root/
root@rpi3:~# ls -al
total 50984
drwx------  2 root root     4096 Dec 14 22:10 .
drwxr-xr-x 21 root root     4096 Jan  8  2018 ..
-rw-------  1 root root      248 Dec 14  2017 .bash_history
-rw-r--r--  1 root root      570 Jan 31  2010 .bashrc
-rw-r--r--  1 root root      148 Aug 17  2015 .profile
-rwxr-xr-x  1 root root 52184576 Apr  8  2020 bb
root@rpi3:~#
root@rpi3:~# uname -a
Linux rpi3 4.14.0-3-arm64 #1 SMP Debian 4.14.12-2 (2018-01-06) aarch64 GNU/Linux
root@rpi3:~# ./bb
Babashka v0.0.81-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=>

[sogaiu]

I followed the article here to set up an emulated RPi3 environment in qemu-system-aarch64: https://translatedcode.wordpress.com/2018/04/25/debian-on-qemus-raspberry-pi-3-model/

• the usual pi user didn't seem to be there, so used root
• the typical default password for rpi did work
• used qemu 5.0.0rc2 -- this lead to a gui starting and choosing View -> serial0 helped to see stuff on the screen
• there is no network support for the particular bits I used, so used guestfish to copy bb to one of the partitions (see the article for details -- one uses copy-in instead of copy-out and leave off the --ro flag when starting guestfish)

[sogaiu]

Apparently if one installs a 64-bit kernel (and possibly does some configuration tweaking), one may be able to run aarch64 binaries (even without userland being made 64-bit).

https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=250730

If you have such a setup, the binaries that have been posted already would be the ones to try.

[sogaiu]

I found this thread that mentions a number of different aarch64-based RPi distributions:

https://raspberrypi.stackexchange.com/questions/100926/64-bit-os-on-raspberry-pi-4/101802#101802

Two that stood out were:

• Manjaro's 64-bit RPi4 option
• Ubuntu's 64-bit RPi3 and RPi4 option

If you are more comfortable with Arch-based systems, the Manjaro option should probably be pretty familiar.

If you are more comfortable with Debian-based systems, the Ubuntu option might be one to try.

The thread is probably worth examining as it lists other options (though I urge caution in choice) as well as detailed instructions, tips, etc.

[iku000888]

Cheers on the work/research @sogaiu . Would be fun if it works on raspberry pis!

[borkdude]

Babashka is now built using java11. This means that the tweaks related to java11 in this thread should not be relevant anymore.

[sogaiu]

@iku000888 I think someone mentioned at #babashka on slack that they were going to try with a Raspberry Pi 4. Not sure how that turned out.

[borkdude]

Interesting article about compiling Java to native with GraalVM for Rasbperry:

https://quarkus.io/blog/quarkus-native-on-a-raspberry-pi/

[harryvederci]

@sogaiu the binaries you shared here work on my Raspberry Pi 4 (aarch64, kernel release 5.4.75-v8+).

I'd like to be able to build them myself from that device as well, though. I'm not familiar with the process, and reading the above I don't think I'd be able to reproduce what you did (example: "commenting out one line of script/compile").

Did you do it manually, or do you have a script that I could use for this purpose? If not: I'm really happy with the binaries, so thanks :)

[sogaiu]

@harryvederci Glad to hear things are working for you :)

Sorry I don't have a script -- however, I did note which line I commented out:

https://github.com/borkdude/babashka/issues/241#issuecomment-610688377

It might have been better to have put all of the pieces in one place -- my apologies wrt that.

I didn't check out the article borkdude mentioned: https://github.com/borkdude/babashka/issues/241#issuecomment-721634861

It could be that things are a lot easier now. May be it's worth looking at that?

[brdloush]

After reading this discussion, I wasn't really sure if it's possible to get a recent pre-compiled babashka for my raspberry 4b (aarch64). Or at least how easy/hard is it to build the native image myself.

TLDR: it's surprisingly easy to build bb on real raspberry pi 4 :+1: :clap:

Since I didn't find any recent pre-built aarch64 binaries in downloads, I tried to compile one myself.

I was building babashka directly on my raspberry pi 4, running ubuntu 20.04.1 LTS 64bit:

ubuntu@ubuntu:~/projects/babashka$ cat /proc/cpuinfo  | grep Model
Model       : Raspberry Pi 4 Model B Rev 1.4

ubuntu@ubuntu:~/projects/babashka$ uname -a
Linux ubuntu 5.4.0-1026-raspi #29-Ubuntu SMP PREEMPT Mon Dec 14 17:01:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

ubuntu@ubuntu:~/projects/babashka$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Compilation was easy. I only had one issue: missing libz.a file. After installing the missing dependency (I believe libz-dev did the job), the compilation worked like a charm! Here's I believe all the simple steps I had to do:

# get gcc, jdk, libz library
sudo apt install openjdk-11-jdk libz libz-dev build-essentials rlwrap curl

# get fresh copy of graalvm including native-image
mkdir ~/app ~/projects
cd ~/app
wget https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0/graalvm-ce-java11-linux-aarch64-21.0.0.tar.gz
tar xf graalvm-ce-java11-linux-aarch64-21.0.0.tar.gz
rm graalvm-ce-java11-linux-aarch64-21.0.0.tar.gz
export GRAALVM_HOME=/home/ubuntu/app/graalvm-ce-java11-21.0.0 # better to add this to ~/.bashrc
export PATH=$PATH:$GRAALVM_HOME # better to add this to ~/.bashrc
gu install native-image

# clone babashka and its child modules
cd ~/projects
git clone https://github.com/borkdude/babashka --recursive
cd babashka

# build uberjar
time ./script/uberjar
# real  1m53.673s
# user  4m29.222s
# sys   0m24.092s

# build native bb image
time ./script/compile # this takes a quite a long time (~ 20 minutes), but it's totally worth it...
# real  19m10.409s
# user  70m35.943s
# sys   0m29.795s

After compilation, bb binary seems to work just fine. Even my non-trivial babashka scripts which I use on my regular x86 amd64 laptop are working correctly. And bb itself is pretty fast on PI4! Well done! :clap:

ubuntu@ubuntu:~/projects/babashka$ ./bb --version
babashka v0.2.8-SNAPSHOT

ubuntu@ubuntu:~/projects/babashka$ time ./bb -e ':hello'
:hello

real    0m0.044s
user    0m0.013s
sys 0m0.033s

ubuntu@ubuntu:~/projects/babashka$ time ./bb -e '(-> (curl/get "https://postman-echo.com/get?foo=1") :body (json/parse-string true) :headers clojure.pprint/pprint)'
{:x-forwarded-proto "https",
 :x-forwarded-port "443",
 :host "postman-echo.com",
 :x-amzn-trace-id "Root=1-6008a83d-361da9650e87109f40d92ff1",
 :user-agent "curl/7.68.0",
 :accept "*/*",
 :accept-encoding "deflate, gzip, br"}

real    0m0.827s
user    0m0.099s
sys 0m0.070s

[borkdude]

@brdloush This is awesome! Do you think it would be worthwhile to distribute the Raspberry binary? I'm not sure how portable these binaries are - once compiled with GraalVM linux aarch64, they can run on any other Raspberry?

[brdloush]

As for the compatibility... well, I grabbed the binary that @sogaiu made and published in one of the previous posts (https://github.com/babashka/babashka/issues/241#issuecomment-610938199) and it worked without issues. I'd expect that if you grab that binary built for aarch64, it should be possible to run it on any device running 64bit arm cpu running 64bit linux OS. @sogaiu was using it on android tablet, so I believe the binary may be considered "quite portable" :smile:

For me babashka on raspberry makes perfect sense. Both It's startup time and runtime performance looks really good so far. I wonder... is there any bb benchmark that I could run on it?

I'm really looking forward to hacking some bb scripts that will just scrape some stuff from the web, parse it, process it and store it to some local storage. It will benefit from the fact that such script will be running on a tiny cheap, but relatively powerful computer which consumes really small amount of electricity and has no fans. In other words: it can run 24/7 in my bedroom.

If I'm not mistaken (?), currently just the 64-bit ARM native-images can be built by native-image. Which might be a problem for older Raspberries.. Most would probably be running Raspberry Pi OS (32-bit), Raspbian (32-bit) or Ubuntu (32-bit). The reason behind this is that older models of raspberries used to have smaller amount of memory (RPi3 still had only 1GB!). But with RPi4, there are 1GB, 2GB, 4GB and even 8GB options. And 8GB one costs around 75-100 USD. Running 64-bit OS on 8GB is a reasonable option I think.

According to list of distributions offered by the official Raspberry Pi Imager v1.5 tool, it seems that one can use 64-bit Ubuntu on RPi3, RPi4 and Rpi400.

[borkdude]

Feel free to zip the binary and upload it here for other people to try, as an "unofficial" distribution, like what @sogaiu did.

A tip for faster compilation: you can disable some of the libraries, if you aren't using most of them:

https://github.com/babashka/babashka/blob/master/doc/build.md#feature-flags

The BABASHKA_LEAN flag only keeps the most essential libs.

[brdloush]

Ok, here's the zipped aarch64 binary:

bb-aarch64-2021-01-20_git_rev_14c33625ecb203af2513eb5f527db63e77848ae5.zip

As for LEAN babashka: thanks for the tip. But 20 minutes in not that bad for building the real thing. But I'll try it, just for fun :)

[brdloush]

just FYI:

LEAN native image gets built in 16m12s (compared to 19m10.409s for default bb). It results in 56MB bb binary, default bb is 70M. From the startup performance perspective: they seem to start up in the same time.

[harryvederci]

Ok, here's the zipped aarch64 binary:

Works on my machine!

[den1k]

Incredibly exciting to imagine the possibilities of running babashka + datalevin on raspberry pi

And impeccable timing as @huahaiy is working on the missing piece: https://github.com/juji-io/datalevin/issues/6

[borkdude]

This is great. We can make a datalevin pod, like we already have SQL pods:

• https://github.com/babashka/babashka-sql-pods/
• https://github.com/babashka/pod-babashka-sqlite3

and/or add it as a feature flag and then that should work when also compiled on linux/aarch64.

[eigenhombre]

Super excited to see ARM progress for RPi. Question: I have a new Pi running Raspbian, which I think is still 32-bit. Is the new ARM Babashka 64-bit only?

[borkdude]

@eigenhombre Babashka is compiled using GraalVM and there is no 32-bit release of native-image for aarch.

[brdloush]

@eigenhombre you might want to look at this article. It seems that it's possible to use 32-bit raspbian (so all the installed libraries are 32-bit) and switch to 64-bit kernel. But be sure to backup before trying.

Also first make sure that your raspberry's CPU is actually 64-bit. If it's PI3 or PI4, it should be. Not sure about previous PIs.

[borkdude]

This babashka binary should work on RaspberryPi 64bit. It's built on CircleCI now!

https://15839-201467090-gh.circle-artifacts.com/0/release/babashka-0.2.14-SNAPSHOT-linux-aarch64.tar.gz

Anyone wants to try?

[brdloush]

Works on my RPI4

$ time ./bb -e ':hello'
:hello

real    0m0.045s
user    0m0.008s
sys 0m0.039s

[harryvederci]

Works on my Chromebook as well!

Bedankt/thanks Michiel!

[borkdude]

What does uname -m print for you? aarch64? I can use this to update the install script.

[harryvederci]

Exactly that on the Chromebook: aarch64

[borkdude]

Thanks! I updated the install script so it should recognize your architecture and install babashka correctly on aarch64.

[harryvederci]

Works! Thanks a lot!

[hamann]

I did the kernel upgrade from 32-bit to 64-bit today as mentioned in https://github.com/babashka/babashka/issues/241#issuecomment-764899056, but for me bb isn't working

~ pi@raspberrypi
❯ grep Model /proc/cpuinfo
Model       : Raspberry Pi 4 Model B Rev 1.2

~ pi@raspberrypi
❯ uname -a
Linux raspberrypi 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux

~ pi@raspberrypi
❯ file /usr/local/bin/bb
/usr/local/bin/bb: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=0dbe834f277b844079eba279ecd50988d91e213e, for GNU/Linux 3.7.0, with debug_info, not stripped

~ pi@raspberrypi
❯ /usr/local/bin/bb
zsh: no such file or directory: /usr/local/bin/bb

I guess a 64-bit userland is also required.