search

babashka / babashka

Native, fast starting Clojure interpreter for scripting
https://babashka.org
Eclipse Public License 1.0
4.04k stars 248 forks source link

ARM binaries #241

Closed j-cr closed 3 years ago

j-cr commented 4 years ago

It would be nice to have prebuilt ARM binaries too, so babashka could be run on a raspberry pi and the likes.

Related links: https://www.reddit.com/r/Clojure/comments/epowjk/clojure_for_iot_gpio_i2c/ https://github.com/dvlopt/clojure-raspberry-pi

borkdude commented 4 years ago

From the GraalVM slack, native-image channel:

Screenshot 2020-01-21 17 26 37
j-cr commented 4 years ago

Also, from the reddit discussion: https://github.com/oracle/graal/issues/1329 so only newer 64-bit ARMs are supported, that means it would work only on raspberry 4 and future versions (but not on raspberry pi zero).

borkdude commented 4 years ago

More info:

Screenshot 2020-02-10 21 07 42

https://github.com/rmh78/quarkus-simple

borkdude commented 4 years ago

@j-cr There is now a graalvm-ce-java8-linux-aarch64-20.0.0.tar.gz download available:

https://github.com/graalvm/graalvm-ce-builds/releases

borkdude commented 4 years ago

Waiting for someone stepping up to work on this. I'll re-open when there's any new activity.

borkdude commented 4 years ago

Which CIs can build for arm64?

sogaiu commented 4 years ago

I was able to build and run clj-kondo for aarch64, so I'm now trying babashka.

So far the following line needed to be commented out for the build to proceed:

https://github.com/borkdude/babashka/blob/548ffa851b37c9f0e28e4ae542b9fb72779128c5/script/compile#L20

sogaiu commented 4 years ago

babashka-aarch64

sogaiu commented 4 years ago

The procedure basically involved:

As you can see it takes over 10 minutes on this tablet.

sogaiu commented 4 years ago

Some tests fail and I noticed that (System/getenv "PATH") also doesn't succeed.

sogaiu commented 4 years ago

Here's a zip of the build.

bb.zip

sogaiu commented 4 years ago

Some additional notes:

sogaiu commented 4 years ago

Had some success setting up a qemu environment with aarch64. The following article was the best that I found on the topic: https://translatedcode.wordpress.com/2017/07/24/installing-debian-on-qemus-64-bit-arm-virt-board/

It was able to execute the bb binary built on the Android device with similar results to how it behaved within the "proot" environment.

It still showed similar errors, but borkdude mentioned that there was a known issue with jdk 11 and he has a fix:

https://github.com/borkdude/clj-reflector-graal-java11-fix

Am currently trying that out.

project.clj is being modified by:

A complete build on the tablet takes around 20 min.

Building in the qemu environment is quite slow. Allocating 8 GB of memory doesn't seem to have helped with the speed, but perhaps that is not too surprising. For subsequent runs I will try increasing the number of cores as well as allocate some more disk space.

sogaiu commented 4 years ago

The resulting binary appears to behave better :)

Here it is:

bb.zip

sogaiu commented 4 years ago

Tangential, but here's clj-kondo:

clj-kondo.zip

sogaiu commented 4 years ago

It may be that building would be faster if one tweaks the -Xmx setting in the compile script.

sogaiu commented 4 years ago

It appears to help a lot (though also added disk space for more swap):

[bb:1935]      [total]: 7,017,894.94 ms

For comparison, the number for clj-kondo was:

[clj-kondo:1049]      [total]: 4,099,406.93 ms

These are long times, but the previous attempts did not complete.

So babashka is a bit under 2 hours and clj-kondo is more than 1 hour.

sogaiu commented 4 years ago

I tried tweaking -Xmx for the tablet but values of 4 and 5 gigs lead to termux being killed by Android if I'm interpreting things correctly.

So it looks like the default of 3g is somewhere close to the limit -- and that means taking in the neighborhood of 20 min.

sogaiu commented 4 years ago

@j-cr Regarding use on the RPi models, it does seem to be the case that typical distributions don't use aarch64 for RPi 2 and 3 (notably Raspbian).

It appears though that technically one can run aarch64 based distributions on RPi 2 and 3 (at least scanning some of the forum posts seem to suggest so). I haven't tried it, but FWIW.

Update: Some RPi 2 are 32-bit and some are 64-bit -- unfortunately, it appears mine is in the former category.

sogaiu commented 4 years ago

@borkdude I didn't manage to find a graalvm-ce-java8-linux-aarch64-20.0.0.tar.gz at: https://github.com/graalvm/graalvm-ce-builds/releases (checking via your comment: https://github.com/borkdude/babashka/issues/241#issuecomment-587925422)

Not sure if it was there at one point or not, but FWIW.

sogaiu commented 4 years ago

I was able to run an emulated rpi3 via qemu-system-aarch64 -- and the built bb seems to at least start there:

Last login: Thu Dec 14 22:10:24 UTC 2017 on ttyAMA0
Linux rpi3 4.14.0-3-arm64 #1 SMP Debian 4.14.12-2 (2018-01-06) aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Please change the root password by running passwd
root@rpi3:~# mv /boot/firmware/bb /root
root@rpi3:~# cd /root/
root@rpi3:~# ls -al
total 50984
drwx------  2 root root     4096 Dec 14 22:10 .
drwxr-xr-x 21 root root     4096 Jan  8  2018 ..
-rw-------  1 root root      248 Dec 14  2017 .bash_history
-rw-r--r--  1 root root      570 Jan 31  2010 .bashrc
-rw-r--r--  1 root root      148 Aug 17  2015 .profile
-rwxr-xr-x  1 root root 52184576 Apr  8  2020 bb
root@rpi3:~#
root@rpi3:~# uname -a
Linux rpi3 4.14.0-3-arm64 #1 SMP Debian 4.14.12-2 (2018-01-06) aarch64 GNU/Linux
root@rpi3:~# ./bb
Babashka v0.0.81-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=>
sogaiu commented 4 years ago

I followed the article here to set up an emulated RPi3 environment in qemu-system-aarch64: https://translatedcode.wordpress.com/2018/04/25/debian-on-qemus-raspberry-pi-3-model/

sogaiu commented 4 years ago

Apparently if one installs a 64-bit kernel (and possibly does some configuration tweaking), one may be able to run aarch64 binaries (even without userland being made 64-bit).

https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=250730

If you have such a setup, the binaries that have been posted already would be the ones to try.

sogaiu commented 4 years ago

I found this thread that mentions a number of different aarch64-based RPi distributions:

https://raspberrypi.stackexchange.com/questions/100926/64-bit-os-on-raspberry-pi-4/101802#101802

Two that stood out were:

If you are more comfortable with Arch-based systems, the Manjaro option should probably be pretty familiar.

If you are more comfortable with Debian-based systems, the Ubuntu option might be one to try.

The thread is probably worth examining as it lists other options (though I urge caution in choice) as well as detailed instructions, tips, etc.

iku000888 commented 4 years ago

Cheers on the work/research @sogaiu . Would be fun if it works on raspberry pis!

borkdude commented 4 years ago

Babashka is now built using java11. This means that the tweaks related to java11 in this thread should not be relevant anymore.

sogaiu commented 4 years ago

@iku000888 I think someone mentioned at #babashka on slack that they were going to try with a Raspberry Pi 4. Not sure how that turned out.

borkdude commented 3 years ago

Interesting article about compiling Java to native with GraalVM for Rasbperry:

https://quarkus.io/blog/quarkus-native-on-a-raspberry-pi/

harryvederci commented 3 years ago

@sogaiu the binaries you shared here work on my Raspberry Pi 4 (aarch64, kernel release 5.4.75-v8+).

I'd like to be able to build them myself from that device as well, though. I'm not familiar with the process, and reading the above I don't think I'd be able to reproduce what you did (example: "commenting out one line of script/compile").

Did you do it manually, or do you have a script that I could use for this purpose? If not: I'm really happy with the binaries, so thanks :)

sogaiu commented 3 years ago

@harryvederci Glad to hear things are working for you :)

Sorry I don't have a script -- however, I did note which line I commented out:

https://github.com/borkdude/babashka/issues/241#issuecomment-610688377

It might have been better to have put all of the pieces in one place -- my apologies wrt that.

I didn't check out the article borkdude mentioned: https://github.com/borkdude/babashka/issues/241#issuecomment-721634861

It could be that things are a lot easier now. May be it's worth looking at that?

brdloush commented 3 years ago

After reading this discussion, I wasn't really sure if it's possible to get a recent pre-compiled babashka for my raspberry 4b (aarch64). Or at least how easy/hard is it to build the native image myself.

TLDR: it's surprisingly easy to build bb on real raspberry pi 4 :+1: :clap:

Since I didn't find any recent pre-built aarch64 binaries in downloads, I tried to compile one myself.

I was building babashka directly on my raspberry pi 4, running ubuntu 20.04.1 LTS 64bit:

ubuntu@ubuntu:~/projects/babashka$ cat /proc/cpuinfo  | grep Model
Model       : Raspberry Pi 4 Model B Rev 1.4
ubuntu@ubuntu:~/projects/babashka$ uname -a
Linux ubuntu 5.4.0-1026-raspi #29-Ubuntu SMP PREEMPT Mon Dec 14 17:01:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@ubuntu:~/projects/babashka$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Compilation was easy. I only had one issue: missing libz.a file. After installing the missing dependency (I believe libz-dev did the job), the compilation worked like a charm! Here's I believe all the simple steps I had to do:

# get gcc, jdk, libz library
sudo apt install openjdk-11-jdk libz libz-dev build-essentials rlwrap curl

# get fresh copy of graalvm including native-image
mkdir ~/app ~/projects
cd ~/app
wget https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-21.0.0/graalvm-ce-java11-linux-aarch64-21.0.0.tar.gz
tar xf graalvm-ce-java11-linux-aarch64-21.0.0.tar.gz
rm graalvm-ce-java11-linux-aarch64-21.0.0.tar.gz
export GRAALVM_HOME=/home/ubuntu/app/graalvm-ce-java11-21.0.0 # better to add this to ~/.bashrc
export PATH=$PATH:$GRAALVM_HOME # better to add this to ~/.bashrc
gu install native-image

# clone babashka and its child modules
cd ~/projects
git clone https://github.com/borkdude/babashka --recursive
cd babashka

# build uberjar
time ./script/uberjar
# real  1m53.673s
# user  4m29.222s
# sys   0m24.092s

# build native bb image
time ./script/compile # this takes a quite a long time (~ 20 minutes), but it's totally worth it...
# real  19m10.409s
# user  70m35.943s
# sys   0m29.795s

After compilation, bb binary seems to work just fine. Even my non-trivial babashka scripts which I use on my regular x86 amd64 laptop are working correctly. And bb itself is pretty fast on PI4! Well done! :clap: 

ubuntu@ubuntu:~/projects/babashka$ ./bb --version
babashka v0.2.8-SNAPSHOT
ubuntu@ubuntu:~/projects/babashka$ time ./bb -e ':hello'
:hello

real    0m0.044s
user    0m0.013s
sys 0m0.033s
ubuntu@ubuntu:~/projects/babashka$ time ./bb -e '(-> (curl/get "https://postman-echo.com/get?foo=1") :body (json/parse-string true) :headers clojure.pprint/pprint)'
{:x-forwarded-proto "https",
 :x-forwarded-port "443",
 :host "postman-echo.com",
 :x-amzn-trace-id "Root=1-6008a83d-361da9650e87109f40d92ff1",
 :user-agent "curl/7.68.0",
 :accept "*/*",
 :accept-encoding "deflate, gzip, br"}

real    0m0.827s
user    0m0.099s
sys 0m0.070s
borkdude commented 3 years ago

@brdloush This is awesome! Do you think it would be worthwhile to distribute the Raspberry binary? I'm not sure how portable these binaries are - once compiled with GraalVM linux aarch64, they can run on any other Raspberry?

brdloush commented 3 years ago

As for the compatibility... well, I grabbed the binary that @sogaiu made and published in one of the previous posts (https://github.com/babashka/babashka/issues/241#issuecomment-610938199) and it worked without issues. I'd expect that if you grab that binary built for aarch64, it should be possible to run it on any device running 64bit arm cpu running 64bit linux OS. @sogaiu was using it on android tablet, so I believe the binary may be considered "quite portable" :smile:

For me babashka on raspberry makes perfect sense. Both It's startup time and runtime performance looks really good so far. I wonder... is there any bb benchmark that I could run on it?

I'm really looking forward to hacking some bb scripts that will just scrape some stuff from the web, parse it, process it and store it to some local storage. It will benefit from the fact that such script will be running on a tiny cheap, but relatively powerful computer which consumes really small amount of electricity and has no fans. In other words: it can run 24/7 in my bedroom.

If I'm not mistaken (?), currently just the 64-bit ARM native-images can be built by native-image. Which might be a problem for older Raspberries.. Most would probably be running Raspberry Pi OS (32-bit), Raspbian (32-bit) or Ubuntu (32-bit). The reason behind this is that older models of raspberries used to have smaller amount of memory (RPi3 still had only 1GB!). But with RPi4, there are 1GB, 2GB, 4GB and even 8GB options. And 8GB one costs around 75-100 USD. Running 64-bit OS on 8GB is a reasonable option I think.

According to list of distributions offered by the official Raspberry Pi Imager v1.5 tool, it seems that one can use 64-bit Ubuntu on RPi3, RPi4 and Rpi400.

borkdude commented 3 years ago

Feel free to zip the binary and upload it here for other people to try, as an "unofficial" distribution, like what @sogaiu did.

A tip for faster compilation: you can disable some of the libraries, if you aren't using most of them:

https://github.com/babashka/babashka/blob/master/doc/build.md#feature-flags

The BABASHKA_LEAN flag only keeps the most essential libs.

brdloush commented 3 years ago

Ok, here's the zipped aarch64 binary:

bb-aarch64-2021-01-20_git_rev_14c33625ecb203af2513eb5f527db63e77848ae5.zip

As for LEAN babashka: thanks for the tip. But 20 minutes in not that bad for building the real thing. But I'll try it, just for fun :)

brdloush commented 3 years ago

just FYI:

LEAN native image gets built in 16m12s (compared to 19m10.409s for default bb). It results in 56MB bb binary, default bb is 70M. From the startup performance perspective: they seem to start up in the same time.

harryvederci commented 3 years ago

Ok, here's the zipped aarch64 binary:

Works on my machine!

den1k commented 3 years ago

Incredibly exciting to imagine the possibilities of running babashka + datalevin on raspberry pi

And impeccable timing as @huahaiy is working on the missing piece: https://github.com/juji-io/datalevin/issues/6

borkdude commented 3 years ago

This is great. We can make a datalevin pod, like we already have SQL pods:

and/or add it as a feature flag and then that should work when also compiled on linux/aarch64.

eigenhombre commented 3 years ago

Super excited to see ARM progress for RPi. Question: I have a new Pi running Raspbian, which I think is still 32-bit. Is the new ARM Babashka 64-bit only?

borkdude commented 3 years ago

@eigenhombre Babashka is compiled using GraalVM and there is no 32-bit release of native-image for aarch.

brdloush commented 3 years ago

@eigenhombre you might want to look at this article. It seems that it's possible to use 32-bit raspbian (so all the installed libraries are 32-bit) and switch to 64-bit kernel. But be sure to backup before trying.

Also first make sure that your raspberry's CPU is actually 64-bit. If it's PI3 or PI4, it should be. Not sure about previous PIs.

borkdude commented 3 years ago

This babashka binary should work on RaspberryPi 64bit. It's built on CircleCI now!

https://15839-201467090-gh.circle-artifacts.com/0/release/babashka-0.2.14-SNAPSHOT-linux-aarch64.tar.gz

Anyone wants to try?

brdloush commented 3 years ago

Works on my RPI4 

$ time ./bb -e ':hello'
:hello

real    0m0.045s
user    0m0.008s
sys 0m0.039s
harryvederci commented 3 years ago

Works on my Chromebook as well!

Bedankt/thanks Michiel!

borkdude commented 3 years ago

What does uname -m print for you? aarch64? I can use this to update the install script.

harryvederci commented 3 years ago

Exactly that on the Chromebook: aarch64

borkdude commented 3 years ago

Thanks! I updated the install script so it should recognize your architecture and install babashka correctly on aarch64.

harryvederci commented 3 years ago

Works! Thanks a lot!

hamann commented 3 years ago

I did the kernel upgrade from 32-bit to 64-bit today as mentioned in https://github.com/babashka/babashka/issues/241#issuecomment-764899056, but for me bb isn't working 

~ pi@raspberrypi
❯ grep Model /proc/cpuinfo
Model       : Raspberry Pi 4 Model B Rev 1.2

~ pi@raspberrypi
❯ uname -a
Linux raspberrypi 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux

~ pi@raspberrypi
❯ file /usr/local/bin/bb
/usr/local/bin/bb: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=0dbe834f277b844079eba279ecd50988d91e213e, for GNU/Linux 3.7.0, with debug_info, not stripped

~ pi@raspberrypi
❯ /usr/local/bin/bb
zsh: no such file or directory: /usr/local/bin/bb

I guess a 64-bit userland is also required.