Pre-compiled Terra distribution for aarch64 (Raspberry Pi) Linux

[renatoathaydes]

Would it be possible to add a pre-compiled binary for Raspberry Pi to the release distributions?

I think Terra can be a really good language to use in a contrained environment like the Pi and would love to try it out, but it seems that currently the only option is to compile Terra itself on the Pi (Which the docs seems to discourage due to the big LLVM dependency?).

[elliottslaughter]

I confess, I don't know much about Raspberry Pi. What kind of software environment does it run? Is it Linux? Is it anything like a full distro or is this bare-metal binaries only? Is there a package manager, or at least an SDK?

Depending on how much this is like a "full" Linux system, I could see a couple possible routes forward:

1. If there's a package manager and it has LLVM, you could download that and build Terra against it. If you have a C/C++ compiler on the device, this is probably the easiest path.
2. Otherwise if there's an emulator, the next easiest option might be to just use that. That is, assuming the emulator works well. I've had bad experiences with qemu in the past, but that was a long time ago, so maybe the situation is better now. However, this will still require a C/C++ compiler stack on the device if you want to do anything nontrivial.
3. We could cross-compile Terra, but the build system hasn't really been used for this, and so I'm guessing we'd find some things that wouldn't work. E.g., we'd probably need to build two copies of LuaJIT: one for the host and one for the device.
4. The last option to consider is not cross-compiling Terra at all. Instead, Terra itself can be used as a cross-compiler. There are a few things that might break this way, but for basic programs it may actually be the easiest thing to do out of all these options. We already cross-compile on a regular basis for NVIDIA/AMD GPUs, so cross-compiling for a CPU architecture doesn't seem like that much of a stretch.

That's maybe a bit more than you were looking for. :-) Overall, I'm happy do what I can on my end, but it'll probably take some effort on your part to look into these options and figure out the path forward.

[renatoathaydes]

It can run any OS that supports ARM 32-bit or 64-bit (I have both, but use 64-bit mostly). In my case, I use a full Ubuntu 22 installation with the usual apt package manager, but I had used before Arch Linux (pacman) and the PI's default OS which is based on Debian (so I think it also uses apt).

Building Terra wouldn't be a problem if it only requires C++ and LLVM... I was just worried about the warning in the docs against building it... and it would obviously be more convenient to download a binary or just do apt install terra.

[elliottslaughter]

LLVM is frequently the victim of packaging bugs. But if you pick your versions and distros carefully, you can make it work.

Here's a set of instructions that work on Ubuntu 22.04 on x86_64. Hopefully they might also work on ARM?

apt-get install build-essential cmake git llvm-11-dev libclang-11-dev clang-11 libedit-dev libncurses5-dev zlib1g-dev libpfm4-dev
git clone https://github.com/terralang/terra.git
cd terra/build
cmake -DCMAKE_INSTALL_PREFIX=$PWD/../install ..
make install -j4
ctest --output-on-failure -j4

(I am picking LLVM 11 here because I know that LLVM 12 has packaging issues on Ubuntu 22.04. Like I said, they frequently seem to mess this up.)

[elliottslaughter]

I discovered today that Docker has built-in support for multiarch container builds that seems to work (nearly) out of the box. I'm honestly shocked at how easy this was to get started.

Anyway, I've got a build going in my branch. Once the build completes, you should get access to binaries for ARM64 on Ubuntu 22.04: https://github.com/elliottslaughter/terra/actions/runs/2478120437 (edit: direct link: https://github.com/elliottslaughter/terra/suites/6887565506/artifacts/266822797)

Having said that, the test suite is only passing about 75% of tests right now. I'm guessing there will be handful of stupid mistakes, followed by a long tail of stuff that will be harder to address. Depending on what you want to do, this may or may not meet your needs. Anyway, it's available now, and if I get this working well enough I'm happy to merge to master.

Note that, since I'm using Ubuntu binaries for LLVM, you will need the following dependencies installed to use this binary: build-essential libedit-dev libncurses5-dev zlib1g-dev libpfm4-dev. You do not need LLVM/Clang because Terra statically links those dependencies.

[renatoathaydes]

That's really great. Thanks for that. I will play around with this when I get some time!

[renatoathaydes]

Are the dependencies you listed above required because of the Lua interpreter included in Terra or something else?

[elliottslaughter]

These are LLVM dependencies.

Ubuntu packagers seem to follow a policy of maximalism when it comes to LLVM dependencies. They seem to just turn everything on that they can. As a result, Terra built against Ubuntu's LLVM binaries inherits those dependencies.

For x86, we go to a lot of trouble to build our own LLVM, specifically in order to minimize those dependencies. I literally turn off everything that can be turned off. Not only does this mean fewer dependencies for the end user, it's actually essential to making Terra binaries that are portable across distros/releases, since those dependencies are often incompatible across distros/releases (in contrast to, say, libc, which is usually backwards compatible). I maintain the binary builds here: https://github.com/terralang/llvm-build/

Doing the same for ARM will probably require abandoning the Github Actions infrastructure (at least for the ARM builds). On x86, the LLVM builds already take 2-3 hours, and with the factor 3-4x slowdown I'm seeing with emulation... it just won't be feasible to do so on Github. I guess that's survivable, but it's just a bit unfortunate since I've worked hard to make sure all our other binary builds live on Github Actions.

[elliottslaughter]

By the way, under our own custom LLVM build, the dependencies would just drop down to a working C compiler (capable of linking, at least) and headers for libc.

[elliottslaughter]

Ok, I bit the bullet and built a custom LLVM (about 6 hours on a 20 core machine... ouch). But the upside is a new Terra build on ARM: https://github.com/elliottslaughter/terra/suites/6895033898/artifacts/267435048

This is a dependency-minimized build, so you should only need a working C compiler and standard headers/libraries (build-essential should more than cover it).

If you can, it would be nice to confirm that this gets the expected pass rate (around 72% currently, I believe). So far I've done all the builds and tests on qemu, and I wonder how good a job the emulator actually does vs. actual hardware.

[renatoathaydes]

I unpacked and tried this on my Raspberry Pi, but it's failing to run... only the help page shows up:

renato@renato-desktop:~/terra-Linux-arm64-54ac68a$ bin/terra hi.t 
bad light userdata pointer
renato@renato-desktop:~/terra-Linux-arm64-54ac68a$ bin/terra -h

Terra -- A low-level counterpart to Lua
Release 1.0.0-15-g54ac68a

Homepage: https://terralang.org/
Project: https://github.com/terralang/terra
Community: https://terralang.zulipchat.com/

terra [OPTIONS] [source-file] [arguments-to-source-file]
    -v enable verbose debugging output
    -g enable debugging symbols
    -h print this help message
    -i enter the REPL after processing source files
    -e 'chunk' : execute command-line 'chunk' of code
    -  Execute stdin instead of script and stop parsing options
renato@renato-desktop:~/terra-Linux-arm64-54ac68a$ bin/terra
bad light userdata pointer

To run the tests, I need to clone this repo and do this as shown in the README?

cd tests
<terra-installation>/bin/terra run

[renatoathaydes]

Seems to be a LuaJIT issue: https://github.com/LuaJIT/LuaJIT/pull/230

[elliottslaughter]

Ok, here's a new build with LuaJIT updated to the latest commit. Hopefully this fixes that particular issue?

https://github.com/elliottslaughter/terra/suites/6899663236/artifacts/267748577

[renatoathaydes]

Everything seems to work now! Just a few tests failed:

=================
= FAILING tests
=================
vars2.t
zeroreturn2.t
coverage3.t
vars.t
gettype.t
simpleglobal.t
ssimple.t
let2.t
luabridge.t
defercond.t
class6.t
atomicrmw.t
teststd.t
constanttypes.t
luabridge2.t
vecarith.t
zeroreturn.t
vecobj.t
canon2.t
expvec.t
class.t
cconv.t
=================

525 tests passed. 22 tests failed.

[elliottslaughter]

Awesome! Glad it's working now.

That's actually a higher pass rate than in emulation, which is interesting.

=================
= FAILING tests
=================
vars.t
zeroreturn2.t
bf.t
class.t
globals.t
coverage3.t
ssimple.t
atomicrmw.t
vars2.t
pthreads.t
gettype.t
luabridge2.t
defercond.t
hasbeenfrozen.t
rvaluerecv.t
expvec.t
teststd.t
simpleglobal.t
vecarith.t
let2.t
constanttypes.t
class6.t
cconv.t
canon2.t
zeroreturn.t
luabridge.t
vecobj.t
benchmark_fannkuchredux.t
=================

519 tests passed. 28 tests failed.

I'll spend some time cleaning this up so that it can get merged, and then we can have official ARM binaries.

[elliottslaughter]

This was merged in #568 and I plan to do a release soonish so that we get official/stable binaries.

[elliottslaughter]

Official binaries are now up in release 1.0.1 and will be included going forward.

I think at this point we can move discussion of any specific bugs/test failures to individual issues, so I'll close this one.

[renatoathaydes]

That is really great. Thanks for your effort.

[elliottslaughter]

If you update to 1.0.5, AArch64 should now be fully supported and pass 100% of the test suite.