Arm Architecture (Raspberry Pi) - Would it be possible ?

[memento]

Hi everyone !

A lot of people are adopting the Raspberry Pi as a solution for self hosting. I understand (if I get things right) that you guys had to choose an architecture once in for all for compiling the sandstorm application server and make it easy for users to download and install already compiled apps. I understand enabling a new architecture means compiling every version of sandstorm and apps that goes available (just like we do for linux distros and packages). However, taking into account the growing "market share" (if you allow me) of arm platforms, I think it would be a good idea. Of course, I'm willing to gather informations about your process/guidelines when it comes to compilation and release and to do this task, if you accept.

It would be great to have a sandstorm instance running on a Raspberry Pi 3 (for instance). Here are the specs from www.raspberrypi.org

Specifications SoC: Broadcom BCM2837 CPU: 4× ARM Cortex-A53, 1.2GHz GPU: Broadcom VideoCore IV RAM: 1GB LPDDR2 (900 MHz) Networking: 10/100 Ethernet, 2.4GHz 802.11n wireless Bluetooth: Bluetooth 4.1 Classic, Bluetooth Low Energy Storage: microSD GPIO: 40-pin header, populated Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface (CSI), Display Serial Interface (DSI)

[ocdtrekkie]

I am still kinda surprised that this request comes up so much, given that 1 GB RAM spec. I don't know how much the OS and Sandstorm itself would use, but running Sandstorm grains average about 100 MB of RAM each. So, I suppose it'd theoretically work, but resources would be incredibly constrained.

I also think earlier on MongoDB didn't work on ARM, which Sandstorm requires. I heard they got it working, allegedly, since.

Given the other complications you mentioned above, note that Sandstorm is still very early in development, and it probably makes sense to get it stable on one architecture before adding more.

@paulproteus: IMHO, there should be an FAQ question somewhere for "why doesn't Sandstorm currently support ARM?". It gets asked a lot. By definition, a frequently asked question.

[ocdtrekkie]

@kentonv from about a month ago: " Sandstorm will support ARM someday but it's going to require a large investment in tooling in order to be painless for developers."

[zarvox]

@ocdtrekkie did a pretty good job of explaining it already, but to summarize the technical reasons we don't plan to support ARM in the near future:

• MongoDB (which Meteor and Sandstorm rely on) doesn't officially support ARM yet: https://jira.mongodb.org/browse/SERVER-1811
• Since Sandstorm packages contain native code, either app developers would need to build packages for every platform supported, or we'd need to have some way to automate building these packages, lest we wind up with significant app ecosystem fragmentation. Building that tooling would take time and attention from other things which we think will better serve a greater number of users right now.

As a result of these factors, supporting Sandstorm on ARM is not a priority for us right now, though I agree it's a thing we want to work in the longer future.

[paulproteus]

Hi @memento ,

I wrote up some docs in #2432 about the status quo.

Are you still possibly interested in working on this? If so, then I'm happy to discuss details.

Let me know what you think!

[zenhack]

Just some observations:

• 1 GiB of memory in an arm system is actually pretty easy to come by these days.
• Memory usage would probably be a bit lower anyway on a 32-bit platform.

[zenhack]

Err.. I may have misunderstood what was being said about the ram; sorry for the noise :/

[3mbr4c3]

For some environments 1gb of RAM (on RPIs) could be enough. Everyone should agree that there is no such thing as running massive databases doing big data analysis or comparable operations on RPIs, but personally I see some usage in having container orchestration on these small ARM boards; e.g. I run container for all my little nifty home automatization scripts / tools on my RPIs. In fact, it shouldn't be that amount of work to optimize or strip down sandstorm for the arm64 architecture to keep it lightweight.

@memento : let's fork sandstorm to arm64 and just leave the building of compatible arm64 sandstorm apps to the current app developers.

[zenhack]

A fork seems premature; the developers haven't said they're closed to the idea, just that it isn't a priority for them. @paulproteus expressed openness to it.

It would be good to enumerate what actually needs to be done to get sandstorm to work on non x86_64 besides re-compiling. The mongodb issue is apparently solved. Here's what else I can think of, not being terribly familiar with the code:

1. I suspect the seccomp-bpf rules would have to be extended for each arch, since rules match per-arch. I imagine this is a fairly mechanical change though.
2. Per @zarvox's second point, we'll need to figure out how to deal with the possibility of native code for different CPUs. I'm sure this is soluable, but is going to require a bit of design work. I suspect the most challenging part of this is to find a way to make it easy for developers to build apps for different architectures.

Any other things people know of that would have to change?

//cc @kentonv

[zenhack]

Re (2), perhaps worth looking at how various linux distros deal with this; I know debian has a fair amount of machinery, though I'm not intimately familiar with them.

[memento]

@paulproteus @3mbr4c3

Hey guys. Still interrested in that plaform (a lot). I'm currently deep into freelancing right now. As soon as I earn enough to finish the year (the sinews of war) I'ld like to participate on this task (a few months I guess).

Chears,

[kentonv]

FWIW, people often find that 1GB of RAM is not enough for Sandstorm. It's pretty RAM-hungry since for some reason everyone wants to write apps in bulky garbage-collected dynamic languages like Ruby or Java. :)

But yeah, I'm happy to merge patches that make Sandstorm build on ARM if you want to play around with it.

I think it's pretty clear how we'd extend the low-level package format to support multi-arch. Take a look at the Archive type at the bottom of package.capnp -- that's the actual package format. We could add a way for a file to have multiple versions targeting different archs, and then the unpacker would only extract the one for the target arch.

The hard part, though, is providing tools to the developers such that they're able to construct these packages. The current design of the tooling isn't exactly well-designed for this, since it sort of slurps up binaries off the host system. If we designed tools where the user specifies a more precise package list (e.g. from Debian or maybe Nix) then it would be a lot easier to grab the packages for multiple archs. I think work on such tooling could be a pretty interesting project that someone could easily own and work on without requiring lots of coordination with other parts of the project, if someone is interested!

[3mbr4c3]

@memento: just get in contact, I'll open my SandstormArm64 repo this weekend.

I currently ported just 20% of core sandstorm to arm64, it will take some more time before I / we can take a look into the toolchain. As stated before, I will open my repo which I just forgot right now, but I think refactorization of the toolchain is no dependency so I welcome everyone to commit in this part (propably as dedicated project), too!

[Michael-S]

This is probably obvious to everyone in the discussion, but I suspect the Raspberry Pi 4 or 5 will have 2GB of RAM or more. And there are other single board computers only marginally more expensive than the Pi that have 2GB already. Any guesses as to whether that's sufficient for a Sandstorm installation? Or even then would users need to restrict the amount of concurrent applications they use?

[zenhack]

I've got sandstorm running on an x86_64 box with 2GiB of ram, so that's good enough for my own use. I'm pretty much the only user though.

[kentonv]

Yes, people have reported success with 2GB. Note that the amount of RAM you need is proportional to the number of active users -- or, really, the number of active grains. (We have plans to improve the situation someday using https://github.com/sandstorm-io/snappy-start but not sure when that will happen.)

[zenhack]

Quoting Kenton Varda (2017-01-27 02:35:06)

Take a look at the Archive type at the bottom of package.capnp -- that's the actual package format.

Curious, what's the difference between that and the actual .spk file? capnp decode tells me it's not a capnp message, and It's apparent from the file sizes I've seen that there's some compression going on as well.

I think work on such tooling could be a pretty interesting project that someone could easily own and work on without requiring lots of coordination with other parts of the project, if someone is interested!

Might be up for it if I can find some time, but I can't really promise much at the moment. I have a separate interest in reproducible builds as well, so all of the magic in the current tooling is something that is very much the opposite of what I want :/.

[kentonv]

Curious, what's the difference between that and the actual .spk file?

See the comments on stuff defined right above "Archive" -- the full file is actually two capnp messages (Signature and Archive), xz-compressed, with an 8-byte "magic number" prefix.

[necrose99]

ekam_bin_arm64.zip

i've meddled with ekam and building on rpi3 arm64 , tends to eat disk.... however looking to get a rockpro64 for home soon , could try to build on cloud.scaleaway. com .. as i have 8 gigs... of ram etc...

[abliss]

I have a khadas vim2 with 3gb of ram, and (if I can find the time) I'm going to try to get sandstorm running on it.

[xet7]

BTW, Wekan is made with Meteor, and Wekan already runs on RasPi3 and RasPi4 on arm64 and any other Node.js/MongoDB supported CPU architecture:

• RasPi3 info
• Bundle info
• Download

[ocdtrekkie]

I think the largest problem is that even though a lot of dependencies for Sandstorm probably support ARM now, that we have a hard enough time getting apps packaged and updated for Sandstorm right now. Adding a CPU architecture would require all the apps get packaged for it as well.

[xet7]

@ocdtrekkie

Yes, Sandstorm build pipeline work for current x64 architecture should be continued.

[xet7]

And only after that think of other architectures.

[zenhack]

I think it's worth putting some thought into what implications multiarch has for the build pipeline, rather than get something working for x86_64 and hope it still makes sense elsewhere. I haven't investigated what multiarch looks like in the docker world, I should find some time to read up on where things are there and what that means for docker-spk's prospects of being able to handle this.

[xet7]

@zenhack

Yes, I really would like to figure out multiarch Wekan Docker and Snap. I have tried but I will sometime try to research more, I don't have it working yet.

I don't know would is be possible to run Sandstorm in Docker or Snap.

[necrose99]

ekam will build on arm64 , which is needed for some of the building.. some time ago for captnproto however for rpi3 or less , you may need to make -j2 1 etc lowering memory foot print. {however (https://www.scaleway.com/en/virtual-instances/arm-instances/) debian native , 8-16 gigs of ram should be easy to build and exportbuilt artifacts ... to RPI etc... }

https://github.com/capnproto/ekam "Ekam's primary user and maintainer right now (Sandstorm.io) is itself highly Linux-specific, so there is not much pressure. (Let us know if you want to help.)"

[zenhack]

I was thinking about this again recently. For dev tooling, we could likely make use of the kernel's binfmt_misc with qemu user mode to create a vagrant vm that can be used to develop arm packages on an x86_64 host. This way devs could use their own machines (even if they don't have access to an arm box) and much of the vagrant-spk workflow would be the same. This still leaves devs needing to test each arch manually though.

[xet7]

Does something need building on arm64? I have access to bare metal arm64 sever that has 125 GB RAM and CPU that has 96 cores.

[ocdtrekkie]

@zenhack I think the biggest issue is with already having difficulty getting people to update their packages, asking them to make two packages and update both is pretty painful. My trauma from Windows on ARM-related travesties comes to mind.

I'd want our vagrant-spk scripts to have well-documented methods for splitting out "on arm do this, on x64 do this", so one doesn't have to maintain fully separate scripts for each architecture, despite the fact that a lot of the build instructions are probably architecture-specific. Presumably we'd want vagrant-spk devs to pass an arm argument somewhere in their vagrant-spk commands to trigger testing and packing the ARM version of a given package.

[xet7]

By default, meteor only supports x64 (and maybe x32). For Wekan, I first create x64 bundle .zip file, and then make changes to it with maintainer bash script: https://releases.wekan.team/raspi3/

For arm64, MongoDB 3.x is installed from Ubuntu repos.

For IBM Z Series mainframes, I only add fibers, and use it with Node and MongoDB for s390x: https://github.com/wekan/wekan/wiki/s390x#install

[xet7]

That arm64 server is running Ubuntu 18.04. There I created lxc container that has Ubuntu 20.04 and I'm currently trying to build Sandstorm in that container.

[xet7]

==== compiling bootstrap ekam ====
g++ -Isrc -std=c++14 -pthread tmp/base/Promise.o tmp/base/sha256.o tmp/base/Debug.o tmp/base/OwnedPtr.o tmp/base/Hash.o tmp/os/ByteStream.o tmp/os/EventManager.o tmp/os/DiskFile.o tmp/os/Socket.o tmp/os/EpollEventManager.o tmp/os/Subprocess.o tmp/os/OsHandle.o tmp/os/File.o tmp/os/EventGroup.o tmp/ekam/initNetworkDashboardStub.o tmp/ekam/CppActionFactory.o tmp/ekam/SimpleDashboard.o tmp/ekam/Tag.o tmp/ekam/ekam.o tmp/ekam/ExecPluginActionFactory.o tmp/ekam/MuxDashboard.o tmp/ekam/Driver.o tmp/ekam/ConsoleDashboard.o tmp/ekam/ActionUtil.o tmp/ekam/Action.o tmp/ekam/Dashboard.o -o bin/ekam-bootstrap
make[1]: Leaving directory '/home/wekan/repos/sandstorm/deps/ekam'
==== downloading Clang binaries from Chromium project ====
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-n346557-4e0d9925-2.tgz .......... Done.
==== configuring BoringSSL ====
cd deps/boringssl/build && cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="/home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang" -DCMAKE_CXX_COMPILER="/home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang++" -DCMAKE_C_FLAGS="-fPIE" -DCMAKE_CXX_FLAGS="-fPIE" ..
-- The C compiler identification is unknown
-- Check for working C compiler: /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang
-- Check for working C compiler: /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang -- broken
CMake Error at /usr/share/cmake-3.16/Modules/CMakeTestCCompiler.cmake:60 (message):
  The C compiler

    "/home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/make cmTC_89ea3/fast && make[1]: Entering directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'
    /usr/bin/make -f CMakeFiles/cmTC_89ea3.dir/build.make CMakeFiles/cmTC_89ea3.dir/build
    make[2]: Entering directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_89ea3.dir/testCCompiler.c.o
    /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang   -fPIE    -o CMakeFiles/cmTC_89ea3.dir/testCCompiler.c.o   -c /home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp/testCCompiler.c
    make[2]: /home/wekan/repos/sandstorm/deps/llvm-build/Release+Asserts/bin/clang: Command not found
    make[2]: *** [CMakeFiles/cmTC_89ea3.dir/build.make:66: CMakeFiles/cmTC_89ea3.dir/testCCompiler.c.o] Error 127
    make[2]: Leaving directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'
    make[1]: *** [Makefile:121: cmTC_89ea3/fast] Error 2
    make[1]: Leaving directory '/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeTmp'

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:14 (enable_language)

-- Configuring incomplete, errors occurred!
See also "/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeOutput.log".
See also "/home/wekan/repos/sandstorm/deps/boringssl/build/CMakeFiles/CMakeError.log".
make: *** [Makefile:236: deps/boringssl/build/Makefile] Error 1
wekan@sand:~/repos/sandstorm$ uname -a
Linux sand 4.15.0-36-generic #39-Ubuntu SMP Wed Sep 26 06:05:44 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
wekan@sand:~/repos/sandstorm$ cat /etc/issue
Ubuntu 20.04 LTS \n \l

[abliss]

Looks like it's downloading the x86 clang binary. Is there a corresponding arm binary to download, and can the make system autodetect which one to download?

[zenhack]

We should be able to just use the system clang. I think that got added at some point because we were using some C++ features that were very new at the time, and a lot of folks couldn't build sandstorm with their system compilers.

We could probably rip that out and use the system compiler unconditionally now without much trouble.

Quoting Adam Bliss (2020-04-27 17:32:51)

Looks like it's downloading the x86 clang binary. Is there a corresponding arm binary to download, and can the make system autodetect which one to download?

[yajo]

I haven't investigated what multiarch looks like in the docker world, I should find some time to read up on where things are there and what that means for docker-spk's prospects of being able to handle this.

Docker is making this quite simple these days: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

[kentonv]

FWIW the Clang binary that we download from the Chromium project is a multi-target compiler. It can cross-compile to arm. Ekam is capable of cross-compiling and at Cloudflare we regularly use it to cross-compile to arm64, at least.

But if you want to run the build on arm (not cross-compile)... that's a different story, yeah.

[zenhack]

@xet7, I think it should work to just override CC and CXX with the systems' clang and clang++ respectively (assuming they're new enough, which they probably are). I raised the question of un-bundling clang in #3331, but we decided against it. If building on Arm becomes common we can revisit how the build system should work.

[xet7]

@kentonv @zenhack

So how can I either: a) cross compile Sandstorm to ARM ? b) compile Sandstorm on ARM ?

For me ARM has been common for a long time. I have:

• RasPi4 with 4 GB RAM running Wekan on Ubuntu arm64 and Raspbian armhf
• RasPi3 with 1 GB RAM
• RasPi1
• bare metal arm64 server that has 128 GB RAM

I would also like to cross compile to s390x, etc

[kentonv]

Getting Sandstorm working on arm will almost certainly require some code changes, e.g. in the seccomp sandbox. It's not going to be trivial. I don't know what will be involved, though, since I haven't tried it...

[xet7]

@kentonv

What kind of changes you image there could be in the seccomp sandbox? How can I try it?

[kentonv]

seccomp filters are architecture-specific, because many syscalls differ across architectures. So it will probably need to be tweaked for the architecture.

[xet7]

@kentonv

Where in Sandstorm source code are those architecture-specific syscalls? What syscalls are used?

[kentonv]

No, that's not what I mean. Many common syscalls, like clone(), have different signatures on different architectures. This is just normal. glibc typically wraps the details. But seccomp filters have to operate on the raw syscalls. Sandstorm employs a seccomp filter as part of its sandbox. This filter will likely need to be tweaked to work correctly on other architectures.

The seccomp code is here:

https://github.com/sandstorm-io/sandstorm/blob/69336bf38433a8bc72ea3aefd7d2510e6e616f54/src/sandstorm/supervisor.c++#L1085-L1240

[kentonv]

I hate to be discouraging here, but I don't think an arm port is likely to pan out without a lot of work, and I don't think I have time to provide the assistance that would be needed from me to get through it. So I'd recommend leaving this on the shelf for now.

[xet7]

@kentonv

Is there some existing lightweight sandboxing library or software that already works on multiple CPUs? That could be used instead, for example sandboxing Wekan and other apps? Or is Docker only option?

Or alternatively, is sandboxing easier in some other programming language?

[xet7]

Well, having Wekan run currently on arm64 and s390x is already complicated, because when adding some new dependencies those in some cases do not work on other than x64 without modifications. Currently I'm figuring out can ostrio-files run on those architectures, and reverting changes while figuring it out is in progress.

[kentonv]

It shouldn't be too hard to tweak the seccomp filter for any particular CPU if you know what you're doing -- maybe half an hour of work. Trying to integrate some other sandbox would be much, much harder.

This is just meant to be one example of a reason why porting Sandstorm to arm is not trivial. There will be many, many other issues like this.

[zenhack]

Quoting Lauri Ojansivu (2020-05-25 06:38:40)

b) compile Sandstorm on ARM ?

Doing something like:

CC=clang CXX=clang++ make

Should get it to use the system's clang instead of the downloaded one, which will at least get you past the failures from before.

As @kentonv mentions, Sandstorm will probably need code changes to work correctly, but it will be interesting to see how far we can get.

Note that even if it appears to work correctly it may not be blocking everything it should, so is not necessarily safe. We'll have to audit and test.

-Ian

[xet7]

@zenhack

Thanks! I will test it a little.

[nniro]

I attempted the compilation process following @zenhack 's suggestion and this is the compilation log :

compile.log

This was attempted on a pinebook pro with an aarch64 distribution.

Edit : I thought uploading a file would show it's content, here's the important part :

✔ test: capnp/serialize-async-test
✔ test: capnp/fuzz-test
✘ bpf_asm: sandstorm/seccomp-bpf/filter.s
    error: syntax error at line 810
✘ compile: node-capnp/capnp.cc
    /ekam-provider/canonical/node-capnp/capnp.cc:29:10: fatal error: 'node.h' file not found
    #include <node.h>
             ^~~~~~~~
    1 error generated.
✘ jstest: node-capnp/capnp-test.js
    full log: tmp/node-capnp/capnp-test.js.log
✘ install: sandstorm/sandstorm.ekam-manifest
    sandstorm/run-bundle: not found
✘ compile: sandstorm/supervisor.c++
    /ekam-provider/canonical/sandstorm/supervisor.c++:91:10: fatal error: 'sandstorm/seccomp-bpf/filter.h' file not found
    #include <sandstorm/seccomp-bpf/filter.h>
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
✘ install: node-capnp/node-capnp.ekam-manifest
    node-capnp/capnp.node: not found
✘ link: sandstorm/run-bundle.o
    /usr/bin/ld: tmp/kj/async-io-unix.o: in function `kj::(anonymous namespace)::SocketAddress::lookupHost(kj::LowLevelAsyncIoProvider&, kj::String, kj::String, unsigned int, kj::_::
      NetworkFilter&)::$_3::operator()(kj::(anonymous namespace)::SocketAddress::LookupParams&&) const':
    /ekam-provider/canonical/kj/async-io-unix.c++:1007: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for
      linking
    /usr/bin/ld: tmp/sandstorm/run-bundle.o: in function `kj::Own<sandstorm::SupervisorMain> kj::heap<sandstorm::SupervisorMain, kj::ProcessContext&>(kj::ProcessContext&)':
    /ekam-provider/c++header/kj/memory.h:450: undefined reference to `sandstorm::SupervisorMain::SupervisorMain(kj::ProcessContext&)'
    /usr/bin/ld: tmp/sandstorm/run-bundle.o: in function `~SupervisorMain':
    /ekam-provider/canonical/sandstorm/supervisor.h:30: undefined reference to `vtable for sandstorm::SupervisorMain'
    /usr/bin/ld: /ekam-provider/canonical/sandstorm/supervisor.h:30: undefined reference to `vtable for sandstorm::SupervisorMain'
    /usr/bin/ld: tmp/kj/async.o(.debug_info+0x1569c): R_AARCH64_ABS64 used with TLS symbol _ZN2kj12_GLOBAL__N_120threadLocalEventLoopE
    /usr/bin/ld: tmp/kj/exception.o(.debug_info+0xaba1): R_AARCH64_ABS64 used with TLS symbol _ZN2kj12_GLOBAL__N_116currentExceptionE
    /usr/bin/ld: tmp/kj/exception.o(.debug_info+0xabbc): R_AARCH64_ABS64 used with TLS symbol _ZN2kj12_GLOBAL__N_119threadLocalCallbackE
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:268: tmp/.ekam-run] Error 1