Installing Prysm on ARM Hardware - fails for armv6l (Raspberry Pi Zero W)

chrishobcroft commented 4 years ago

Description of the bug

I tried to install and run Prysm on a Raspberry Pi Zero W, armv6l chipset, running Raspbian 10.

It failed with the error:

Starting Prysm beacon-chain --clear-db
/home/pi/prysm/dist/beacon-chain-v0.3.9-linux-armv6l: line 1: Not: command not found

Steps to reproduce

Environment: armv6l (Raspberry Pi Zero W), running Raspbian 10 (full spec below)
Follow these instructions: Installing Prysm on ARM Hardware · Prysm 'Sapphire' Testnet.pdf on a Raspberry Pi Zero

When you get to "Run the prysm.sh script", it fails with the following error:

pi@raspberrypi:~/prysm $ ./prysm.sh beacon-chain --clear-db
Latest Prysm version is v0.3.9.
Downloading beacon chain@v0.3.9 to /home/pi/prysm/dist/beacon-chain-v0.3.9-linux-armv6l (automatically selected latest available version)
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100     9  100     9    0     0      9      0  0:00:01 --:--:--  0:00:01     9
Downloading validator@v0.3.9 to /home/pi/prysm/dist/validator-v0.3.9-linux-armv6l (automatically selected latest available version)
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100     9  100     9    0     0      8      0  0:00:01  0:00:01 --:--:--     8
Starting Prysm beacon-chain --clear-db
/home/pi/prysm/dist/beacon-chain-v0.3.9-linux-armv6l: line 1: Not: command not found

Further to some conversation with @nisdas and @mratsim, and some further investigation, it would appear that the prysm.sh script downloads two files:

/dist/beacon-chain-v0.3.9-linux-armv6l
/dist/validator-v0.3.9-linux-armv6l

and then attempts to run beacon-chain-v0.3.9-linux-armv6l.

It fails, because the files in /dist/* both contain Not Found, presumably as a result of a failed download from a dead URL.

What I expect to happen

First, I expected that there would be a script for armv6l, and that the process would run Prysm on a Raspberry Pi Zero W (I was so excited, and then disappointed!).

When it failed, now I see that what I expect to happen is that there is an error showing that armv6l is not supported, per @nisdas, with a message also for @prestonvanloon :)

Hardware Specs

Raspberry Pi Zero W (~$10):

pi@raspberrypi:~/prysm/dist $ lscpu
Architecture:        armv6l
Byte Order:          Little Endian
CPU(s):              1
On-line CPU(s) list: 0
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           1
Vendor ID:           ARM
Model:               7
Model name:          ARM1176
Stepping:            r0p7
CPU max MHz:         1000.0000
CPU min MHz:         700.0000
BogoMIPS:            997.08
Flags:               half thumb fastmult vfp edsp java tls

pi@raspberrypi:~/prysm/dist $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

0xKiwi commented 4 years ago

Hello @chrishobcroft ! Thank you for trying Prysm on your Pi Zero W haha

Looks like we don't support this currently, sorry about that! @prestonvanloon is this something we can support in the future?

mfiumara commented 4 years ago

It looks like you'd have to add ARM32 support in order to run the client on any raspberry pi (raspberry pi model 3 is armv7, which is also a 32 bit architecture). I'm not sure how hard that is regarding this project, maybe as easy as adding the ("linux", "arm32") pair to tools/binary_targets.bzl?

nisdas commented 4 years ago

Hey @mfiumara @chrishobcroft , there are certain blockers to us supporting 32-bit architectures. We currently use bolt-db as our primary key value store. Due to it being memory-mapped, can only support database sizes of up to 2gb in a 32-bit address space. For the requirements of the beacon-chain, that will never be enough.

We will need to support larger database sizes as a minimum in order to have a functional beacon node, for more clarity on this you can take a look at: https://github.com/boltdb/bolt/issues/280

A way to unblock us would be to support an alternative database, which supports larger database sizes on 32-bit architectures. The implementation would have to conform to this interface: https://github.com/prysmaticlabs/prysm/blob/master/beacon-chain/db/iface/interface.go#L130

This would be a non-trivial task to do as you can see, as a lot of existing functionality of the beacon database would have to be ported over to the new implementation.

Our BLS library previously was blocked on this too as we only supported 64 bit architectures, however with our recent improvements to it and with some additional build tooling, if I am not wrong we should be able to support arm32. @prestonvanloon can probably chime in as he has been mostly working on it.

mratsim commented 4 years ago

Nimbus can run in 32-bit mode and but even on our side the Raspberry Pi Zero is probably stretching it ;), well it can probably run but may not support more than 1 peer due to cryptography + epoch processing + fork choice bottlenecks. Maybe Sigma-Prime @paulhauner can chime in as well since they spend the last 2 months on optimization but I'm not hopeful.

SuburbanDad commented 4 years ago

FWIW, the beefiest pi, 4b+ with 4gb is the least I have had any success with for beacon on the current testnet. Things may change on the new test net, but I would be very suprised if beacon ran reliably on anything less.

Validators can run on a 2b/3b, though not with raspbian since it is 32 bit, and as @nisdas mentioned, 64bit is necessary for prysm(not just processor but the OS also). It'd be a neat trick to see a pi zero/w work as a validator tho. Might give pi zero a whirl with nimbus validator on the new testnet .

paulhauner commented 4 years ago

Lighthouse (LH) doesn't support 32-bit, either. Even if we did, I think a Pi Zero is really stretching it for a LH beacon node, especially one that has any importance. Running a LH validator client on it is probably quite achievable, however I certainly wouldn't stake my own ETH on it.

Getting a Pi Zero beacon node to sync would be one thing but keeping it up during network instability or an attack would be a whole other thing.

prestonvanloon commented 4 years ago

@mfiumara Adding 32 bit ARM cross compile support involves adding another bazel toolchain. I have removed that legacy bit of starlark logic that you are referencing (https://github.com/prysmaticlabs/prysm/pull/5329).

@SuburbanDad If you are interested here, could you add a toolchain and config for arm 32 bit cross compile?

In any case, we will not support running a beacon node on 32 bit architectures due to the reasons @nisdas and others have outlined above. However, it would be interesting to see the performance of a validator on rpi zero in a testnet with a remote beacon chain instance on another machine (cloud or local network).

criticaltv commented 4 years ago

Hey @SuburbanDad I'm ready to test on armv6l architecture...

Running a Validator with a remote Beacon Chain could be a very cool thing to do, for people wanting to dedicate some infrastructure to run their own staking service, and their own 32 ETH

rauljordan commented 4 years ago

Hi. @criticaltv and @chrishobcroft would you consider this issue resolved? We do not aim to offer 32 bit architecture support at the moment, unfortunately.

alex-miller-0 commented 4 years ago

@prestonvanloon We would love to look into running a validator (not beacon) on the @gridplus Lattice hardware wallet. We have a 32-bit MIPS architecture, though. If you were to add that toolchain we would be grateful!

prestonvanloon commented 4 years ago

@alex-miller-0 Thanks for the feedback!

Would an armv6l binary work for your hardware our is there something special about compiling for mipsel? (Is it little endian or endian?)

alex-miller-0 commented 4 years ago

@prestonvanloon It is a 32bit MIPS architecture that I believe can be compiled with either BE or LE. I'm not sure if armv6l would work, but I suspect not, since it's not an ARM architecture. I suppose we could try it haha.

Here's some info on MIPS vs ARM; looks like MIPS has more registers and there are a few differences in instruction sets (which are in the same family, so mostly pretty similar).

Tagging @maevyn11 who is our resident Omega2+ expert.

prestonvanloon commented 4 years ago

I think the biggest hurdle might be the BLS encryption library. Most of these have hardware accelerated assembly instructions.

Current options are:

Pure go implementation -- slow, may not be up to date with latest draft spec, but cross compile is a non-issue
Herumi's library -- More supported architectures than blst, including armv6l
Supranational blst -- Faster than Herumi, only supports x86_64 and ARM64 at the moment

None of the above have been audited yet and only the latter two options are serious candidates for Prysm. The pure go implementation might work since validator workloads are generally very lightweight.

alex-miller-0 commented 4 years ago

Well, pure go would probably work fine. We already cross compile our daemon using GOARCH=mipsle GOOS=linux go build. Depending on what operations are being done, the latency may or may not be a problem. Keep in mind that our core is quite slow, so if there are a ton of operations it may be problematic.

Where is the BLS encryption used in the validator code? Is is just part of signing, or does the block producing stuff need BLS too? (or is there other functionality that needs it?)

We would be implementing signing in embedded C within our secure environment (I mentioned earlier how we would need to add our own signing module anyway), so if BLS is only needed for signing, we could potentially fork that requirement out of the go build. I'm not too familiar with how this all works, though.

Edit: Just doing a quick scan of your validator codebase, it looks like bls is only used in the client package when referencing the keystore and asking for signatures via keyManagerV2. Again, I am a bit of a noob wrt eth2 stuff, but I believe all we are looking for is the client package itself (aggregating, proposing, attesting) and a hook into an external keystore (using a module that we would write). So perhaps the best strategy here is just forking out the keystore stuff and adding our own module.

Edit2: Ah, there is also sig validation, obviously. So yeah, I guess we would still need the golang bls module. I'm assuming the performance constraint would be validating all the tx sigs.

prysmaticlabs / prysm