search

spesmilo / electrum

Electrum Bitcoin Wallet
https://electrum.org
MIT License
7.44k stars 3.09k forks source link

AppImage for ARM Linux #5159

Open SomberNight opened 5 years ago

SomberNight commented 5 years ago

At the moment we only build AppImages for x86_64 Linux. See https://github.com/spesmilo/electrum/tree/master/contrib/build-linux/appimage

It would be nice to be able to do the same for ARM (maybe armhf?).

Ideally, the build script would NOT assume the host to be ARM, so e.g. it would cross-compile or similar.

bauerj commented 5 years ago

There are Docker images available that can run ARM-based operating systems on an amd64 CPU. See e.g. https://www.balena.io/docs/reference/base-images/base-images/#building-arm-containers-on-x86-machines

SomberNight commented 5 years ago

Nice! I've been looking at https://ownyourbits.com/2018/06/27/running-and-building-arm-docker-containers-in-x86/

SomberNight commented 5 years ago

I did a bit of research on this. To avoid redoing it next time I look at this, some random useful links:

robster7674 commented 2 years ago

I gave this a try on my PinePhone using Manjaro/PhoSh beta 23. Result was an error from Docker:

💬 INFO:  building docker image.
Sending build context to Docker daemon  24.06kB
Step 1/13 : FROM ubuntu:18.04@sha256:9bc830af2bef73276515a29aa896eedfa7bdf4bdbc5c1063b4c457a4bbb8cd79
docker.io/library/ubuntu:18.04@sha256:9bc830af2bef73276515a29aa896eedfa7bdf4bdbc5c1063b4c457a4bbb8cd79: Pulling from library/ubuntu
4028d4a2ab03: Pull complete 
Digest: sha256:9bc830af2bef73276515a29aa896eedfa7bdf4bdbc5c1063b4c457a4bbb8cd79
Status: Downloaded newer image for ubuntu:18.04@sha256:9bc830af2bef73276515a29aa896eedfa7bdf4bdbc5c1063b4c457a4bbb8cd79
 ---> d7efed8348f7
Step 2/13 : ENV LC_ALL=C.UTF-8 LANG=C.UTF-8
 ---> Running in bb2dd5bfa82e
Removing intermediate container bb2dd5bfa82e
 ---> 7623cde525b0
Step 3/13 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Running in 6308eee79232
Removing intermediate container 6308eee79232
 ---> 3fa6fe2d054f
Step 4/13 : RUN apt-get update -q &&     apt-get install -qy         git=1:2.17.1-1ubuntu0.9         wget=1.19.4-1ubuntu2.2         make=4.1-9.1ubuntu1         autotools-dev=20180224.1         autoconf=2.69-11         libtool=2.4.6-2         autopoint=0.19.8.1-6ubuntu0.3         xz-utils=5.2.2-1.3         libssl-dev=1.1.1-1ubuntu2.1~18.04.15         libssl1.1=1.1.1-1ubuntu2.1~18.04.15         openssl=1.1.1-1ubuntu2.1~18.04.15         zlib1g-dev=1:1.2.11.dfsg-0ubuntu2         libffi-dev=3.2.1-8         libncurses5-dev=6.1-1ubuntu1.18.04         libncurses5=6.1-1ubuntu1.18.04         libtinfo-dev=6.1-1ubuntu1.18.04         libtinfo5=6.1-1ubuntu1.18.04         libsqlite3-dev=3.22.0-1ubuntu0.4         libusb-1.0-0-dev=2:1.0.21-2         libudev-dev=237-3ubuntu10.53         libudev1=237-3ubuntu10.53         gettext=0.19.8.1-6ubuntu0.3         libzbar0=0.10+doc-10.1build2          libdbus-1-3=1.12.2-1ubuntu1.2         libxkbcommon0=0.8.0-1ubuntu0.1         libxkbcommon-x11-0=0.8.0-1ubuntu0.1         libxcb1=1.13-1         libxcb-xinerama0=1.13-1         libxcb-randr0=1.13-1         libxcb-render0=1.13-1         libxcb-shm0=1.13-1         libxcb-shape0=1.13-1         libxcb-sync1=1.13-1         libxcb-xfixes0=1.13-1         libxcb-xkb1=1.13-1         libxcb-icccm4=0.4.1-1ubuntu1         libxcb-image0=0.4.0-1build1         libxcb-keysyms1=0.4.0-1         libxcb-util1=0.4.0-0ubuntu3         libxcb-render-util0=0.3.9-1         libx11-xcb1=2:1.6.4-3ubuntu0.4         libc6-dev=2.27-3ubuntu1.5         libc6=2.27-3ubuntu1.5         libc-dev-bin=2.27-3ubuntu1.5         &&     rm -rf /var/lib/apt/lists/* &&     apt-get autoremove -y &&     apt-get clean
 ---> Running in a6b4f2fff62c
failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown

A similar error was mentioned recently on https://discuss.getsol.us/d/8202-docker-not-working-related-to-recent-update/3. Is this perhaps due to kernel config? Currently running:

[manjaro@manjaro-arm appimage]$ uname -r
5.16.16-1-MANJARO-ARM

It does indeed seem that BPF_CGROUP_DEVICE is not enabled:

[manjaro@manjaro-arm appimage]$ zgrep BPF /proc/config.gz 
CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# BPF subsystem
# CONFIG_BPF_SYSCALL is not set
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_DEFAULT_ON=y
# end of BPF subsystem
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
# CONFIG_BPFILTER is not set
# CONFIG_TEST_BPF is not set

Note: imo we should have the image build on non-Docker installations as well. Since the project ships a binary, not a container image, requiring people to run Docker just for this purpose seems counter-intuitive.

robster7674 commented 2 years ago

Researching further shows that it's CONFIG_BPF_SYSCALL that must be enabled, not (just) CONFIG_CGROUP_BPF, see https://github.com/opencontainers/runc/issues/2959#issuecomment-988464142. I guess we'll need to file a Manjaro issue on this to get the kernel config changed.

robster7674 commented 2 years ago

Filed https://github.com/manjaro-pinephone/phosh/issues/35. Note that it's CONFIG_CGROUP_BPF that should also be enabled, not BPF_CGROUP_DEVICE.

SomberNight commented 2 years ago

Note: imo we should have the image build on non-Docker installations as well. Since the project ships a binary, not a container image, requiring people to run Docker just for this purpose seems counter-intuitive.

I think there is a misunderstanding here. Docker is used to build the binary. This dockerfile is not for the enduser to run Electrum on their machine. It is there to set up a container that can be used to build the binary. Prior discussion in this thread is about how to change these build scripts so that they can be used to "cross-compile" the binary - i.e. build a binary on a physical amd64 machine that can be run by endusers on arm64 machines.

Another approach could be to adapt the build scripts so that at least they can also be used on arm64 machines and build arm64 binaries there (so no cross-compiling). -- I would discourage this approach as the project maintainers only have amd64 machines and would prefer to build all binaries on such machines.

Also, if you are trying to run the current build scripts on arm, note that the ubuntu base image is hash-pinned, and I guess the hash might be architecture-dependent: https://github.com/spesmilo/electrum/blob/250795f137281b8d2c8d29081949ee781a1c0b2a/contrib/build-linux/appimage/Dockerfile#L4

robster7674 commented 2 years ago

@SomberNight : thanks for the background information. No, there is no misunderstanding: I am aware Docker is used for building and not needed for running. Still, I feel that on a non-Docker machine, the binary should be able to build as well.

Your comment on cross-compilation makes sense. If it's possible to cross-compile for arm64 on amd64, isn't it possible to do the other way around as well? Let me dig into that.

If hardware is an issue, I am happy to contribute to get some arm64 hardware, like a Raspberry Pi or a PinePhone, to the developers so tests can be run natively on that hardware.

robster7674 commented 2 years ago

Looks like "sudo docker run --platform linux/amd64 ubuntu" could work on arm64. However, I won't know until I get the 5.17 kernel running on my beta23 PinePhone, see https://github.com/manjaro-pinephone/phosh/issues/35#issuecomment-1088367719.

robster7674 commented 2 years ago

Got a bit further now that I have 5.17 kernel. I ran "sudo pacman-mirrors --api --set-branch unstable" to get 5.17.1-2:

[manjaro@manjaro-arm appimage]$ uname -r
5.17.1-2-MANJARO-ARM

I re-ran ./build.sh, and got an error about pyqt5:

ERROR: Could not find a version that satisfies the requirement PyQt5==5.15.6 (from versions: none)
ERROR: No matching distribution found for PyQt5==5.15.6

Then I installed python-pyqt5, and re-ran ./build.sh. Same error, apparently because the version I get is not exactly the same:

[manjaro@manjaro-arm appimage]$ yay -Q python-pyqt5
python-pyqt5 5.15.6-7

5.15.6-7 as opposed to the required 5.15.6. Provided I can figure out where to change the requirement, should I try?

SomberNight commented 2 years ago

PyQt5 is installed here: https://github.com/spesmilo/electrum/blob/ad41f4aed01a8848ce9139f22f84a433c3b7c3cc/contrib/build-linux/appimage/make_appimage.sh#L123-L124 this references contrib/deterministic-build/requirements-binaries.txt: https://github.com/spesmilo/electrum/blob/ad41f4aed01a8848ce9139f22f84a433c3b7c3cc/contrib/deterministic-build/requirements-binaries.txt#L79 (which is committed to the repo to pin hashes; it is generated from: https://github.com/spesmilo/electrum/blob/ad41f4aed01a8848ce9139f22f84a433c3b7c3cc/contrib/requirements/requirements-binaries.txt)

So it is using pip, which downloads packages from PyPI. There are no binary releases for arm linux, see: https://pypi.org/project/PyQt5/5.15.6/#files There is a source release (tar.gz) though. I think because of the --no-binary :all: --only-binary PyQt5,PyQt5-Qt5,cryptography flags, pip will not attempt to get the tar.gz and compile PyQt5 from source -- I would expect that to fail btw.

The easy workaround is to install PyQt5 from the system package manager (note that this is all running inside a docker container, so I am not talking about your host system). You can add it around here: https://github.com/spesmilo/electrum/blob/ad41f4aed01a8848ce9139f22f84a433c3b7c3cc/contrib/build-linux/appimage/Dockerfile#L10 (you would also need to remove pyqt5 from contrib/deterministic-build/requirements-binaries.txt)

SomberNight commented 9 months ago

https://github.com/bitcoin-core/HWI/pull/615 is interesting and simple, but it still leaves the question of what to do with pyqt.

kiminuo commented 8 months ago

Is https://github.com/bitcoin-core/HWI/pull/691 possibly relevant for you?

SomberNight commented 8 months ago

Only to the degree that pyside6 has pre-built wheels for manylinux aarch64. There is discussion re switching to Qt6 in https://github.com/spesmilo/electrum/issues/8007 Though note that atm we are using pyqt, not pyside. pyqt6 does not have pre-built wheels for manylinux aarch64.

Perhaps we should build Qt ourselves, just like we do it for the Android build. Then again, that might blow up the build times, just like we have seen for Android.

SomberNight commented 2 months ago

pyqt6 does not have pre-built wheels for manylinux aarch64

Note: the latest pyqt6 release now also includes wheels for aarch64: https://pypi.org/project/PyQt6/6.7.1/#files https://www.riverbankcomputing.com/pipermail/pyqt/2024-July/045950.html

thecockatiel commented 1 month ago

is there any particular reason the project does not use pyside6?

SomberNight commented 1 month ago

The historical reason is that when we started using PyQt4, pyside did not exist. PyQt works well, so to switch, there should be a good reason. Is there one? :)

thecockatiel commented 1 month ago

~~well, the tools provided by the qt website expect pyside (Qt design studio, etc), which can help developing GUI easier, not that it's needed at this point in this project, but say if someone wants to edit the project to add new stuff to it, it would be easier but yeah I guess there's no good reason compared to the work required~~ update: while the provided tools expect pyside, I noticed there's more stuff required for it to work (so it won't work out of the box with pyside6 either). the only other thing now I'd say pyside6 has that pyqt6 does not have, is stuff like charts (which is irrelevant to the project at the moment).

thanks for explaining the reason, I was just curious to know