armv7 Hydra channel?

[doronbehar]

Is your feature request related to a problem? Please describe.

At my work we often use Armv7 devices like Raspberry Pi. Often my colleagues complain about issues with outdated packages in the default Debian systems these devices use by default and the lack of declarative management of their deployments. Naturally Nix is the answer. However, these devices usually have low memory & CPU power, making it hard to build full systems on them.

Describe the solution you'd like

Taking https://github.com/NixOS/nixos-org-configurations/pull/142 as an example, is it that simple and legitimate to add a channel for an architecture? Having an armv7 channel building and caching armv7 builds would enable me to dramatically change the experience with such devices.

Describe alternatives you've considered

nixbuild seems like an awesome service that could benefit teams such as those I occupy. However, I still think that a tier 1 support for armv7 could motivate many big software companies to use Nix, see e.g this. Also, nixbuild's pricing is the fairest I could have imagined, but as a junior in a non Nix team, it's hard to convince people to use such a service which even myself is not experienced with.

Additional context

https://github.com/NixOS/nixpkgs/issues/83049

[samueldr]

The main problem is having build machine to build for armv7l.

Most AArch64 providers only have machines supporting 64 bit systems.

Furthermore, the builds have to happen on an armv7l kernel, not AArch64 since the personality subsystems in Linux are not as robust for ARM, and ambiant impurities leaks into the build. So for the few systems allowing 32 bit execution, we need to "burn" the virtualisation capabilities by making armv7l VMs on the AArch64 host.

There has been some exploratory work for this earlier, I don't know if there was anything conclusions coming from that or if it was simply not completed due to lack of people-power to work on that.

[doronbehar]

The main problem is having build machine to build for armv7l.

So is this a problem of merely finding a provider for armv7l? Perhaps the provider of nixbuild.net would fit for nixos.org?

[samueldr]

Mainly yes. It depends how nixbuild.net does it. If they simply flipped the option to run 32 bit on aarch64 systems there's always some possibility of getting ambiant impurities in. (Though less so than doing the same with armv6l in mind.)

That is for the technical aspect.

Though there might be other roadblocks that are more organizational and social to get it going, but nothing major I hope. One of these is supporting the architecture. Though tbf it can't happen without first getting good regular build coverage.

[misuzu]

nixbuild.net uses aarch64 kernel with personality patch: https://discourse.nixos.org/t/support-for-aarch64-linux-armv7l-linux-and-i686-linux-is-now-generally-available-on-nixbuild-net/21331/3 I'm also using this approach and regularly fix issues. There's issues with ambiant impurities (for example, patchelf crashes with SIGBUS error when building go-bootstrap which is not reproducible when building on armv7l-only machine), but these should be easy to spot and fix once armv7l-linux is added to the hydra. It's a lot less work than cross-compiling IMO.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/extend-cross-compilation-support-in-hydra/26783/1

[RossComputerGuy]

I think this can be done via Ampere machines, they have the ability to execute 32-bit ARM code and not just 64-bit ARM code. These machines tend to have a lot of resources, I think we already have some Ampere machines running Hydra right now but I am not familiar with their configuration.

The other thing to think about is how many people actually use armv7 nowadays. aarch64 is more common and if only a few people use armv7, it would just bloat the cache. I know this issue is kinda old but I wonder what the current sentiment with this is for the infra folks.

[vcunat]

I'd say that over the past two years or so, armv7 has been getting less relevant (slowly) with more switching to aarch64. NixOS packages tend to be relatively bloated, so the binaries probably aren't suitable for the most embedded cases (but I don't know embedded stuff really).

[RossComputerGuy]

Yeah, that's kinda my point. I wonder if people are using self hosted caches or their own infra for embedded stuff.

[samueldr]

(hi, got notified from an old subscription to this issue)

---

I think this can be done via Ampere machines, they have the ability to execute 32-bit ARM code and not just 64-bit ARM code.

Same warning as before stands. The personality features of the kernel are not sufficient for correct 32/64 bit shenanigans, unlike on x86_64, where they are. [citation needed]

Running the builders as VMs would be necessary. Consider this in the analysis, coupled with the way nested virt is YMMV in general on aarch64.

---

so the binaries probably aren't suitable for the most embedded cases (but I don't know embedded stuff really).

Binary size and embeddedness is orthogonal to ISA. It shouldn't matter in the equation for support level.

A specific [general purpose] ISA cannot be presumed to be used in "embedded" situations.

Additionally, "embedded" basically has lost all meaning once Linux is involved, and multi-gigabyte storage for the system for "embedded" systems is not unheard of either.

[RossComputerGuy]

Same warning as before stands. The personality features of the kernel are not sufficient for correct 32/64 bit shenanigans, unlike on x86_64, where they are. [citation needed]

Yeah, I was curious as what Ampere's 32-bit support is like and it's EL0 which means userspace only (source). This means no 32-bit kernel. However, you can run 32-bit programs. At the very least, this means you can cross compile without an emulator.

Running the builders as VMs would be necessary.

Would a VM always be necessary, even if there's 32-bit EL0-EL3 support?

[samueldr]

I don't know enough about 32 bit execution modes to say for sure. It was an issue to run a 32 bit system on a 64 bit kernel, as it ended-up introducing ambient impurities from "ambient capabilities" [if you excuse the vague term] available.

The only way it was found to have outputs that reliably work on actual armv7 devices was with virtual machines, back in like 2018 or 2019 IIRC.

I'll note that it's not too different from running a build on the first 64 bit x86_64 systems compared to the most recent x86_64 ones, except the general class of issues is generally more kept in mind and fixed by other actors [upstream, other distros] too when platform "capabilities" are sniffed at during build time. The build sandbox does not sandbox from the host's architecture.

[misuzu]

You can build natively for armv7l-linux on the Community aarch64 NixOS box (it has this patch):

$ uname -a
Linux aarch64.nixos.community 5.15.163 #1-NixOS SMP Thu Jul 18 11:07:45 UTC 2024 aarch64 GNU/Linux
$ linux32 uname -a
Linux aarch64.nixos.community 5.15.163 #1-NixOS SMP Thu Jul 18 11:07:45 UTC 2024 armv7l GNU/Linux
$ nix build nixpkgs#legacyPackages.armv7l-linux.hello
$ nix run nixpkgs#file -- ./result/bin/hello
./result/bin/hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/bzbk14p68xvynv457qnyz9f31f6rw7dx-glibc-2.39-52/lib/ld-linux-armhf.so.3, for GNU/Linux 3.10.0, not stripped
$ ./result/bin/hello
Hello, world!

Until I've accidentally fried my last armv7l board, I've built full NixOS systems like this for a year and half and had almost no compatibility issues when building (see https://github.com/NixOS/nixpkgs/pull/255804) and zero compatibility issues when running on the actual armv7l device.

Any Ampere Altra Q80-30 server with the above patch and 6.1+ kernel should work fine to build natively for armv7l-linux. If we could use deterministic-uname everywhere the kernel patch would not be needed.

The only way it was found to have outputs that reliably work on actual armv7 devices was with virtual machines, back in like 2018 or 2019 IIRC.

Nix didn't set PER_LINUX32 flag for 32-bit arm before https://github.com/NixOS/nix/pull/5276

[doronbehar]

Thank you very much @misuzu for sharing! For years I was looking for a Nix friendly aarch64 build service that would be cheap enough for me :). That legacyPackages.armv7l-linux attribute makes sense for building a package, but it is a bit hard for me to guess how to setup a flake.nix etc and what Nix command to run on that flake for a system. I also wonder, how do you copy the build results to your arm machine? Do you use cachix?

[misuzu]

That legacyPackages.armv7l-linux attribute makes sense for building a package, but it is a bit hard for me to guess how to setup a flake.nix etc and what Nix command to run on that flake for a system.

The simplest flake looks like this

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  outputs = inputs: {
    nixosConfigurations.my-nixos = inputs.nixpkgs.lib.nixosSystem {
      system = "armv7l-linux";
      modules = [ ./configuration.nix ];
    };
  };
}

Use nix build .#nixosConfigurations.my-nixos.config.system.build.toplevel to build the system. The non-flakes equivalent is nix-build '<nixpkgs/nixos>' -A config.system.build.toplevel --system armv7l-linux -I nixos-config=./configuration.nix

I also wonder, how do you copy the build results to your arm machine? Do you use cachix?

I've used something like this.