search

Robertof / nixos-docker-sd-image-builder

Build custom SD images of NixOS for your Raspberry Pi (or any other supported AArch64 device) in 5-20 minutes.
MIT License
209 stars 35 forks source link

Support for RPi2 #16

Open bensleveritt opened 4 years ago

bensleveritt commented 4 years ago

Please shut this down immediately if it's a non-starter, but what would it take to support RPi2 (Armv7)?

I managed to bumble my way to installing NixOS with a pre-made 19.03 image, but any attempt to rebuild (ie. put cachix or turn SSH on) fails horribly due to the low RAM capability on device. My attempts to build an image on my Mac haven't worked thus far (though I am deeply out of my depth), but many of the tools you're using are the same as I'm trying to understand.

Many thanks for your work!

Robertof commented 4 years ago

Hi!

Thanks for considering using this! I would love to, but there are a few key obstacles:

  1. I don't have a RPi 2 on hand! :p so obviously I can't really test if what I'm doing makes sense.
  2. This repo is specifically tailored for the AArch64 architecture, which the Pi 2 doesn't support.
  3. The default Nix caches don't have any package for armv7, but only amd64 and aarch64.

I won't say that this is impossible, but convoluted! So to do this you would need to try to:

  1. In the QEMU emulation container, you'd need to replace the usage of qemu-aarch64-static with probably something similar to qemu-arm-static (see here for a list of executables). https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/setup-qemu/build/setup-image.sh#L34 https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/setup-qemu/scripts/setup-qemu.sh#L5
    1. Then, you'd need to update the magic masks used to interpret aarch64 executables with the ARM masks available here. https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/setup-qemu/scripts/update-binfmt.sh#L4-L7
  2. Remove/replace the aarch64-tester with an ARMv7 binary or just get rid of this altogether (a sleep would suffice): https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/build-nixos/wait-for-qemu.sh#L1-L9
  3. Configure any ARMv7 cache here, perhaps either this cache or this.
  4. Finally, change the Nix build command line with an appropriate system argument and perhaps other changes I'm not aware of that might be needed: https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/docker-compose.yml#L39-L47

Obviously, these are high level guidelines and I'm not sure of what else can blow up. I'd love to experiment with this but I'm in a pretty busy period right now, so I have very little time to play around with that. If you have, though, please keep me posted -- I'm completely open at the idea of adapting this repo/creating another one to generate RPi 2 images as well!

bensleveritt commented 4 years ago

Wow, thank you for your very considered response, especially since you're busy.

This does seem like a good place for me to start, I'm certainly keen to see how it pans out. I'll see how I get on, and will be sure to feedback successes (hopefully) I have!

Robertof commented 4 years ago

Glad that my response was of inspiration! I'll do my best to help if anything comes up, thanks again for considering the use of this!

bensleveritt commented 3 years ago

I've made a some changes, and things seem to be going in the right direction, but I'm stuck on this error:

build-nixos_1   | error: Package ‘nixos-sd-image-20.03post-git-arm-linux.img’ in /home/nixos/nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix:134 is not supported on ‘arm-linux’, refusing to evaluate.

Any idea what's happpening? I'd assume binfmt would have prevented it. Perhaps I've got the wrong architecture specified somewhere.

Also: what's stopping us from using these Rpi images as base-derivatives? I don't understand derivations enough...

Robertof commented 3 years ago

Hey! Huge apologies for the delay in getting back to you -- unfortunately things haven't gotten less busy recently!

That error is quite weird -- I suppose you're using the 20.03 branch which has really nothing much interesting there:

https://github.com/NixOS/nixpkgs/blob/release-20.03/nixos/modules/installer/cd-dvd/sd-image.nix#L134

The RPi image derivation is actually used as a base -- however, to be able to change some of the hardcoded options there I've just duplicated them in this tree!

What value did you use for --option system in docker-compose.yml? You should use --option system armv7l-linux. Regardless, I would suggest two course of actions here:

Unfortunately the documentation for building NixOS non-natively for ARMv7 devices is slowly shrinking as AArch64 rises, so documentation is quite sparse! Let me know if that works.

Thank you, Roberto