Cachix for mesa?

[n8henrie]

Hi,

Thanks for your incredible work on this project!

Sorry if this has been explored elsewhere; I'm a flakes user looking at the options for enabling GPU support without losing purity. It seems like "rebuilding the world" is the most obvious approach.

Still relatively new to nix -- just wondering, would setting up a shared binary cache on Cachix alleviate the difficulties here? (I know there is even a GitHub Action, seems possible that the CI might even be able to do some of the work over time.)

Thanks again for your work, sorry if this kind of question would have been better posted elsewhere!

[tpwrules]

This could work, but I am not sure how to get one and I don't really have a good way to populate it.

I don't think losing purity is a huge concern for flakes, as that only applies to evaluation. If it's really important to you, you could test evaluation in pure mode with the "overlay" GPU option then switch to "replace" and impure mode to actually build your system.

[n8henrie]

Re: purity, to be honest I'm mostly concerned about having to pass --impure all the time and as a result potentially allowing in other impurities and not realizing it.

It's very easy to set up and populate from GitHub Actions, but I think the free tier ends at 5GB, which I imagine would fill up quickly.

[tpwrules]

Another big problem is that there are no ARM GitHub Actions builders (unless someone self-hosted one?) and Nix treats cross-compiled binaries as different from native ones.

[n8henrie]

Oh, that's a good point. I hadn't realized that cross compiled binaries wouldn't work.

IIRC arm64 GA builders aren't expected until this fall.

[ingenieroariel]

I have a NixOS M1 Ultra online on a public IP address / DNS and I set it up to rebuild the world. If others want I could enable a public binary cache.

[rowanG077]

@ingenieroariel Yes please!

[mariesavch]

@ingenieroariel yes!

[zzywysm]

Perhaps you could leverage this as a solution?

https://determinate.systems/posts/magic-nix-cache

[n3oney]

You can try using https://garnix.io/ which can cache nixosConfigurations, has ARM runners, and is free and easy to set up. All you need is a garnix.yaml in the repo root (also need a flake)

builds:
  exclude: []
  include:
    # Remove what doesn't apply
    - '*.x86_64-linux.*'
    - '*.aarch64-linux.*'
    - 'nixosConfigurations.*'
    - 'formatter.*'
    - 'devShell.*'

then register on garnix.io, add the repo there, and add the garnix cache to your config.

[ingenieroariel]

I recently installed NixOS from the USB installer (I used flakes in the past so my versions never matched what a new user may see).

It is based on the 2024-01-17 release.

nix-info -m
 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.0-asahi, NixOS, 24.05 (Uakari), 24.05.20240115.c3e128f`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

You can use it with cachix use ceon

Config on the machine that pushed is:

  boot.loader.efi.canTouchEfiVariables = false;
  hardware.opengl.enable = true;
  hardware.asahi.experimentalGPUInstallMode = "overlay";
  hardware.asahi.useExperimentalGPUDriver = true;
  hardware.asahi.addEdgeKernelConfig = true;
  hardware.asahi.withRust = true;
  hardware.asahi.peripheralFirmwareDirectory = ./firmware;

If anyone can test it would be appreciated, I can set up a workflow to update the cache every time there is a new release. Also any suggestions for how to keep up are welcome. The machine building it is an M2 Ultra.

Current list of packages is:

   environment.systemPackages = with pkgs; [
     vim 
     wget
     htop
     ripgrep
     firefox
     dmenu
     sway
     foot
     alacritty
     kitty
     git
     cachix
     arcan
     glmark2
     mesa-demos 
     fishPlugins.done
     fishPlugins.fzf-fish
     fishPlugins.forgit
     fishPlugins.hydro
     fzf
     fishPlugins.grc
     grc
     xarcan
     gnumake
     unzip
     cmake
     patch
     openssl
     qemu
     texinfo
     bison
     zip
     autoconf
     automake
     libtool
     ninja
     m4
     tmux
     binutils
     ladybird
   ];

[n8henrie]

@ingenieroariel I decided to try the overlay option and just added ceon -- thank you. Though I'm on an M1, so I'm not entirely sure if I'll be able to reuse the same cached binaries -- hopefully!

I can set up a workflow to update the cache every time there is a new release. Also any suggestions for how to keep up are welcome.

I would recommend setting up a (public) GitHub repo that hosts a configuration with the above packages (and perhaps you could take some requests as well). Set up GitHub Actions to be able to build the configuration, and set it to use cachix. Set up a daily cron job that runs something along the lines of nix flake update nixos-silicon, and if an update was available, it builds the configuration and pushes the result to cachix. If the build was successful could also configure to self-commit the new flake.lock.

I have some of this working for aarch64-linux (via binfmt-misc) here: https://github.com/n8henrie/nixos-btrfs-pi/blob/master/.github/workflows/build.yml

Now that GitHub has M1 runners, an alternative (maybe better?) possibility would be specifying an Apple Silicon runner and building with darwin.linux-builder.

[montchr]

I would really love to see this become a thing. Relates to #217.

I wonder if this kind of infrastructure might ideally be managed through nix-community somehow? They already provide a lot of CI infrastructure (link) for their projects, but I think that only applies to nix-community-owned projects.