search

tpwrules / nixos-apple-silicon

Resources to install NixOS bare metal on Apple Silicon Macs
MIT License
838 stars 80 forks source link

Flake support, project reorganization, asahi overlay #47

Closed oati closed 1 year ago

oati commented 1 year ago

Resolves #34 (and #36 #37) Supersedes #35 Supersedes #46 and closes #45 (probably)

This PR does a couple of things:

CI and docs would need to be updated.

I'm currently using the nixos module output of the flake for my system configuration, but the iso is yet to be tested.

oati commented 1 year ago

Hmm, including a mesa override in the overlay causes rebuilds for everything that depends on mesa, regardless of whether the experimental GPU option is actually enabled. The option ends up being kind of silly, since it doesn't actually change anything. I'll fix this tomorrow.

oati commented 1 year ago

Actually, I think this is good enough. Overriding mesa globally causes so many rebuilds that it probably shouldn't be easy to do so.

The overlay now lets you reference pkgs.mesa-asahi-edge from anywhere, so you could just override packages as needed, or add an overlay yourself to override globally:

nixpkgs.overlays = [ (final: prev: { mesa = final.mesa-asahi-edge; }) ];

(regarding #45)

tpwrules commented 1 year ago

While I appreciate the effort, this is a big change for the project and I'm not sure about it.

Primarily, I'm not ready to go flakes-only. Though popular, they are still experimental and require extra effort from users to set up, and the simplicity of the current install is useful. There are certainly things I could adjust for a better flakes experience but I don't regularly use them in this context so some suggestions here would be helpful.

I do think the code needs reorganization and was hoping #35 would provide a good solution to that, but it doesn't seem to have happened. I will probably put together a PR myself for this which cribs ideas from this PR and #35 and solicit feedback from you and the community on if that meets your needs. It's hard to accept a big PR like this one without having fiddled with it myself first.

As for the Mesa fixes, I have a better solution to that I am working on which avoids so much recompilation and hope to get it out soon.

sequencer commented 1 year ago

wondering is possible for these tricks to be a part of NixOS/nixos-hardware

oati commented 1 year ago

Primarily, I'm not ready to go flakes-only. Though popular, they are still experimental and require extra effort from users to set up, and the simplicity of the current install is useful. There are certainly things I could adjust for a better flakes experience but I don't regularly use them in this context so some suggestions here would be helpful.

The nixos module can still easily be used without using flakes.

I decided to go with flakes-only for the iso and package outputs because flakes solve many of the problems that default.nix tries to solve in more ad-hoc, partial, and/or impure ways.

Namely,

oati commented 1 year ago

If you'd prefer, (and if you end up agreeing that flakes would be the better solution,) I could get rid of the flake-parts dependency to make the simplest "basic nix code" flake, and figure out flake-compat to maintain the current installation instructions.

flake-compat (very popular) would let you use flake.lock in place of pins.nix, without needing to use flakes. This lets you keep one lockfile for both flake and non-flake users.

oati commented 1 year ago

As for just ensuring a better experience for flake users, the nixos module is easy enough to use as a non-flake input (specifically since the nixos module doesn't have any required project depnendencies):

imports = [
  "${inputs.nixos-m1-non-flake}/nixos-module"
];

You'd just have to get rid of the /. + in https://github.com/tpwrules/nixos-m1/blob/main/nix/m1-support/peripheral-firmware/default.nix#L24 so that /boot/asahi doesn't break as a non-flake input.

One small advantage (for the nixos module) for having the project be a flake, however, is that flake users would gain the ability to easily pin the asahi packages to the "working versions" specified by the nixpkgs in the flake lockfile.

oati commented 1 year ago

(to clarify) The flake in this PR does not require users to have a flake-based system configuration. The nixos module can still be imported manually, through a channel, or through a tarball. The PR merely adds a standardized and convenient way to import the module from a flake.

This PR does require either an extra CLI flag --experimental-features 'nix-command flake' (or a nix user config enabling these) in order to build packages or the iso with the pinned nixpkgs and rust-overlay.

However, this can be completely worked around with via flake-compat, in which case users will never have to touch nix flakes for any use case.

tpwrules commented 1 year ago

Thanks for the clarification on your intentions and the outcomes. Yeah, I agree the current situation is rather ad-hoc and clunky. You raise some good points about how flakes would make it cleaner. I'm not aware that there are any actual problems here at the moment in particular with regard to purity, but making it impossible to make those kinds of mistakes is good. I don't think flake-compat is necessary at this point, I'm not a never-flakes person either.

It's very important to me that a user be able to download an ISO off GitHub, boot it, and install quickly without having to download and rebuild a whole kernel on their system. Nix does make it easy, but it's still not a great time on the lower-end machines. Some of the impurities as I recall were there to make this work. AFAIK these should be possible to deal with when involving flakes but they do need to work properly.

Please give me a week or so to review and play with this in depth. Please also rebase on top of the latest release. I might do another release in a couple days once the recent nixpkgs staging merge makes it to the NixOS release channels.

tpwrules commented 1 year ago

wondering is possible for these tricks to be a part of NixOS/nixos-hardware

@sequencer Some people have tried this (and moving support for these machines into nixpkgs too) but I haven't kept much track and I don't think it's feasible until the kernel upstreaming matures a little more.

oati commented 1 year ago

Ok, I'm running into new issues with the recent changes on main: https://github.com/NixOS/nixpkgs/issues/199162 https://github.com/NixOS/nix/issues/5868

system.replaceRuntimeDependencies just fails in pure evaluation due to builtins.storePath, and therefore on flake-based nixos configurations. I haven't found a workaround.

tpwrules commented 1 year ago

Hm, that was another memory around impurity that is now surfacing. I used to have system.replaceRuntimeDependencies in the default config for another package. I don't think there is a workaround either.

I can add an option to overlay the new mesa I suppose but that's rather unfair for flakes users because essentially everything needed to be rebuilt.

shaunsingh commented 1 year ago

Instead of using fetchFromGitHub to grab the source for m1n1/kernel/etc, could we just put those as pinned inputs in the flake.nix, that way if users wanted to override the source for e.g. mesa to a newer version or fork for debugging they easily could

oati commented 1 year ago

I did consider that option while working on this.

One issue is that it would become much more difficult for non-flake users to incorporate the nixos module into their system, since the overlay would depend on this project's flake.lock.

Flake users can already pretty easily override sources with flake inputs via overlays:

final: prev: {
  mesa-asahi-edge = prev.mesa-asahi-edge.overrideAttrs (old: {
    src = inputs.mesa-asahi;
  });
}
oati commented 1 year ago

TODO: Try to get m1n1 and uboot-asahi to cross-compile without using pkgsCross.aarch64-multiplatform. Cross-compilation should be handled using the crossSystem and localSystem arguments of nixpkgs, which we can control from our flake.

This will make m1n1 and uboot-asahi cross-compile "natively" using nixpkgs, in the same way that everything else in the iso is cross-compiled (and other packages in the overlay).

Also, localPkgs in the kernel package is now always pkgs, since builtins.currentSystem is impure. (I'm not sure if localPkgs does anything for us. The only system that we're building for is aarch64-linux, so we don't need the config to be reusable.)

tpwrules commented 1 year ago

The catch with m1n1 is that it has to be a native build so that the tools like the proxy client and command line come out for x86_64-linux but also a cross build so that the m1n1 stub is built for aarch64-linux. Might be possible to split this into two separate derivations?

I think your solution is workable for U-Boot though, it always expects to be run using an aarch64 stdenv.

The bit with the kernel package is necessary to support the "install a kernel from a cross-compiled ISO without rebuilding it" scenario. The comment might be a little bit incorrect but we do in fact build the config derivation "for" x86_64-linux when cross-building the ISO, even though the contents never change. There might be a better way to do that. The current clause there should be compatible with flakes as it specifically checks for currentSystem to exist and assumes you are running natively if not.

oati commented 1 year ago

In the absolute worst case, we could explicitly re-import nixpkgs with crossSystem and localSystem to avoid problems with pkgsCross.aarch64-multiplatform (which is I think itself a re-import of nixpkgs with crossSystem.config = "aarch64-unknown-linux-gnu", which causes problems because it's not the same as importing with crossSystem.system = "aarch64-linux").

As for the kernel config IFD, we can just replace builtins.currentSystem with localSystem as a pure alternative.

note: nixpkgs uses native builds when crossSystem = localSystem (in some sense). It doesn't (won't) cross-build from the same platform. (although I think there's an open issue to make everything always cross-build?)

tpwrules commented 1 year ago

Okay, this should be the final rebase. I am still interested in this, just had to make sure everybody's builds didn't break.

oati commented 1 year ago

Some notes about the default firmware directory:

builtins.pathExists is impure, and it (I think) always outputs false in pure mode. https://github.com/NixOS/nix/issues/3998 (stale bot....)

Maybe we should set the default to "/boot/asahi" and override this in the installer configuration?

(edit: actually "/boot/asahi" as a default makes no sense in pure mode since it can't be accessed anyway.)

tpwrules commented 1 year ago

Yes, the fact that builtins.pathExists is always false in pure mode is fine. The documentation covers this case and provides the alternate path for the user to point somewhere their flake can access.

The reason for the two paths is evaluation in the installer ISO. The actual installer gets the firmware by running asahi-fwextract automatically after booting, the configuration options don't matter for it. But when the user runs nixos-install, their configuration will be evaluated when the ESP is mounted at /mnt/boot/asahi (assuming they followed the installation instructions) and that's when the second path option comes into play.

oati commented 1 year ago

I've incorporated the recent changes from main. a1e2ef9 8361211 might need testing for cross-builds, but I think it should be correct.

ingenieroariel commented 1 year ago

Thanks for all this work, I have replaced my local config with the latest version of this PR, at the end of this comment I paste a copy of my flake.nix on /etc/nixos.

FYI, The overlay strategy with the latest version of this flake gives:

error: infinite recursion encountered

       at /nix/store/mhx0418frlp5p3i20bk6b0m41klps12m-source/apple-silicon-support/packages/mesa-asahi-edge/default.nix:5:2:

            4|
            5| (mesa.override {
             |  ^
            6|   galliumDrivers = [ "swrast" "asahi" ];
(use '--show-trace' to show detailed location information)
{
  description = "A very basic flake";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/master";
    home-manager = {
      url = github:nix-community/home-manager;
      inputs.nixpkgs.follows = "nixpkgs";
    };
    rust-overlay = {
      url = github:oxalica/rust-overlay;
      inputs.nixpkgs.follows = "nixpkgs";
    };

    nixos-m1-overlay = {
      url = github:oati/nixos-m1/flake;
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.rust-overlay.follows = "rust-overlay";
    };

    };

  outputs = {
    self,
    nixpkgs,
    rust-overlay,
    nixos-m1-overlay,
    home-manager,
  }: let
    system = "aarch64-linux";

    pkgs = import nixpkgs {
      inherit system;
      config.allowUnfree = true;
      overlays = [ 
          rust-overlay.overlays.default
          nixos-m1-overlay.overlays.default
      ];
    };
    lib = nixpkgs.lib;
  in {
    nixosConfigurations = {
      nixos = lib.nixosSystem {
        inherit system;
        modules = [
          ./configuration.nix
          nixos-m1-overlay.nixosModules.default
          home-manager.nixosModules.home-manager
          {
            home-manager.useGlobalPkgs = true;
            home-manager.useUserPackages = true;
          }
        ];
         };
    };
  };
}
ingenieroariel commented 1 year ago

I have verified the latest version of this PR on a new (flake-based) configuration and it works.

This set up is very flexible for downstream users like me. Here is an example:

I let nix flakes update the rust overlay to latest and nixpkgs to latest, and it worked after adding this to my configuration.nix (blacklist-types is now blocklist-types).

  boot.kernelPatches = [
      {
        name = "edge-config";
        patch = null;
        # derived from
        # https://github.com/AsahiLinux/PKGBUILDs/blob/stable/linux-asahi/config.edge
        extraConfig = ''
          DRM_SIMPLEDRM_BACKLIGHT n
          BACKLIGHT_GPIO n
          DRM_APPLE m
          APPLE_SMC m
          APPLE_SMC_RTKIT m
          APPLE_RTKIT m
          APPLE_MAILBOX m
          GPIO_MACSMC m
          DRM_VGEM n
          DRM_SCHED y
          DRM_GEM_SHMEM_HELPER y
          DRM_ASAHI m
          SUSPEND y
        '';
      }
      {
        name = "blocklist-type";
        patch = builtins.fetchurl "https://patch-diff.githubusercontent.com/raw/AsahiLinux/linux/pull/109.diff";
      }
    ];