Support for Apple Silicon (aarch64-darwin)

[arianvp]

Apple will be transitioning to ARM in the near future, starting with their Big Sur release.

It would be great if we can get nixpkgs support for compiling ARM packages on Darwin

The NixOS Foundation has access to a Developer Transition Kit for testing, thanks to the friendly folks at Apple.

For now @edolstra and @rbvermaa have access to this DTK.

Note that we can get additional DTKs approved for anyone interested working on this ticket. We have a contact at Developer Relations who can help us with this. Please contact me if you would like to order a DTK to work on Nixpkgs support. (Cost is 500 US Dollars)

We also could get access to cloud machines through MacStadium. This might be interesting at a later stage for Hydra?

The original email that I got from Apple DevRel

Firstly, the most expedient way of obtaining a DTK would be for companies who have Nix maintainers and are registered as Apple developers can learn about the Universal Quick Start Program and apply for a DTK here:

https://developer.apple.com/programs/universal/

If this is the way your maintainers want to go, I would be happy to approve the applications to purchase the program. Someone noted on the thread they had applied and not been approved. I have approved then. Please let me know if there are any others.

Secondly, we have a relationship with MacStadium where qualified open source projects can simply get access to a DTK in the cloud. The maintainers of Nix should email opensource@macstadium.com.

The process for qualified OSS developers will be:

Developer signs up for a general MacStadium account so the MSA can be accepted. MacStadium will manually add the DTK asset to their account at no cost. Developer will have access to screen sharing on a DTK and can log in right away

[dhess]

My company has a DTK, so I'm planning to test Nixpkgs on it as soon as Nixpkgs is ready :)

There is already support for a aarch64-darwin platform, but it looks like that was added awhile ago for iOS support. Do we need a new platform tuple?

[matthewbauer]

Yeah - we will probably have to assume aarch64-darwin means Mac instead of iOS now. With the LLVM triple we do have a way to disambiguate:

aarch64-apple-ios

vs.

aarch64-apple-darwin

Which is even more confusing, since iOS is technically still using the Darwin kernel! But nevermind that, I suspect this distinction will get even more blurred with Mac Catalyst. I'd be interested if you can just execute an iOS binary from CLI on the DTK.

Unfortunately, pure-darwin stdenv is left to wait for update of https://github.com/tpoechtrager/cctools-port. We can probably either use the native stdenv or useiOSPrebuilt stuff to get something working though.

[thefloweringash]

Unfortunately, pure-darwin stdenv is left to wait for update of https://github.com/tpoechtrager/cctools-port. We can probably either use the native stdenv or useiOSPrebuilt stuff to get something working though.

What's required from cctools-port for a pure darwin stdenv?

[abathur]

https://eclecticlight.co/2020/08/22/apple-silicon-macs-will-require-signed-code/

[edolstra]

The code signing requirement sounds like the end of the line for macOS support in Nix - or at the very least, for having prebuilt binaries in the binary cache. Users could still build stuff locally using their local signing key (through some impure mechanism, I guess), but they wouldn't be able to share the result with other users and the binaries wouldn't be reproducible.

[arianvp]

There's a similar thread in homebrew about this https://github.com/Homebrew/brew/issues/7857 as they're gonna face similar issues with the code-signing requirement.

From what I understand; signing is done ad-hocly and needs no user interaction and thus does not pose any issues for the availability of a binary cache.

Quoting:

Yep I can confirm signing is all done automatically. It's not associated with any particular identity. The code signing seems to be purely for checksumming and not identity verification. No further action should be needed here. Anyone here updating from beta 1-3 should note however that you may need to rebuild anything already installed. Also, as always, ensure you are running the latest version of Xcode/CLT.

So it sounds like we're good?

[arianvp]

Or is homebrew not backed by a binary cache? i.e. always built from source?

[edolstra]

Yep I can confirm signing is all done automatically. It's not associated with any particular identity.

Then it's not clear to me what the signing key is (or what the point of this kind of signing is...). The release notes mention "a simple ad-hoc signature issued locally" but if that's tied to the machine then you wouldn't be able to share binaries.

[shlevy]

@arianvp There's a purity/reproducibility problem still, but I guess that's not the worst thing.

Absent proper support for detached signatures the closest thing to a "right" way to do this is to teach Nix's hashing to ignore the signature field :sob:

[claui]

Or is homebrew not backed by a binary cache? i.e. always built from source?

brew install downloads and installs binary bottles whenever it can unless you tell it not to.

[thefloweringash]

As far as I can tell adhoc signatures are a hash of the contents of the binary and do not involve an identity, private key or any certificate chain. If so, there's no problem with reproducibility or distribution. My understanding is mostly based of a description of adhoc signing on stack overflow but I haven't found a more authoritative source.

I did a quick test of linking on a DTK. If another DTK user is able to produce the same binary from the same sources it'd be good evidence that there's no machine local key.

[thefloweringash]

I've now thoroughly investigated ad-hoc signatures and implemented a minimal generator: thefloweringash/sigtool. The short version is that it is a list of sha256 hashes of the binary contents up to the start of the embedded signature. I have a work in progress branch of nixpkgs that is able to cross compile and run a basic set of utilities on a dtk. I'm currently using this tool in a setup hook (adding to fixupOutputHooks) that signs every Mach-O file in the output.

[shepting]

@thefloweringash Do you have a link to your branch, or instructions for other folks with a DTK to try out the currently-partially-working version of Nix?

[dhess]

Now that it appears Big Sur is sorted, what’s the status on this one?

[domenkozar]

Note that the foundation has one developer toolkit machine and can set it up for access if that helps anyone work on this. You can request access here :)

Once stdenv compiles, foundation will order a few machines from https://opencollective.com/nixos donations.

[thefloweringash]

@thefloweringash Do you have a link to your branch, or instructions for other folks with a DTK to try out the currently-partially-working version of Nix?

I've tagged something that works as apple-silicon-1. It's incredibly messy, the commit history is equally messy, and still needs a ton of work. However, it can cross-compile bootstrap tools to aarch64-darwin, and then natively build things like hello and nix. My development branch is parrot-principles, but I make no guarantees about anything working there and will rebase and rewrite it frequently.

My cross compilation source machine is x86_64-darwin running 10.15. It should be possible to "cross compile" on aarch64 Mac via rosetta, but I haven't tried it.

---

To natively compile, install regular x86_64-darwin nix and update nix.conf to include

extra-platforms = aarch64-darwin

Natively building hello

$ nix-build --argstr localSystem aarch64-darwin -A hello

---

Cross compiling hello:

$ nix-build --argstr crossSystem aarch64-darwin -A hello

---

Cross compiling bootstrap tools:

$ nix-build pkgs/stdenv/darwin/make-bootstrap-tools.nix --argstr crossSystem aarch64-darwin -A build

[danieldk]

I am not sure if this is the right issue. But would this a good moment to move away from /nix/store? We probably won't have another good opportunity like this on Darwin.

This would mean that the whole dance around creating a Nix store volume would not be necessary on Apple Silicon Macs.

[kloenk]

As long as mac uses something like apfs, and the nix installer has a option to encrypt the partition I would like to keep a partition for the nix store. And keeping it under /nix makes the differences between oses smaller.

[emilazy]

Another advantage of APFS is being able to opt-in to case-sensitivity, which is required for cross-compiling Linux, some nixpkgs ecosystem tools like hackage2nix, and probably other things.

[danieldk]

Another advantage of APFS is being able to opt-in to case-sensitivity, which is required for cross-compiling Linux, some nixpkgs ecosystem tools like hackage2nix, and probably other things.

But that's regardless of whether the store lives in /nix or somewhere else. I think case-sensitive APFS works with most programs nowadays. Also, even if the Nix store is in some other location, you could still use a separate APFS volume if that's a requirement.

[emilazy]

Sure, but I don’t think ”reinstall your entire system on a new case‐sensitive APFS volume” is a reasonable thing for us to suggest users do, and if we’re going to support creating an APFS Nix volume for that reason then we might as well also continue to use it to keep the store prefix the same across platforms. I think that if we can get an automatically‐encrypted‐on‐FileVault, case‐sensitive APFS /nix volume setup in the installer then standardizing on that for all users would be the best approach. (This is kind of off‐topic for this issue, though… AFAIK this has mostly been discussed in #nix-darwin but maybe it would be good to solicit opinions on the Discourse too?)

[uri-canva]

I am not sure if this is the right issue. But would this a good moment to move away from /nix/store?

I think it's unrelated to apple silicon support, so this isn't the right issue.

We probably won't have another good opportunity like this on Darwin.

I'd say it's the opposite: if we were to change the path of the nix store I would prefer if we'd do it either completely before or completely after the apple silicon support: I would not want x86 macOS installations to be different from aarch64 macOS installations in ways that are not strictly necessary, and aarch64 support is likely to be buggy at the start, and I would not want to make that coincide with the change of the path of the nix store, a change that is itself very likely to introduce bugs.

[daniels]

I think it's unrelated to apple silicon support, so this isn't the right issue.

It is related as far as the main reason not to move to a new location has been that the transition of existing systems would be to intrusive, and that relocating only on some systems would mean they can't use the binary cache. If nix on Apple Silicon would get a new binary cache anyway, that is the window of opportunity to move the store and these changes would need to be synced.

[daniels]

Sure, but I don’t think ”reinstall your entire system on a new case‐sensitive APFS volume” is a reasonable thing for us to suggest users do, and if we’re going to support creating an APFS Nix volume for that reason then we might as well also continue to use it to keep the store prefix the same across platforms. I think that if we can get an automatically‐encrypted‐on‐FileVault, case‐sensitive APFS /nix volume setup in the installer then standardizing on that for all users would be the best approach.

From my perspective, expecting (especially new) users to partition their disks to try out nix is already unreasonable enough. It's a long way from the "just delete the /nix dir if you want to remove nix" that gave me the confidence it was worth a shot.

[abathur]

expecting (especially new) users to partition their disks to try out nix is already unreasonable enough.

Not that I don't roughly agree with the sentiment, but I have some nits:

• Unless they want it to be encrypted, we don't ask them to do anything. They just have to let the installer do it.
• It's just adding an APFS volume, which isn't the same as repartitioning. They're pretty simple to create and delete. We aren't allocating fixed amounts of space.

It's a long way from the "just delete the /nix dir if you want to remove nix" that gave me the confidence it was worth a shot.

The uninstall story, however, is (currently) a mess--but I'm actually feeling pretty hopeful here.

I'm working on fixing the encryption case mentioned above, and in the process it ended up making sense to me to add some code for "curing" leftover artifacts (I already feel like the installer halts and dumps directions too often--I'm trying to avoid making that problem worse). This process is actually producing some code and UI/X idioms that (I think) naturally lend themselves to the "uninstall" task. If/when this is done-and-merged, I think adding an --uninstall flag to the installer will be pretty feasible with some help.

[arianvp]

I agree with the premise that considering we have to bootstrap a new binary cache for M1 anway; that switching the nix store path is a very good thing to consider.

However objection wasn't that a binary cache was hard to set up (it isn't) it's that we would have to run two separate hydra infrastructures as I think the path assumptions run pretty deep throughout our CI tooling.

It just means somebody needs to be willing to put in the effort to keep a second set of infra for MacOS. The current hydra maintainers didn't seem to be wanting to put that effort in.

[edolstra]

We've already had a discussion on moving /nix, let's not repeat it here.

[lilyball]

@thefloweringash Are you trying to solve both the code signing requirement and building natively for Apple Silicon at the same time?

Would it be reasonable to just focus on the code signing issue (and just doing it for all darwin targets) with the expectation of running Nix and Nix-installed tools under Rosetta2? That way we could get Nix functional first, and then focus on solving whatever is necessary for native compilation.

Or does Rosetta2 already allow for unsigned binaries? I wish I had an Apple Silicon machine to test with, but in my case, Nix being functional on Apple Silicon is a hard requirement for me before I'm willing to buy Apple Silicon.

[emilazy]

As I understand it, since a recent installer change Nix and nixpkgs work fine under Rosetta 2, and binaries don’t need to be signed, but code signing of native executables is required on Apple Silicon.

Edit: (though there are still infelicities about actually building stuff on Big Sur as I understand it)

[thefloweringash]

As I understand it, since a recent installer change Nix and nixpkgs work fine under Rosetta 2, and binaries don’t need to be signed, but code signing of native executables is required on Apple Silicon.

This is my understanding too. I just tested, and can build and run x86_64-darwin hello on the dtk on current nixpkgs master. I'll confirm this as soon as I get a new Mac.

Edit: (though there are still infelicities about actually building stuff on Big Sur as I understand it)

This should be resolved as soon as the channel updates.

The only thing missing is a release of the installer change in https://github.com/NixOS/nix/pull/4243.

[angerman]

What @thefloweringash did was enough to pave the way for nix on Big Sur, and nix on Big Sur on Apple Silicon chips, when running via rosetta 2. I've posted some install instructions from test the other day here: https://gist.github.com/angerman/cbe02d814d81a8e4d4ced56b19046c19

I'm very much looking forward to having native arm64 nix, however this will likely result in a lot of breakage across the board for a while until enough software is updated and config.sub and other files are adjusted. Having Rosetta as a gateway drug is very helpful. I'm already starting to fear the time it's going to be dropped.

[domenkozar]

@rbvermaa ordered 4 mac minis with 16GB ram via foundation for the build farm, ETA Dec 16-23.

[angerman]

@rbvermaa ordered 4 mac minis with 16GB ram via foundation for the build farm, ETA Dec 16-23.

Ohh man, I hope no one has to work through christmas to set them up. I doubt the linux based vm solution will work.

[domenkozar]

I've backported the installer fix to 2.3-maintenance branch, so once @edolstra makes a Nix release, we have it working on Silicon chip via Rosseta.

[domenkozar]

https://github.com/NixOS/nix/releases/tag/2.3.9

[benpye]

My Apple Silicon device finally arrived and I would love to help get Nix working here (natively). @thefloweringash Given that code signing appears to be a base level requirement would it make sense to try and get that change, and I suppose any changes required for basic build tools upstreamed so that it's easier for the rest of us to try and get our favourite applications building? I'm unsure if there are any limitations with your current implementation but given we'd be going from nothing -> something it seems like a positive change.

[angerman]

@benpye I've confirmed a while back that @thefloweringash and I were able to produce the same (identical) ad-hoc code signature hashes on different machines. This might thus be a little less of an issue than we initially feared.

[Ericson2314]

With the LLVM triple we do have a way to disambiguate:

aarch64-apple-ios

vs.

aarch64-apple-darwin

Which is even more confusing, since iOS is technically still using the Darwin kernel! But nevermind that, I suspect this distinction will get even more blurred with Mac Catalyst. I'd be interested if you can just execute an iOS binary from CLI on the DTK.

We should be able to use aarch64-apple-ios and aarch64-apple-macos. See https://github.com/NixOS/nixpkgs/blob/f7876c6bbffa47b2471c12c8889851aedf71155f/lib/systems/parse.nix#L273-L275 I guess now is time to pull the trigger!

[thefloweringash]

I've opened a pull request with what I have so far, see #105026.

[rbvermaa]

The NixOS Foundation just received the first 4 M1 Mac Mini's for the buildfarm. We hope to have them online in the next 2 weeks for hydra.nixos.org usage. We will get an additional 2 M1 Mac Mini's in a couple of weeks.

[spease]

Moved my comment about getting nix to work on a clean M1 to a separate ticket #106422 since most of it was non-M1 issues. But I did run into an issue with rosetta detection with the current install script on the website, that I suspect may not be an issue with install.in (either that or installing the developer tools changed something, or the install script doesn't detect the lack of developer tools and initially failed before the rosetta error).

Thanks.

[rbvermaa]

We now have a total of 6 M1 Mac Minis. In the next week, I hope to have them online (our current provisioning system doesn't work with them, unfortunately). Will update this ticket once this is done.

[angerman]

@rbvermaa once https://github.com/AsahiLinux is done, the existing virtualized on linux/nixOS might work. But it's a long way to go. I'll just post this for reference here: https://www.youtube.com/watch?v=aMTfPSzrjXs

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/macos-m1-issues/10771/5

[rbvermaa]

Update: install took a bit longer than expected. Went through a manual install yesterday, and created a new bootstrap script for the new macminis. Will test on a newly wiped macmini tomorrow, and if all goes well, they all should be up tomorrow.

[rbvermaa]

Unfortunately hit a snag restoring MacOS, the procedure of the old Intel Mac Mini doesn't work anymore. Luckily was able to restore eventually, but will only have time tomorrow to work on it further. Keep you posted.

[rbvermaa]

I was finally able to bootstrap the new mac mini's properly and finished the last one yesterday evening. :tada: They will be hooked up to hydra.nixos.org in the next hour.

Also, all mac minis from hydra.nixos.org have been moved into a new home (please don't judge me for my poor cable management ;-)

[rbvermaa]

OK, the macminis are now also configured as builders in hydra.nixos.org's config. This means from now on, hydra.nixos.org can be used to build aarch64-darwin builds. To build / test on the platform, hydra jobset still needs to be created, but infrastructure wise everything should be ready .

[vcunat]

Well, I could easily make the trunk jobset include this platform, but it probably doesn't make much sense until someone says that the builds aren't completely broken (I don't know and can't test except through Hydra.nixos.org). EDIT: I see it won't even evaluate ATM due to missing stdenv definition.

I expect we eventually aim simply for

--- a/pkgs/top-level/release.nix
+++ b/pkgs/top-level/release.nix
@@ -14 +14 @@
-, supportedSystems ? [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" ]
+, supportedSystems ? [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ]

(without need to change this jobset's configuration)

[vcunat]

Anyway, why not try the infrastructure already. It does appear to be building successfully. See https://github.com/NixOS/nixpkgs/pull/105026#issuecomment-756707848