Add Darwin support for Swift

[matthewbauer]

Swift should support macOS/iOS correctly. Currently, it's disabled for Darwin stdenvs.

/cc @NixOS/darwin-maintainers

[dtzWill]

/me subscribes JIC

[matthewbauer]

Haven't gotten to this yet. Will move to 19.03.

[infinisil]

@matthewbauer Is this still something you want to do for 19.03?

[matthewbauer]

I think it's best to move it to 19.09 for now. It's not a huge priority and looks a little bit harder than I originally thought.

[veprbl]

Turns out newer versions of mpv need swift: #79344

[burke]

This is going to become a blocker for us at some point later this year too.

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[stephank]

I was making an attempt at this, but it's very much on-the-side. I don't actually use Swift nor know much about compiler toolchains.

Got a decent part of the build going. May look into this a bit more later: https://github.com/stephank/nixpkgs/commit/2b035eff96722093ac458fe34fcef31ea304c7ae

[stale[bot]]

I marked this as stale due to inactivity. → More info

[stephank]

I've picked this up again, and am making good progress. Also attempting to upgrade Swift to 5.6.2, and will attempt to fix the currently broken build on Linux as well.

[stephank]

I realized it's been 3 weeks since I gave an update on this, but I'm still making progress. :)

My first attempt was to simply iterate on the existing build to see if I could get it working on Darwin. I spent a good amount of time fighting Swift's build-script, but hit a dead end. I got up to successfully building libswiftCore, so was actually building some Swift code, but after that I got errors that were difficult to debug, plus I was dealing with spurious failures earlier in the LLVM build, and iterating on the build was slow and difficult in general.

I discovered a little late that build-script has a --dry-run flag which shows the underlying CMake invocations. I'm now making an attempt at splitting up the build in parts based on that. As it turns out, many of the flags passed to CMake are already defaults, and tying the whole build together using CMake was less magic than expected. This is great, because it allows for reusing existing Nixpkgs infrastructure to deal with CMake, and just gives us more control in general.

In both cases I also worked on wrapping the compiler. I'm applying cc-wrapper to the internal Clang in the Swift build, and built a swiftc wrapper. This has helped a great deal with hooking up dependencies on Darwin.

For now I'm working on this new approach on Linux, because the hardware is cheaper to rent. (I have GH Sponsors setup if anyone wants to pitch in with these costs, wink wink) The hope is it'll translate more easily to Darwin, or at least be easier to debug and iterate on.

Current stage I'm at (on Linux) is that I have a working Nix build of swiftpm via CMake and can use that to build swiftpm itself outside Nix. I want to make a simple swift2nix like solution to complete the bootstrap by building swiftpm with swiftpm. I found 2 existing swift2nix tools, but they appear to be either outdated, or using builtins.fetchGit (undesirable because impure and evaluation-time).

It's difficult to share or PR anything at this point, because few things are set in stone. I frequently force push to branches, but that can also change:

• https://github.com/NixOS/nixpkgs/compare/master...stephank:nixpkgs:test/swift is the old Darwin attempt, but contains some stdenv fixes that I'll likely need for the new attempt as well. If any of these fixes are still necessary, they will become separate PRs on staging first.
  • I already created one PR as a result of this: https://github.com/NixOS/nixpkgs/pull/181431
• https://github.com/NixOS/nixpkgs/compare/master...stephank:nixpkgs:test/swift2 is the new Linux split build attempt.

So making good progress all in all, but more work than expected. 🙂

[Atemu]

@stephank are you aware of https://opencollective.com/nix-macos?

Also make sure you're not being blocked by our ancient SDK. An optional binary version of SDK11 was merged not too long ago.

[stephank]

@stephank are you aware of https://opencollective.com/nix-macos?

I wasn't! Will look into it.

Also make sure you're not being blocked by our ancient SDK. An optional binary version of SDK11 was merged not too long ago.

Yes, it already depends on that, because Swift needs at least 10.15. I started working the day after it was merged. 🙂

[stephank]

Two weeks passed, another update:

Domen Kožar was kind enough to reimburse the hardware costs via OpenCollective, so that’s great! My time is still my own so far, and I believe I’ve been at this for 2 to 3 days a week for almost two months now. So, bigger effort than expected!

I wrote a small bash + jq tool ‘swiftpm2nix’ (to avoid ‘swift2nix’, because there are already two of those) that generates a Nix expression for fetching project dependencies, and a configurePhase snippet that prepares everything for swift-build to work offline. Using this, I was able to successfully bootstrap swiftpm with itself on Linux.

I also added a small setup-hook, so that nativeBuildInputs = [ swiftpm ]; gives you a default buildPhase and checkPhase, plus a helper swiftpmBinPath that can help you find build products when writing an installPhase for your package.

Using these integrations, I was also able to package other toolchain components: swift-docc, sourcekit-lsp, swift-driver.

I’m now tackling issues on Darwin again.

I first ran into https://github.com/NixOS/nixpkgs/pull/181485. This took some debugging, because I was not immediately using the right search terms and trying my own hand at fixing it.

After that, I ran into an assertion failure in ld. This again led to several days of debugging. One of the solutions I attempted was to simply upgrade cctools, which unfortunately didn’t work, but I made a PR for any way: https://github.com/NixOS/nixpkgs/pull/185656. I eventually debugged the issue with lldb, and discovered that https://github.com/NixOS/nixpkgs/pull/181485 was the cause, and created a workaround in https://github.com/NixOS/nixpkgs/pull/185766.

The reason I went down this path was because I was trying to introduce LLVM >=12 into the build, because Swift uses @available checks that are only implemented as of that version.

I was still seeing link errors related to @available, and it turns out our compiler-rt doesn’t contain any builtins on Darwin. It’s a little surprising this hasn’t caused trouble before. I fixed this on my swift branch, but haven’t created a PR yet, because I first need some improvements in xcbuild, for which I created https://github.com/NixOS/nixpkgs/pull/185969.

Next, I discovered import Darwin doesn’t work because libsystem doesn’t contain the required modulemaps. PR: https://github.com/NixOS/nixpkgs/pull/186251

I’m now at a point where it looks like ObjC interop is using Swift modules that may be binary-only in the macOS SDK, and figuring out what to do with those.

[stephank]

Sorry for the lack of updates, but there is now a PR: https://github.com/NixOS/nixpkgs/pull/189977

I was able to get the Swift compiler itself working on Darwin shortly after the last update, but then spent a lot of time tweaking installation details and issues with SwiftPM. There is a lot of Darwin-specific logic in the whole toolchain.

So status is that we're able to build basic (CLI) stuff on all platforms using SwiftPM, but I've not yet got it to build a simple SwiftUI app.

[sheldonneuberger-sc]

I noticed that https://github.com/NixOS/nixpkgs/pull/189977 just made it into nixpkgs-unstable, thanks @stephank! Does this unblock the ability to have cli-based swift programs in nixpkgs?

[stephank]

Yes, CLI-based apps should definitely work. Some of the tools from the Swift toolchain itself are Swift CLI apps, like swiftpm and sourcekit-lsp, and they work fine.

We should be able to build Cocoa stuff too! I tested a very basic SwiftUI app, and we should also be able to move forward with the Swift components in mpv, for example.

There's a follow up PR (to master) that further upgrades to Swift 5.7.3, and fixes using Swift in a Nix shell (or other dev env): https://github.com/NixOS/nixpkgs/pull/213936

Beyond that, we just have to tackle issues as we go. For example, we did end up finding a compilation issue in the Perfect web framework (https://github.com/cachix/devenv/pull/312#issuecomment-1399328024). So far, that's the only one I know of.

But I think we're good to close this issue?

[Atemu]

Yes, thank you so much for your work @stephank!