Build failure: Wezterm (x86-64 darwin)

augustebaum commented 1 year ago

Build log

Builds have been failing for the past 5 days: https://hydra.nixos.org/job/nixpkgs/trunk/wezterm.x86_64-darwin/all

Notify maintainers

@SuperSandro2000 @mimame

Metadata

- system: `"x86_64-darwin"`
- host os: `Darwin 21.6.0, macOS 10.16`
- multi-user?: `yes`
- sandbox: `no`
- version: `nix-env (Nix) 2.13.3`
- channels(auguste): `"darwin, home-manager"`
- nixpkgs: `/nix/store/q336zlh9g3f5k0vabn6xzwhxasnhk7hv-source`

ncfavier commented 1 year ago

Looks like the same error as in https://github.com/NixOS/nixpkgs/issues/231291

ncfavier commented 1 year ago

cc @NixOS/darwin-maintainers

uri-canva commented 1 year ago

My first thought is that it might be using an older sdk on x86_64-darwin compared to aarch64-darwin, but first I'm going to let a bisect run overnight on my x86_64-darwin laptop, it's quite old so it's going to take a while.

toonn commented 1 year ago

This does look like an SDK issue. The annotation states it's only available from 10.16, which is 11.

Try calling the package with darwin.apple_sdk_11_0.callPackage in top-level.

ncfavier commented 1 year ago

That's already the case:

  wezterm = darwin.apple_sdk_11_0.callPackage ../applications/terminal-emulators/wezterm {
    inherit (darwin.apple_sdk_11_0.frameworks) Cocoa CoreGraphics Foundation UserNotifications System;
  };

emilazy commented 1 year ago

Apparently this might be a toolchain bug?

Some stuff I dug up:

I'm not sure if we're running into the exact same thing or what that thing even is though. It looks like __darwin_check_fd_set_overflow is only available on macOS 11 and above and should be in libSystem. We could try -Wl,-U,___darwin_check_fd_set_overflow at the risk of breaking things. Or maybe we need to set -mmacosx-version-min/MACOSX_DEPLOYMENT_TARGET (but I guess darwin.apple_sdk_11_0.callPackage might already handle that)?

emilazy commented 1 year ago

See https://github.com/mono/mono/issues/19393#issuecomment-762830412 in particular.

toonn commented 1 year ago

Hmm, puzzling. Maybe similar to #233815? The 10.12 Libsystem being brought in somehow?

I don't think the deployment target should matter. If it's older than 11 the symbol should be linked weakly.

toonn commented 1 year ago

The mono issue seems unrelated. They are/were building with -no_weak_imports, which means the build fails with an older deployment target because a weak reference is necessary but not allowed.

emilazy commented 1 year ago

Hmm, puzzling. Maybe similar to #233815? The 10.12 Libsystem being brought in somehow?

That does kind of seem like what https://github.com/NixOS/nixpkgs/pull/233136 may have caused?

Edit: But that was trying to fix the same error, so probably not the root cause.

emilazy commented 1 year ago

FWIW, it seems to build and run fine with this diff:

diff --git a/pkgs/applications/terminal-emulators/wezterm/default.nix b/pkgs/applications/terminal-emulators/wezterm/default.nix
index 9743197d832..36a8bbc8e05 100644
--- a/pkgs/applications/terminal-emulators/wezterm/default.nix
+++ b/pkgs/applications/terminal-emulators/wezterm/default.nix
@@ -88,7 +88,7 @@ rustPlatform.buildRustPackage rec {

   buildFeatures = [ "distro-defaults" ];

-  env.NIX_LDFLAGS = lib.optionalString stdenv.isDarwin "-framework System";
+  env.NIX_LDFLAGS = lib.optionalString stdenv.isDarwin "-U ___darwin_check_fd_set_overflow";

   postInstall = ''
     mkdir -p $out/nix-support

But of course that could just be masking an underlying problem. (Though the -framework System feels like that already.)

toonn commented 1 year ago

 -U symbol_name
                 Specified that it is ok for symbol_name to have no defini-
                 tion.  With -two_levelnamespace, the resulting symbol will be
                 marked dynamic_lookup which means dyld will search all loaded
                 images.

-two_levelnamespace is the default so unless -flat_namespace is passed somewhere this should result in the dynamic lookup mentioned and I think that would work on any x86_64-darwin running macOS 11 or newer.

I do believe this means SDK 10.12 is actually being used though so a real solution would require further digging.

emilazy commented 1 year ago

Do we have any other Rust packages that are using a newer SDK and that we know are definitely linking with it properly? It could be that the hooks for that stuff don't work with Rust builds.

ncfavier commented 1 year ago

$ (builtin cd pkgs/top-level; rg -l buildRust $(rg -Po '(?<=apple_sdk_11_0\.callPackage )[^ {]+' all-packages.nix) | sort)
../applications/blockchains/lighthouse/default.nix
../applications/misc/inlyne/default.nix
../applications/misc/pueue/default.nix
../applications/misc/stork/default.nix
../applications/terminal-emulators/wezterm/default.nix
../applications/version-management/delta/default.nix
../applications/version-management/git-trim/default.nix
../development/compilers/hvm/default.nix
../development/compilers/kind2/default.nix
../development/tools/rbspy/default.nix
../development/tools/rcodesign/default.nix
../development/tools/rust/cargo-watch/default.nix
../development/tools/wizer/default.nix
../servers/search/qdrant/default.nix
../shells/nushell/default.nix
../shells/nushell/plugins/query.nix
../tools/admin/procs/default.nix
../tools/backup/bupstash/default.nix
../tools/filesystems/httm/default.nix
../tools/misc/fclones/gui.nix
../tools/misc/killport/default.nix
../tools/misc/toastify/default.nix
../tools/security/age-plugin-yubikey/default.nix
../tools/system/bottom/default.nix

uri-canva commented 1 year ago

5 steps left on my bisect, will probably take another day.

uri-canva commented 1 year ago

Bisect finished, it's 1073ea7fa621b06f0131e72ec59d9c5a71fad8f2, PR: https://github.com/NixOS/nixpkgs/pull/235250.

emilazy commented 1 year ago

Very strange. I didn't know we even used libxcrypt on Darwin.

uri-canva commented 1 year ago

I think I did the bisect wrong, it doesn't make much sense, libxcrypt is just a dependency of python and perl, and while they're used for building wezterm, they're not linked into it as far as I can tell.

uri-canva commented 1 year ago

Hmm, no, it does get a cache hit on 3e6aebfcd2b. Maybe it's a difference between hydra and my laptop? Will try to build it on my laptop.

emilazy commented 1 year ago

My understanding is that git bisect can only be relied on to give fully accurate results when there is a commit such that all commits before that one pass the test and all commits after fail it. For stdenv/staging type changes that can have unrelated breakage in the middle and hide or expose problems created by other commits, that seems like an unsafe assumption. But I'd be very interested if it does somehow end up being libxcrypt; I looked at the nixpkgs PR and the upstream diff and can't fathom how it would have broken things :)

uri-canva commented 1 year ago

Yeah at first I thought maybe the error was different, but no I got the same linking error.

uri-canva commented 1 year ago

Maybe it's stdenv -> openssh -> openssl -> perl -> libxcrypt. I just noticed the linking error comes from liblibssh-rs-sys, the -sys means it links against the system library, as opposed to using the vendored sources right? But I would think the perl part of the dependency link is a nativeBuildInputs not a buildInputs.

uri-canva commented 1 year ago

Ok the build on https://github.com/NixOS/nixpkgs/commit/3e6aebfcd2b7e8de9052fb90c2cf0a02e47e1eff succeeds (though it says the outputs differ).

uri-canva commented 1 year ago

I tried reverting the commit on current master, dd05e2a, and it didn't fix the build... I don't know what's going on 😂 .

cpick commented 7 months ago

I've reduced this down to a small repo that reproduces the error. It essentially boils down to:

flake.nix:

{
  inputs = {
    nixpkgs.url = "nixpkgs/4471857c0a4a8a0ffc7bdbeaf1b998746ce12a82";
  };

  outputs = { self, nixpkgs }:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-darwin;
    in {

      packages.x86_64-darwin.default = pkgs.darwin.apple_sdk_11_0.callPackage (
        { rustPlatform, perl, System }:
        rustPlatform.buildRustPackage {
          name = "wezterm-build-error";
          src = ./.;
          cargoLock = { lockFile = ./Cargo.lock; };
          nativeBuildInputs = [ perl ];
          buildInputs = [ System ];
        }
      ) {
        inherit (pkgs.darwin.apple_sdk_11_0.frameworks) System;
      };

    };
}

Cargo.toml:

[package]
name = "wezterm-build-error"
version = "0.1.0"
edition = "2021"

[dependencies]
openssl = { version = "0.10", features=["vendored"] }
git2 = { version = "0.16", default-features = false, features = ["https"] }

The openssl crate's "vendored" feature seems to be needed to get its openssl-sys dependency to build.
The git2 crate's "https" features appears to be needed to trigger this issue's stransport.c build failure. My hunch is that this is because it causes git2's libgit2-sys dependency to itself also depend on the openssl-sys crate.

I based this off of the 4471857c0a4a nixpkgs commit somewhat arbitrarily as the current nixpkgs-unstable branch when I started looking into the issue.

I'm new to Nix so it's been a bit slow-going to get to this point. I wanted to share my progress in case this causes someone else to immediately understand the underlying issue.

cpick commented 7 months ago

I believe the git2/libgit2-sys/openssl/openssl-sys crates are all red-herrings.

I reduced my test repo down so that it just uses the cc crate to attempt compile a compile a file that only includes CoreFoundation.h and that fails with the same error.

flake.nix:

{
  inputs = {
    nixpkgs.url = "nixpkgs/4471857c0a4a8a0ffc7bdbeaf1b998746ce12a82";
  };

  outputs = { self, nixpkgs }:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-darwin;
    in {

      packages.x86_64-darwin.default = pkgs.darwin.apple_sdk_11_0.callPackage (
        { rustPlatform }:
        rustPlatform.buildRustPackage {
          name = "wezterm-build-error";
          src = ./.;
          cargoLock = { lockFile = ./Cargo.lock; };
        }
      ) {};

    };
}

Cargo.toml:

[package]
name = "wezterm-build-error"
version = "0.1.0"
build = "build.rs"
edition = "2021"

[build-dependencies]
cc = "1.0.43"

build.rs:

fn main() {
    cc::Build::new().file("inc.c").compile("inc");
}

inc.c:

#include <CoreFoundation/CoreFoundation.h>

Error from nix build:

@nix { "action": "setPhase", "phase": "unpackPhase" }
Running phase: unpackPhase
unpacking source archive /nix/store/c5r7x4fhhcb1gzx7f013p91kamsdarfz-gas5kd03z1fkcn2i1fra4j9qvyk392rd-source
source root is gas5kd03z1fkcn2i1fra4j9qvyk392rd-source
Executing cargoSetupPostUnpackHook
Finished cargoSetupPostUnpackHook
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: patchPhase
Executing cargoSetupPostPatchHook
Validating consistency between /private/tmp/nix-build-wezterm-build-error.drv-1/gas5kd03z1fkcn2i1fra4j9qvyk392rd-source/Cargo.lock and /private/tmp/nix-build-wezterm-build-error.drv-1/cargo-vendor-dir/Cargo.lock
Finished cargoSetupPostPatchHook
@nix { "action": "setPhase", "phase": "configurePhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "buildPhase" }
Running phase: buildPhase
Executing cargoBuildHook
++ env CC_X86_64_APPLE_DARWIN=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc CXX_X86_64_APPLE_DARWIN=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/c++ CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc CC_X86_64_APPLE_DARWIN=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc CXX_X86_64_APPLE_DARWIN=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/c++ CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc CARGO_BUILD_TARGET=x86_64-apple-darwin HOST_CC=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc HOST_CXX=/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/c++ cargo build -j 8 --target x86_64-apple-darwin --frozen --profile release
   Compiling libc v0.2.152
   Compiling cc v1.0.83
   Compiling wezterm-build-error v0.1.0 (/private/tmp/nix-build-wezterm-build-error.drv-1/gas5kd03z1fkcn2i1fra4j9qvyk392rd-source)
The following warnings were emitted during compilation:

warning: wezterm-build-error@0.1.0: In file included from inc.c:1:
warning: wezterm-build-error@0.1.0: In file included from /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:60:
warning: wezterm-build-error@0.1.0: In file included from /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFPropertyList.h:17:
warning: wezterm-build-error@0.1.0: In file included from /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFStream.h:16:
warning: wezterm-build-error@0.1.0: /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1189:83: error: expected ','
warning: wezterm-build-error@0.1.0:     kCFURLBookmarkCreationWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0))  API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 11 ), // Mac OS X 10.7.3 and later, include information in the bookmark data which allows the same sandboxed process to access the resource after being relaunched
warning: wezterm-build-error@0.1.0:                                                                                   ^
warning: wezterm-build-error@0.1.0: /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1190:98: error: expected ','
warning: wezterm-build-error@0.1.0:     kCFURLBookmarkCreationSecurityScopeAllowOnlyReadAccess API_AVAILABLE(macos(10.7), macCatalyst(13.0)) API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 12 ), // Mac OS X 10.7.3 and later, if used with kCFURLBookmarkCreationWithSecurityScope, at resolution time only read access to the resource will be granted
warning: wezterm-build-error@0.1.0:                                                                                                  ^
warning: wezterm-build-error@0.1.0: /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1200:85: error: expected ','
warning: wezterm-build-error@0.1.0:     kCFURLBookmarkResolutionWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0)) API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 10 ), // Mac OS X 10.7.3 and later, use the secure information included at creation time to provide the ability to access the resource in a sandboxed process.
warning: wezterm-build-error@0.1.0:                                                                                     ^
warning: wezterm-build-error@0.1.0: 3 errors generated.

error: failed to run custom build command for `wezterm-build-error v0.1.0 (/private/tmp/nix-build-wezterm-build-error.drv-1/gas5kd03z1fkcn2i1fra4j9qvyk392rd-source)`

Caused by:
  process didn't exit successfully: `/private/tmp/nix-build-wezterm-build-error.drv-1/gas5kd03z1fkcn2i1fra4j9qvyk392rd-source/target/release/build/wezterm-build-error-2b0a4d25197fb65f/build-script-build` (exit status: 1)
  --- stdout
  TARGET = Some("x86_64-apple-darwin")
  OPT_LEVEL = Some("3")
  HOST = Some("x86_64-apple-darwin")
  cargo:rerun-if-env-changed=CC_x86_64-apple-darwin
  CC_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CC_x86_64_apple_darwin
  CC_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = Some("/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc")
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("cmpxchg16b,fxsr,sse,sse2,sse3,sse4.1,ssse3")
  cargo:rerun-if-env-changed=CFLAGS_x86_64-apple-darwin
  CFLAGS_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_apple_darwin
  CFLAGS_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  running: env -u IPHONEOS_DEPLOYMENT_TARGET "/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-arch" "x86_64" "-Wall" "-Wextra" "-o" "/private/tmp/nix-build-wezterm-build-error.drv-1/gas5kd03z1fkcn2i1fra4j9qvyk392rd-source/target/x86_64-apple-darwin/release/build/wezterm-build-error-8a956244ef1895a7/out/inc.o" "-c" "inc.c"
  cargo:warning=In file included from inc.c:1:

  cargo:warning=In file included from /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:60:

  cargo:warning=In file included from /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFPropertyList.h:17:

  cargo:warning=In file included from /nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFStream.h:16:

  cargo:warning=/nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1189:83: error: expected ','

  cargo:warning=    kCFURLBookmarkCreationWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0))  API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 11 ), // Mac OS X 10.7.3 and later, include information in the bookmark data which allows the same sandboxed process to access the resource after being relaunched

  cargo:warning=                                                                                  ^

  cargo:warning=/nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1190:98: error: expected ','

  cargo:warning=    kCFURLBookmarkCreationSecurityScopeAllowOnlyReadAccess API_AVAILABLE(macos(10.7), macCatalyst(13.0)) API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 12 ), // Mac OS X 10.7.3 and later, if used with kCFURLBookmarkCreationWithSecurityScope, at resolution time only read access to the resource will be granted

  cargo:warning=                                                                                                 ^

  cargo:warning=/nix/store/lg50ki620x8j6s5nhrsmn40a1y6fin98-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1200:85: error: expected ','

  cargo:warning=    kCFURLBookmarkResolutionWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0)) API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 10 ), // Mac OS X 10.7.3 and later, use the secure information included at creation time to provide the ability to access the resource in a sandboxed process.

  cargo:warning=                                                                                    ^

  cargo:warning=3 errors generated.

  exit status: 1

  --- stderr

  error occurred: Command env -u IPHONEOS_DEPLOYMENT_TARGET "/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-arch" "x86_64" "-Wall" "-Wextra" "-o" "/private/tmp/nix-build-wezterm-build-error.drv-1/gas5kd03z1fkcn2i1fra4j9qvyk392rd-source/target/x86_64-apple-darwin/release/build/wezterm-build-error-8a956244ef1895a7/out/inc.o" "-c" "inc.c" with args "cc" did not execute successfully (status code exit status: 1).

This is leading me to believe that the cc crate is not working properly with apple_sdk_11_0.

cpick commented 7 months ago

Similar issues:

keybase go package with the original apple_sdk_11_0 patch. Reply by @reckenrode:

I still got some other compile errors with this patch, any idea what's going wrong? Maybe some c compiler version need to be bumped?

That usually happens when you have a 10.12 stdenv trying to build something that is using frameworks from the 11.0 SDK.

I was able to get cronosd to build, but it was hacky. I didn’t have much luck with the provided nix stuff and didn’t want to sink a lot of time into it, so I installed go_1_18, darwin.apple_sdk_11_0.stdenv.cc, and darwin.apple_sdk_11_0.MacOSX-SDK from my branch into my profile and used that. I had to set NIX_CFLAGS_COMPILE_x86_64_apple_darwin and NIX_LDFLAGS_x86_64_apple_darwin to point to the SDK in my profile, but I was able to build by running make. Like I said, hacky. Hopefully that’s close-ish to what you were trying to do.
musescore Discourse issue about another build and this GitHub reply also from @reckenrode:
The issue is portaudio is propagating frameworks from the 10.12 SDK. Mixing frameworks from different SDK versions doesn’t work. It results in errors because the headers aren’t compatible. You need to override portaudio to use the 11.0 SDK.
```
let
  # portaudio propagates Darwin frameworks. Rebuild it using the 11.0 stdenv from Qt and the 11.0 SDK frameworks.
  portaudio' = if !stdenv.isDarwin
    then portaudio
    else portaudio.override {
      inherit stdenv;
      inherit (darwin.apple_sdk_11_0.frameworks) AudioUnit AudioToolbox CoreAudio CoreServices Carbon;
    };
in
```
And this followup: ... I thought Qt would provide an 11.0 SDK one, but it’s apparently still the default 10.12 SDK stdenv. Using the 11.0 SDK stdenv directly allows it to build for me. I apologize for not doing more testing (aside from confirming the expression for portaudio). Here’s a diff with the changes I made to get it to build for x86_64-darwin. ...
This Fyrox rust failure seems quite similar. This was fixed by https://github.com/NixOS/nixpkgs/pull/233815 which looks like it's fixing just this kind of issue for bindgen, but doesn't appear to have helped in this case (which may make sense since the bindgen crate is nowhere to be found in my test repo's Cargo.lock, only cc and libc).
cargo-watch rust failure which looks like it was solved by reverting back to SDK 10.12 (which I don't think is an option for WezTerm).

The above issues lead me to believe that there's something else in rustPlatform.buildRustPackage that's not setting stdenv correctly?

I suppose at the very least I'm learning more and more about Nix internals and macOS SDKs :/

reckenrode commented 7 months ago

I’ll take a look at this more closely when I’m back home this evening. Most likely, it’s an issue with the stdenv used by buildRustPlatform. You have to override its stdenv. It’s not enough just overriding the one used by rustPlatform.

cpick commented 7 months ago

Revisiting the Fyrox/crane issue mentioned above the following comment also seems to fix my test repo:

thanks for the hint @reckenrode, adding libsystem to buildInputs fixes the issue for me. here are the nix files for the default template

default.nix
  ...
  buildInputs = [
    ...
    darwin.apple_sdk_11_0.Libsystem
  ];
  ...

@reckenrode's reply likely applies here as well:

It probably shouldn’t be necessary to manually specify Libsystem. That would be like having to provide glibc explictly on Linux.

I suspect there's a right way to fix this in rustPlatform, but in the meantime I'm going to see if I can make progress on WezTerm using a workaround like the above.

reckenrode commented 7 months ago

I looked at the rustPlatform issue. It’s picking up the wrong stdenv from rust.envVars.setEnv. The following patch makes it build, but I’m not sure it’s the correct solution.

diff --git a/pkgs/development/compilers/rust/make-rust-platform.nix b/pkgs/development/compilers/rust/make-rust-platform.nix
index e22cb6d594af..1caecec68793 100644
--- a/pkgs/development/compilers/rust/make-rust-platform.nix
+++ b/pkgs/development/compilers/rust/make-rust-platform.nix
@@ -1,9 +1,12 @@
-{ lib, buildPackages, callPackage, cargo-auditable, stdenv, runCommand }@prev:
+{ lib, buildPackages, callPackage, cargo-auditable, rust, stdenv, runCommand }@prev:
+
+lib.makeOverridable (

 { rustc
 , cargo
 , cargo-auditable ? prev.cargo-auditable
 , stdenv ? prev.stdenv
+, buildPackages ? prev.buildPackages
 , ...
 }:

@@ -36,5 +39,11 @@ rec {
   # Hooks
   inherit (callPackage ../../../build-support/rust/hooks {
     inherit stdenv cargo rustc;
+    rust = prev.rust.override ({
+      inherit stdenv;
+    } // lib.optionalAttrs stdenv.buildPlatform.isDarwin {
+      buildPackages = buildPackages // { inherit stdenv; };
+    });
   }) cargoBuildHook cargoCheckHook cargoInstallHook cargoNextestHook cargoSetupHook maturinBuildHook bindgenHook;
 }
+)
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index e2b5af7ee3e9..b697a577a5f6 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -3124,7 +3124,10 @@ with pkgs;

   wayst = callPackage ../applications/terminal-emulators/wayst { };

-  wezterm = darwin.apple_sdk_11_0.callPackage ../applications/terminal-emulators/wezterm {
+  wezterm = callPackage ../applications/terminal-emulators/wezterm {
+    rustPlatform = if stdenv.isDarwin
+      then rustPlatform.override { stdenv = overrideSDK stdenv "11.0"; }
+      else rustPlatform;
     inherit (darwin.apple_sdk_11_0.frameworks) Cocoa CoreGraphics Foundation UserNotifications System;
   };

cpick commented 7 months ago

I looked at the rustPlatform issue. It’s picking up the wrong stdenv from rust.envVars.setEnv. The following patch makes it build, but I’m not sure it’s the correct solution.

Thank you very much @reckenrode!

I've thrown together a branch that contains your above fix and it's working well for me locally. Do you have a thought on what may be a good way to try to get it (or something like it) merged?

Dead ends

Some other dead ends I'd investigated in the last two days:

Add Libsystem to buildInputs as mentioned above. This fixed the immediate build issue, but subsequent build errors popped up when it moved on to building the harfbuzz dependency:

warning: harfbuzz@0.1.0: /nix/store/2dvpdf87yq40jdl6drbvqamszz85rb69-libcxx-16.0.6-dev/include/c++/v1/cfloat:78:5: error: <cfloat> tried including <float.h> but didn't find libc++'s <float.h> header.           This usually means that your header search paths are not configured properly.           The header search paths should contain the C++ Standard Library headers before           any C Standard Library, and you are probably using compiler flags that make that           not be the case.

Presumably these could be fixed individually, but I took it as a strong indication that I was barking up the wrong tree.

Use system libraries so that the '*-sys' crates don't have to build their own. Again, this fixed the immediate build issue, but generated a link error:

error: linking with `/nix/store/4mlkc25c87bl6yr4dvdklx2dy5nfj1fv-clang-wrapper-16.0.6/bin/cc` failed: exit status: 1
  |
  = note: env -u IPHONEOS_DEPLOYMENT_TARGET -u TVOS_DEPLOYMENT_TARGET LC_ALL="C"     PATH="/nix/store/brixixymy053gwrwlhji77baxzkhd3b4-r>
  = note: Undefined symbols for architecture x86_64:
            "___darwin_check_fd_set_overflow", referenced from:
                _ssh_poll in liblibssh_rs_sys-63fdb3ba541a1ed7.rlib(poll.o)
          ld: symbol(s) not found for architecture x86_64
          clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

Using a complicated set of feature arguments I think I could force libssh-rs to not vendor libssh, but likewise, feels like I'm going down the wrong path.

Ultimately, it would make sense if the whole reason the wezterm package was bumped to SDK 11.0 is because something it depends on needs it and these piecemeal workarounds aren't going to cut it.

reckenrode commented 7 months ago

I've thrown together a branch that contains your above fix and it's working well for me locally. Do you have a thought on what may be a good way to try to get it (or something like it) merged?

The buildPackages part in rustPlatform is probably wrong, but I don’t know what a good solution is. I reached out on NixOS contributors channel on Matrix to see if there’s an appropriate team or maintainers who might be able to weigh in.

Ultimately, it would make sense if the whole reason the wezterm package was bumped to SDK 11.0 is because something it depends on needs it and these piecemeal workarounds aren't going to cut it.

It needs the UserNotifications framework, which requires the 10.14 SDK or newer. aarch64-darwin satisfies that by default, but x86_64-darwin doesn’t.

cpick commented 5 months ago

@reckenrode did you hear back from any appropriate maintainers? I believe the patch you'd developed above was broken by this commit which changed a call from callPackage to callPackages for https://github.com/NixOS/nixpkgs/pull/281192:

error: anonymous function at /nix/store/kk826ndl1mkjwb5nyy01ig42aynxwiwy-source/pkgs/development/compilers/rust/1_76.nix:12:1 called with unexpected argument 'buildPackages'

       at /nix/store/kk826ndl1mkjwb5nyy01ig42aynxwiwy-source/lib/customisation.nix:98:16:

           97|     let
           98|       result = f origArgs;
             |                ^
           99|
(use '--show-trace' to show detailed location information)

I've been flailing around a bit to try to fix your patch, but am in pretty deep over my head with the customization/splice/callPackage* code and so haven't made much progress.

reckenrode commented 5 months ago

@cpick https://github.com/NixOS/nixpkgs/pull/287609 was the first step in a more general solution. The next step is a PR to add a way to override the SDK used by rustPlatform, which is also needed by a few other Rust-based applications (lapce, neovide).

I’m not sure yet what that will look like. Most likely, something like rustPlatform.overrideSDK or rustPlatform.override { stden = … }.

SuperSandro2000 commented 5 months ago

I think having a general way to override rustPlatform is better for future changes similar to this for other components unless overwriting the SDK is so involved that it needs a higher level function.

reckenrode commented 3 months ago

I think having a general way to override rustPlatform is better for future changes similar to this for other components unless overwriting the SDK is so involved that it needs a higher level function.

It actually is fairly involved. overrideSDK needs to walk all propagated inputs looking for SDK packages to replace them with the version for the requested SDK. If a package hides an input (commonly by referencing it directly in a hook), then overrideSDK has no way of knowing the dependency is there. That’s the problem with rustPlatform.

I don’t think ecosystem-specific workarounds (such as making rustPlatform overridable and play nicely with overrideSDK) are sustainable, which is why I want to try making Darwin behave (as part of the SDK refactor for 24.11) more like a native toolchain using SDKROOT to find the SDK in nixpkgs, but I need to land https://github.com/NixOS/nixpkgs/pull/307880 and https://github.com/NixOS/nixpkgs/pull/256590 first before I can start on that.

alienman5k commented 3 months ago

Would it be a valid option to have another package for Wezterm to use the binary released instead of building using rust so we can pick one or the other (I know one can build it's own derivation for that, but just asking if this could be done at the Nix packages level?)

nixos-discourse commented 3 weeks ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/on-the-future-of-darwin-sdks-or-how-you-can-stop-worrying-and-put-the-sdk-in-build-inputs/50574/3

reckenrode commented 3 weeks ago

Per the above Discourse mention, I tried to build Wezterm on x86_64-darwin. Assuming I would need to determine which SDK I needed to use based on how the build failed, I built it with the default (10.12) SDK. ~~It built. So yay? 🤷🏻‍♂️~~

Update: I just checked all-packages.nix. It’s using apple_sdk_11_0.callPackage, which is changed to add the 11.3 SDK for you to your build inputs. That’s why it built. It successfully used the 11.0 SDK.

The fonts are all messed up, but I don’t know whether I need to do something to make them available.

NixOS / nixpkgs