search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.31k stars 14.29k forks source link

Build failure: ollama (aarch64-darwin) #345333

Open n8henrie opened 1 month ago

n8henrie commented 1 month ago

Steps To Reproduce

Steps to reproduce the behavior:

$ nix build github:nixos/nixpkgs/4d2f662676c79b6e30ec9e8f7a41236d5f883687#ollama

Build log

https://gist.github.com/n8henrie/0b2581aa8ede32fd2e4141c5d2473f66

Notify maintainers

@abysssol @dit7ya @elohmeier @roydubnium

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 24.0.0, macOS 15.0`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.8`
 - channels(n8henrie): `""`
 - channels(root): `""`
 - nixpkgs: `/nix/store/x2qbrd60nz1y86ha15k7mkc2i1d7xb37-source`

Add a :+1: reaction to issues you find important.

winterqt commented 1 month ago

Can you try building with --option sandbox false (or whatever the nix3 equivalent is)?

n8henrie commented 1 month ago

@winterqt -- yes, looks like a sandbox issue.

n8henrie commented 1 month ago

Looks like the error is coming from this ObjC function: https://github.com/ollama/ollama/blob/79d3b1e2bdfc97542a7259b0c839520d39578514/gpu/gpu_info_darwin.m#L5

Not sure why that would be a problem with the sandbox though.

n8henrie commented 1 month ago

Repro:

{ stdenv, darwin }:
stdenv.mkDerivation {
  name = "foo";
  version = "0.0.1";
  src = ./.;
  buildInputs = with darwin.apple_sdk.frameworks; [
    Foundation
    Metal
  ];
  installPhase = ''
    mkdir -p $out/bin
    cp foo $out/bin/
  '';
  doCheck = true;
  checkPhase = "./foo";
}

foo.h:

#import <Metal/Metal.h>
#include <stdint.h>
uint64_t getRecommendedMaxVRAM();

foo.m

#import <Foundation/Foundation.h>
#import <mach/mach.h>
#include "foo.h"

uint64_t getRecommendedMaxVRAM() {
  id<MTLDevice> device = MTLCreateSystemDefaultDevice();
  uint64_t result = device.recommendedMaxWorkingSetSize;
  CFRelease(device);
  return result;
}

int main() {
  uint64 max = getRecommendedMaxVRAM();
  NSLog(@"max vram: %llu", max);
  return 0;
}

Makefile

foo: foo.h foo.m
    clang -framework Metal -framework Foundation foo.m -o foo

Fails with sandbox:

$ nix build --print-build-logs
foo> Running phase: unpackPhase
foo> unpacking source archive /nix/store/l9jk1d2ajicrxqpyh6apqzizplil8xrs-8qpwv9w9yf5p5skhxfjh8bkxhyny25p0-source
foo> source root is 8qpwv9w9yf5p5skhxfjh8bkxhyny25p0-source
foo> Running phase: patchPhase
foo> Running phase: updateAutotoolsGnuConfigScriptsPhase
foo> Running phase: configurePhase
foo> no configure script, doing nothing
foo> Running phase: buildPhase
foo> build flags: SHELL=/nix/store/iafzjk7zbkqaszqfp6n006vvxjrpn4f6-bash-5.2p32/bin/bash
foo> clang -framework Metal -framework Foundation foo.m -o foo
foo> Running phase: checkPhase
foo> /nix/store/gayh4jw7jl2dbmi4lza08ljh9irjzix5-stdenv-darwin/setup: line 1717: 89993 Trace/BPT trap: 5       ./foo
error: builder for '/nix/store/vdhj49pwwh1givc9b30y87jag2d6p7c2-foo.drv' failed with exit code 133;
       last 10 log lines:
       > source root is 8qpwv9w9yf5p5skhxfjh8bkxhyny25p0-source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > no configure script, doing nothing
       > Running phase: buildPhase
       > build flags: SHELL=/nix/store/iafzjk7zbkqaszqfp6n006vvxjrpn4f6-bash-5.2p32/bin/bash
       > clang -framework Metal -framework Foundation foo.m -o foo
       > Running phase: checkPhase
       > /nix/store/gayh4jw7jl2dbmi4lza08ljh9irjzix5-stdenv-darwin/setup: line 1717: 89993 Trace/BPT trap: 5       ./foo
       For full logs, run 'nix log /nix/store/vdhj49pwwh1givc9b30y87jag2d6p7c2-foo.drv'.

Works without sandbox:

$ nix build --print-build-logs --no-sandbox
foo> Running phase: unpackPhase
foo> unpacking source archive /nix/store/l9jk1d2ajicrxqpyh6apqzizplil8xrs-8qpwv9w9yf5p5skhxfjh8bkxhyny25p0-source
foo> source root is 8qpwv9w9yf5p5skhxfjh8bkxhyny25p0-source
foo> Running phase: patchPhase
foo> Running phase: updateAutotoolsGnuConfigScriptsPhase
foo> Running phase: configurePhase
foo> no configure script, doing nothing
foo> Running phase: buildPhase
foo> build flags: SHELL=/nix/store/iafzjk7zbkqaszqfp6n006vvxjrpn4f6-bash-5.2p32/bin/bash
foo> clang -framework Metal -framework Foundation foo.m -o foo
foo> Running phase: checkPhase
foo> 2024-09-30 18:58:08.056 foo[90151:2128039] max vram: 11453251584
foo> Running phase: installPhase
foo> Running phase: fixupPhase
foo> checking for references to /private/tmp/nix-build-foo.drv-0/ in /nix/store/vr41fdkkc2bhxb20w5cx7bkdsbw1ib36-foo...
foo> patching script interpreter paths in /nix/store/vr41fdkkc2bhxb20w5cx7bkdsbw1ib36-foo
foo> stripping (with command strip and flags -S) in  /nix/store/vr41fdkkc2bhxb20w5cx7bkdsbw1ib36-foo/bin
winterqt commented 1 month ago

Thank you! I'll prod at the sandbox profiles a bit, maybe we can set something to get it working.

n8henrie commented 1 month ago

Not sure if this is helpful, I've not yet had luck adding any of these to extra-sandbox-paths:

$ log stream --level Error | awk '$4 == "Error" && $0 ~ "ollama-sandbox" { for (idx=1; idx<4; idx++) { $idx="" }; print $0 }'
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /System/Volumes/Preboot/Cryptexes/OS
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-metadata /System/Volumes/Preboot/Cryptexes/OS/System/Library/dyld/dyld_shared_cache_arm64e
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.system.notification_center
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.logd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /Library/Preferences/Logging/com.apple.diagnosticd.filter.plist
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 1 duplicate report for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /dev/autofs_nowait
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /dev/autofs_nowait
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-metadata /usr/libexec/swift
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-metadata /usr/bin
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) iokit-open-user-client IOSurfaceRootUserClient
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /Library/Preferences/Logging/com.apple.diagnosticd.filter.plist
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /System/Library/Preferences/Logging/Subsystems/com.apple.defaults.plist
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /System/Library/Preferences/Logging/Subsystems/com.apple.defaults.plist
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 7 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-metadata /private/var/empty
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 9 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /System/Library/Preferences/Logging/Subsystems/com.apple.xpc.plist
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 9 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /Library/Preferences/.GlobalPreferences.plist
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 27 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.system.opendirectoryd.membership
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.bsd.dirhelper
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 1 duplicate report for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) iokit-open-user-client AGXDeviceUserClient
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 2 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) file-read-data /System/Library/Preferences/Logging/Subsystems/com.apple.display.reconfig.plist
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 2 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.tccd.system
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 4 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.tccd.system
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 4 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.windowserver.active
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 3 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.windowserver.active
   Error 0x0 0 0 kernel: (Sandbox) Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
   Error 0x0 0 0 kernel: (Sandbox) 5 duplicate reports for Sandbox: ollama-sandbox(4615) deny(1) mach-lookup com.apple.diagnosticd
winterqt commented 1 month ago

Very helpful! It's probably the iokit-open-user-clients. I'll make a profile for it later today, unless you beat me to it.

n8henrie commented 1 month ago

Nope, I'm headed to work for the evening and don't know how to do that -- happy to learn from your example though, thanks!

n8henrie commented 1 month ago

@winterqt I'm still very interested to see what you mean by making a profile for this if you find time (still an issue, at least on unstable).