Initial working patch to build arm64 Nim binaries on M1 macs

[heinthanth]

The patch is quite simple.

choosenim use proxyexe that is statically compiled and statically read and is embedded in choosenim like this:

static: compileProxyexe()

const
  proxyExe = staticRead("proxyexe".addFileExt(ExeExt))

Instead of statically compiled proxyexe, I patched the compileProxyexe proc to compile proxyexe on user machine with embedded source code and previously compiled Nim like this:

const proxyExeSources = {
  "src" / "choosenimpkg" / "proxyexe.nim": staticRead("./proxyexe.nim"),
  "src" / "choosenimpkg" / "proxyexe.nims": staticRead("./proxyexe.nims"),
  "src" / "choosenimpkg" / "common.nim": staticRead("./common.nim"),
  "src" / "choosenimpkg" / "cliparams.nim": staticRead("./cliparams.nim"),
  "choosenim.nimble": staticRead("../../choosenim.nimble")
}.toTable()

I mean I embedded source codes required by proxyexe on choosenim binary. Then, I compiled them with previously compiled arm64 nim and nimble binaries ( choosenim compile nim first, then write proxy ).

Yeah, this is the main trick. And also on M1 macs with rosetta, I appended arch -arm64 flags on most commands to run as arm process like this:

if defined(macosx) and isRosetta():
  command = "arch -arm64 " & command

You can mark this PR as draft. On my mac, it works fine. How about others?

[elcritch]

I cloned this PR and rebuilt choosenim. That seems fine so far.

Running choosenim devel resulted in this:

$ choosenim devel         
Downloading Nim latest-devel from GitHub
[##################################################] 100.0% 0kb/s
 Extracting macosx_x64.tar.xz
 Extracting macosx_x64.tar
    Setting up git repository
   Building Nim #devel
  Compiler: Already built
  Building: ProxyExe
 Exception: switcher.nim(46, 12) `exitCode == 0` ("arch: posix_spawnp: /Users/jaremycreechley/.choosenim/toolchains/nim-#devel/bin/nimble: Bad CPU type in executable\n", "arch -arm64 /Users/jaremycreechley/.choosenim/toolchains/nim-#devel/bin/nimble --nim:\'/Users/jaremycreechley/.choosenim/toolchains/nim-#devel/bin/nim\' c -d:release /var/folders/dl/98d12czj5_b9lyfvzykftzt80000gn/T/proxyexe/src/choosenimpkg/proxyexe.nim")
   Cleaning failed install of #devel
       Tip: 9 messages have been suppressed, use --verbose to show them.
     Error: Installation failed

It seems to work with choosenim 1.4.4:

$ choosenim 1.4.4 
Downloading Nim 1.4.4 from nim-lang.org
[##################################################] 100.0% 0kb/s
 Extracting nim-1.4.4.tar.xz
 Extracting nim-1.4.4.tar
   Building Nim 1.4.4
   Building koch
   Building Nim
   Building tools (nimble, nimgrep, nimpretty, nimsuggest, testament)
  Building: ProxyExe
  Installed component 'nim'
    Prompt: Symlink for 'nimble' detected in '/Users/jaremycreechley/.nimble/bin'. Can I remove it? [y/N]
    Answer: y
    Removed symlink pointing to ../pkgs/nimble-#8f7af86/nimble
  Installed component 'nimble'
  Installed component 'nimgrep'
  Installed component 'nimpretty'
  Installed component 'nimsuggest'
  Installed component 'testament'
  Installed component 'nim-gdb'
   Switched to Nim 1.4.4

# jaremycreechley @ Jaremys-MacBook-Air in ~/projs/nims/third-party/choosenim on git:master o [22:18:28] 
$ nim -v
Nim Compiler Version 1.4.4 [MacOSX: arm64]
Compiled at 2022-07-08
Copyright (c) 2006-2020 by Andreas Rumpf

active boot switches: -d:release

However running choosenim 1.6.4 (I'm on 1.6.6):

$ choosenim 1.6.4
Downloading Nim 1.6.4 from nim-lang.org
[##################################################] 100.0% 0kb/s
 Extracting nim-1.6.4.tar.xz
 Extracting nim-1.6.4.tar
   Building Nim 1.6.4
   Building koch
   Building Nim
   Building tools (nimble, nimgrep, nimpretty, nimsuggest, testament)
  Building: ProxyExe
  Installed component 'nim'
  Installed component 'nimble'
  Installed component 'nimgrep'
  Installed component 'nimpretty'
  Installed component 'nimsuggest'
  Installed component 'testament'
  Installed component 'nim-gdb'
   Switched to Nim 1.6.4

# jaremycreechley @ Jaremys-MacBook-Air in ~/projs/nims/third-party/choosenim on git:master o [22:23:51] 
$ nim -v 
[1]    92907 killed     nim -v

Here's trying to re-install 1.6.6:

$ choosenim 1.6.6
  Building: ProxyExe
 Exception: switcher.nim(46, 12) `exitCode == 0` ("arch: posix_spawnp: /Users/jaremycreechley/.choosenim/toolchains/nim-1.6.6/bin/nimble: Bad CPU type in executable\n", "arch -arm64 /Users/jaremycreechley/.choosenim/toolchains/nim-1.6.6/bin/nimble --nim:\'/Users/jaremycreechley/.choosenim/toolchains/nim-1.6.6/bin/nim\' c -d:release /var/folders/dl/98d12czj5_b9lyfvzykftzt80000gn/T/proxyexe/src/choosenimpkg/proxyexe.nim")
     Error: Installation failed

Lastly:

$ file $(which nim)
/Users/jaremycreechley/.nimble/bin/nim: Mach-O 64-bit executable arm64

[elcritch]

P.S. I'm on a MacOS Monterey 12.0.1 (21A559) on a M1 MacBook Air.

[heinthanth]

I see ... let me try those versions.

Edit: I assumed choosenim to download nim sources. Instead, choosenim download specific x64 nim for those versions. I gonna check that out.

Here're differences:

nim devel: macosx_x64.tar.xz

Downloading Nim latest-devel from GitHub
[##################################################] 100.0% 0kb/s
 Extracting macosx_x64.tar.xz

nim 1.4.4: nim-1.4.4.tar.xz

Downloading Nim 1.4.4 from nim-lang.org
[##################################################] 100.0% 0kb/s
 Extracting nim-1.4.4.tar.xz

Like that.

For 1.6.6, I think you already have x64 Nim binaries.

[elcritch]

It seems mixing up the sources, or partial builds breaks things. I did a fresh install on choosenim, then built this PR. That route was able to download and build Nim 1.6.4 which runs fine.

[elcritch]

Though once I have an arm64 build of 1.6.4 when I try remove and rebuild 1.6.6 I still get a bad compiler build:

# jaremycreechley @ Jaremys-MacBook-Air in ~/projs/nims/third-party/choosenim on git:master o [23:05:15] 
$ ./bin/choosenim remove 1.6.6                         
      Info: Removed version 1.6.6

# jaremycreechley @ Jaremys-MacBook-Air in ~/projs/nims/third-party/choosenim on git:master o [23:05:42] 
$ ./bin/choosenim 1.6.6       
Downloading Nim 1.6.6 from nim-lang.org
[##################################################] 100.0% 0kb/s
 Extracting nim-1.6.6.tar.xz
 Extracting nim-1.6.6.tar
   Building Nim 1.6.6
   Building koch
   Building Nim
   Building tools (nimble, nimgrep, nimpretty, nimsuggest, testament)
  Building: ProxyExe
  Installed component 'nim'
  Installed component 'nimble'
  Installed component 'nimgrep'
  Installed component 'nimpretty'
  Installed component 'nimsuggest'
  Installed component 'testament'
  Installed component 'nim-gdb'
   Switched to Nim 1.6.6

# jaremycreechley @ Jaremys-MacBook-Air in ~/projs/nims/third-party/choosenim on git:master o [23:07:32] 
$ nim -v 
[1]    7211 killed     nim -v

Binary seems like it's an arm64:

$ file $(which nim)      
/Users/jaremycreechley/.nimble/bin/nim: Mach-O 64-bit executable arm64

[heinthanth]

I'm figuring out about why process is killed and I found something like this: "code signing rejecting invalid page at address" So, now, I'm rebuilding ProxyExe when a version is selected. That killed problem should be fine with this commit ( coming soon )

[elcritch]

Great, and yah that follows what I'm seeing with ProxeExe. I re-installed choosenim and then my Nim 1.6.6 arm64 build worked. The ProxyExe seemed to be breaking between choosenim installs.

[heinthanth]

@elcritch You can try again now. I think it should fix killed problem. For the bug that choosenim downloading x64.tar.xz, we need to figure out download.nim.

As far as I tested, I can switch between 1.4.4, 1.6.4, 1.6.6 fine!

[heinthanth]

But for the case that choosenim downloading macos_x64.tar.xz, I think I need @dom96 's help 😁.

[elcritch]

I'm re-testing your updates. I'll let you know how it goes.

[elcritch]

Ok did some testing. It works when using a default choosenim and then use this PR to install a new Nim version.

However when I do a choosenim stable --firstInstall after deleting my ~/.nimble and ~/.choosenim folders. It builds the compiler fine but then gets to this error:

 Executing: arch -arm64 /Users/jaremycreechley/.choosenim/toolchains/nim-1.6.6/bin/nimble --nim:'/Users/jaremycreechley/.choosenim/toolchains/nim-1.6.6/bin/nim' c -d:release /var/folders/dl/98d12czj5_b9lyfvzykftzt80000gn/T/proxyexe/src/choosenimpkg/proxyexe.nim
 Exception: switcher.nim(46, 12) `exitCode == 0` ("  Verifying dependencies for choosenim@0.8.4\n    Prompt: No local packages.json found, download it from internet? [y/N]\nio.nim(156)              raiseEOF\nError: unhandled exception: EOF reached [EOFError]\n    Answer: \n", "arch -arm64 /Users/jaremycreechley/.choosenim/toolchains/nim-1.6.6/bin/nimble --nim:\'/Users/jaremycreechley/.choosenim/toolchains/nim-1.6.6/bin/nim\' c -d:release /var/folders/dl/98d12czj5_b9lyfvzykftzt80000gn/T/proxyexe/src/choosenimpkg/proxyexe.nim")
   Cleaning failed install of 1.6.6
     Debug: Reporting to analytics...
     Debug: Reporting to analytics...
     Debug: Reporting to analytics...
     Error: Installation failed

[heinthanth]

Yeah ... I forgot to add -y in nimble while compiling proxies. For now, I've tested and OK!

[elcritch]

Nice! I added the -y locally and it works for me as well. But is the Symlink for 'nimble' detected... expected? It seems for --firstInstall it should override it?

[elcritch]

Maybe you could add if symlinkExists(proxyPath) and not params.firstInstall: in switcher.nim:160?

[elcritch]

Adding the if symlinkExists(proxyPath) and not params.firstInstall fixes that too. Excellent!

Edit: also just tested this on a linux machine of mine. Works fine.

[elcritch]

Oh, the compiler runs much faster when it's not using Rosetta! ❤️

[heinthanth]

Yes, we can override nimble in firstInstall. Because it's from proxyexe.nimble deps! Cool, thanks for info!

Here:

[heinthanth]

Huge thanks @elcritch. We can build choosenim universal binary after this by tweaking build.sh since we separated proxyExe from choosenim 🚀

[elcritch]

Glad to help!

[heinthanth]

@dom96 We made progress so far 😁 What do you think?

[heinthanth]

The proxyexe is now compiled at choosenim runtime. Can we do it at compile-time as it was done before? I'm concerned that performing this at runtime will be too unreliable.

We can pre-compile proxies as usual. But we need to build choosenim binaries ( amd64 and arm64 ) because proxies must be arm64 on M1 macs and amd64 on amd64. Or we can build universal proxies I think.

In general I'm wondering: do we need proxyexe and choosenim to be compiled as arm64? Can't we just keep the changes in utils.nim and have choosenim make an arm64 Nim build with just these changes?

Here's the case:

1. Let compiler.nim be nim compiler itself ( I think nim internal work something like this ). This invokes C compiler and compiles test.c.

import os, osproc, strutils

let (output, exitCode) = execCmdEx("cc -o $# test.c" % "test".addFileExt(ExeExt))
assert exitCode == 0, output

And it's compiled in arm64.

[ heinthant@macbookpro ] codes/heinthant/nim-demo $ file compiler           
compiler: Mach-O 64-bit executable arm64

2. Let proxies.nim be proxies from choosenim. This starts compiler process.

import os, osproc

const nimBin = "compiler".addFileExt(ExeExt)

let nimProcess = startProcess(nimBin)
let exitCode = nimProcess.waitForExit()
nimProcess.close()

assert exitCode == 0

This is compiled in x86_64.

[ heinthant@macbookpro ] codes/heinthant/nim-demo $ file proxies
proxies: Mach-O 64-bit executable x86_64

The strange case is:

1. When I direct-invoke compiler, the output from test.c is arm64.

[ heinthant@macbookpro ] codes/heinthant/nim-demo $ ./compiler
[ heinthant@macbookpro ] codes/heinthant/nim-demo $ file test
test: Mach-O 64-bit executable arm64

2. But when I invoke compiler via proxies, the output from test.c is amd64 ( although compiler is arm64 but cc is universal )

[ heinthant@macbookpro ] codes/heinthant/nim-demo $ ./proxies
[ heinthant@macbookpro ] codes/heinthant/nim-demo $ file test
test: Mach-O 64-bit executable x86_64

So, to conclude, I believe, although nim binaries are arm64, when invoked via amd64 proxies, the compiler output will be amd64 due to universal C compilers from macOS.

We DON'T NEED choosenim itself to be universal or arm64. But we need proxies to be arm64.

[heinthanth]

@dom96, Ahh ... I get it ... we can build universal proxies on mac. So, we can accomplish with minimal changes

[heinthanth]

@elcritch @dom96 We combined ideas together 🍻

But there's an issue left: Choosenim is downloading x64 tar in devel channel. Other than that, choosenim can build arm64 nim binaries.

The proxyexe is now compiled at choosenim runtime. Can we do it at compile-time as it was done before? I'm concerned that performing this at runtime will be too unreliable.

Solved! ( Proxies are compiled and embed in choosenim )

In general I'm wondering: do we need proxyexe and choosenim to be compiled as arm64? Can't we just keep the changes in utils.nim and have choosenim make an arm64 Nim build with just these changes?

Yeah, like above explanation, proxies need to be arm64 on M1 macs, so I build universal proxies. Solved!

I think we can merge this now after some testing.

[heinthanth]

Cool, resolved suggestions from reviews.

[heinthanth]

Cool, I also fixed choosenim downloading x64 prebuilt binaries on devel channel.

if not isRosetta() and hostCPU == "amd64":

that did the trick. On my machine, choosenim works fine! How about yours?

[heinthanth]

@dom96

macOS without Rosetta

I added several checks ( isAppleSilicon, isMacOSBelowBigSurCompileTime, isMacOSBelowBigSur ).

isMacOSBelowBigSurCompileTime check during compile-time ( build machine )

if build machine is below macOS Big Sur, arm64 proxies won't be embedded and that choosenim build cannot be used on Apple Silicon macs. Otherwise, two proxies ( amd64 and arm64 ) will be embedded.

isMacOSBelowBigSur check during run-time ( user machine )

if user machine is below Big Sur, amd64 proxies will be used. if user machine is MacOS Big Sur and above, I checked isAppleSilicon. if AppleSilicon, choosenim will install arm64 proxies, Otherwise, amd64 proxies will be installed.

So, I think this will solve upcoming problems on MacOS like Catalina, Mojave, etc. ( macOS on Intel but without rosetta ) For older macOS, above trick will works too.

For Linux and Windows, normal host-specific proxyExe is embedded too.

---

choosenim itself cannot run on older macOS

But for this case, I think we need to tweak build.sh providing min-macos version like @elcritch said.

[esafak]

Are we waiting for something to merge this?

[shish]

Also eagerly awaiting any updates here, as I keep having issues with different nim installs (brew, nix, and choosenim) and this issue seems like the closest to being fixed :)

[ujwal-setlur]

Any update on this PR? Looks like it is ready to go?

[lantos1618]

still waiting on approval?

[heinthanth]

I don’t know … but @dom96 seems like inactive on this repo.