search

crystal-lang / distribution-scripts

40 stars 24 forks source link

Build universal macOS builds (x86_64 & arm64) #104

Closed maxfierke closed 2 years ago

maxfierke commented 3 years ago

Depends on #98, supersedes #99

This is an alternative approach to Omnibus builds supporting arm64. This PR instead proposes adding aarch64/arm64 slices to the existing x86_64 build process to enable universal builds that can run on any Mac, Intel or Apple Silicon.

Advantages:

Disadvantages:

Most of the interesting stuff is in 19fe7ff

maxfierke commented 3 years ago

Test builds against Crystal 1.1.0 tag (built on x86_64 MacBook Pro, tested on M1 MacBook Air):

bcardiff commented 3 years ago

I've just noticed the approach of this PR. I like it! I didn't know it was possible to have universal binaries.

An advantage of using the same binary is that we would not need M1 infra to build it. It would be great to have it for testing, but this is promising.

Time to build another llvm I guess :-)

@beta-ziliani @straight-shoota opinions regarding going this way?

straight-shoota commented 3 years ago

Sounds good to me. I'm not very familiar with the macOS ecosystem, though. Double file size don't sound optimal, but it seems easy to use and should help us get started. We can revisit this again later.

jhass commented 3 years ago

I did some testing on my M1 MBP using the preview build you shared, namely I

It all worked flawlessly, great job!

Main issues I had was with libraries not being found from /opt/homebrew/lib, which I solved by exporting LIBRARY_PATH pointing to it. Not sure if we should handle this in the compiler or not, but totally separate discussion from this.

When compiling the bootstrapped compiler and subsequently the specs with it I got a bunch of (and similar):

ld: warning: direct access in function 'ltmp2' from file '/Users/j.hass/projects/crystal/src/llvm/ext/llvm_ext.o' to global weak symbol 'llvm::unwrap(LLVMCodeModel, bool&)' from file '/Users/j.hass/projects/crystal/src/llvm/ext/llvm_ext.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

but it doesn't seem to break anything, so 🤷 .

Another minor thing I noticed is that for some reason the default target triple differs slightly between your universal build (aarch64-apple-darwin) and my bootstrapped compiler (aarch64-apple-darwin20.5.0). I don't think it matters any, I'm just writing down all my observations.

maxfierke commented 3 years ago

Main issues I had was with libraries not being found from /opt/homebrew/lib, which I solved by exporting LIBRARY_PATH pointing to it. Not sure if we should handle this in the compiler or not, but totally separate discussion from this.

I think that's a pretty common issue in the M1 ecosystem. Definitely worth adding to the CRYSTAL_LIBRARY_PATH for the Homebrew formula if it doesn't already get added. Seems nice to add to the compiler though too for the same behavior for other builds.

Another minor thing I noticed is that for some reason the default target triple differs slightly between your universal build (aarch64-apple-darwin) and my bootstrapped compiler (aarch64-apple-darwin20.5.0). I don't think it matters any, I'm just writing down all my observations.

yeah that's LLVM just adding the more specific OS version. I think you'd probably see the same thing on x86_64 if you were to target x86_64-apple-darwin specifically vs compiling with the default LLVM host target.

beta-ziliani commented 3 years ago

Thanks a lot @maxfierke for the effort and the alternative! And thanks @jhass for doing the testing.

In the long run I expect to have split archs to test, but for the moment this is an awesome alternative.

maxfierke commented 3 years ago

Merged in the latest master, so should be a very clean/limited diff now

bcardiff commented 3 years ago

Something is not working on the CI.

I bumped xcode, ruby and distribution-scripts in https://github.com/crystal-lang/crystal/tree/ci/univ but dist-darwin is not happy.

Time: 00:01:30 ============================================== 100% (5270 KB/sec)
   [NetFetcher: llvm_bin] I | 2021-08-19T14:02:38+00:00 | Verifying checksum
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Project 'crystal' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'llvm_bin' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'pcre' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'libatomic_ops' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'bdw-gc' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'libevent' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'crystal' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'libyaml' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'shards' does not contain licensing information.
              [Licensing] W | 2021-08-19T14:02:39+00:00 | Software 'tgz_package' does not contain licensing information.
     [Software: llvm_bin] I | 2021-08-19T14:02:40+00:00 | Could not restore from cache
   [NetFetcher: llvm_bin] I | 2021-08-19T14:02:40+00:00 | Cleaning project directory `/var/cache/omnibus/src/llvm_bin'
   [NetFetcher: llvm_bin] I | 2021-08-19T14:02:40+00:00 | Extracting `/var/cache/omnibus/cache/llvm-10.0.0-3-darwin-x86_64.tar.gz' to `/var/cache/omnibus/src/llvm_bin'
      [Builder: llvm_bin] I | 2021-08-19T14:02:48+00:00 | Starting build
      [Builder: llvm_bin] I | 2021-08-19T14:02:48+00:00 | Build llvm_bin: 0.0s
      [Builder: llvm_bin] I | 2021-08-19T14:02:48+00:00 | Finished build
     [Software: llvm_bin] I | 2021-08-19T14:02:49+00:00 | Dirtied the cache
         [Software: pcre] I | 2021-08-19T14:02:49+00:00 | Building because `llvm_bin' dirtied the cache
       [NetFetcher: pcre] I | 2021-08-19T14:02:49+00:00 | Cleaning project directory `/var/cache/omnibus/src/pcre'
       [NetFetcher: pcre] I | 2021-08-19T14:02:49+00:00 | Extracting `/var/cache/omnibus/cache/pcre-8.40.tar.gz' to `/var/cache/omnibus/src/pcre'
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 | Starting build
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 | Environment:
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   CFLAGS="-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC -arch arm64 -arch x86_64"
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   CPPFLAGS="-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC "
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   CXXFLAGS="-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC -arch arm64 -arch x86_64"
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   LDFLAGS="-Wl,-rpath,/opt/crystal/embedded/lib -L/opt/crystal/embedded/lib"
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   LD_RUN_PATH="/opt/crystal/embedded/lib"
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   OMNIBUS_INSTALL_DIR="/opt/crystal"
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   PATH="/opt/crystal/bin:/opt/crystal/embedded/bin:/Users/distiller/.gem/ruby/2.7.3/bin:/Users/distiller/.rubies/ruby-2.7.3/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin"
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 |   PKG_CONFIG_PATH="/opt/crystal/embedded/lib/pkgconfig"
          [Builder: pcre] I | 2021-08-19T14:02:49+00:00 | $ ./configure --prefix=/opt/crystal/embedded --disable-shared --enable-unicode-properties --enable-utf8
          [Builder: pcre] I | 2021-08-19T14:02:51+00:00 | Execute: `./configure --prefix=/opt/crystal/embedded --disable-shared --enable-unicode-properties --enable-utf8': 2.0583s
          [Builder: pcre] I | 2021-08-19T14:02:51+00:00 | Build pcre: 2.0593s
The following shell command exited with status 77:

    $ CFLAGS=-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC -arch arm64 -arch x86_64 CPPFLAGS=-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC  CXXFLAGS=-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC -arch arm64 -arch x86_64 LDFLAGS=-Wl,-rpath,/opt/crystal/embedded/lib -L/opt/crystal/embedded/lib LD_RUN_PATH=/opt/crystal/embedded/lib OMNIBUS_INSTALL_DIR=/opt/crystal PATH=/opt/crystal/bin:/opt/crystal/embedded/bin:/Users/distiller/.gem/ruby/2.7.3/bin:/Users/distiller/.rubies/ruby-2.7.3/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin PKG_CONFIG_PATH=/opt/crystal/embedded/lib/pkgconfig ./configure --prefix=/opt/crystal/embedded --disable-shared --enable-unicode-properties --enable-utf8

Output:

    checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... no

Error:

    configure: error: in `/private/var/cache/omnibus/src/pcre/pcre-8.40':
configure: error: C compiler cannot create executables
See `config.log' for more details

/Users/distiller/.gem/ruby/2.7.3/bundler/gems/omnibus-1d97cd9e79a0/lib/omnibus/util.rb:139:in `rescue in shellout!'
  /Users/distiller/.gem/ruby/2.7.3/bundler/gems/omnibus-1d97cd9e79a0/lib/omnibus/util.rb:134:in `shellout
onderweg commented 3 years ago

Test builds against Crystal 1.1.0 tag (built on x86_64 MacBook Pro, tested on M1 MacBook Air):

Is there a new (unexpired) link available? 🙏

bcardiff commented 3 years ago

FTR based on https://github.com/actions/virtual-environments/issues/3288 I was able to move on step forward to run this on circleci by adding these changes a92ff99d9beaf618252ce56a46fed41d6c90aaa6 yet they are not enough 😞 

       [Builder: crystal] I | 2021-08-23T23:46:40+00:00 | Execute: `make deps': 1.6887s
       [Builder: crystal] I | 2021-08-23T23:46:40+00:00 | Build crystal: 1.7321s
The following shell command exited with status 2:

    $ CFLAGS=-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC -arch arm64 -arch x86_64 -target aarch64-apple-darwin CPPFLAGS=-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC  CRYSTAL_LIBRARY_PATH=/opt/crystal/embedded/lib CRYSTAL_PATH=/private/var/cache/omnibus/src/crystal/src CXXFLAGS=-I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC -arch arm64 -arch x86_64 -target aarch64-apple-darwin LDFLAGS=-Wl,-rpath,/opt/crystal/embedded/lib -L/opt/crystal/embedded/lib LD_RUN_PATH=/opt/crystal/embedded/lib LIBRARY_PATH=/opt/crystal/embedded/lib OMNIBUS_INSTALL_DIR=/opt/crystal PATH=/var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/bin:/var/cache/omnibus/src/crystal/deps:/opt/crystal/bin:/opt/crystal/embedded/bin:/Users/distiller/.gem/ruby/2.7.3/bin:/Users/distiller/.rubies/ruby-2.7.3/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin PKG_CONFIG_PATH=/opt/crystal/embedded/lib/pkgconfig make deps

Output:

    c++ -c -I/opt/crystal/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector -fPIC -arch arm64 -arch x86_64 -target aarch64-apple-darwin  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/private/var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/include -std=c++14   -fno-exceptions -fno-rtti -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

Error:

    Using /var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/bin/llvm-config [version=10.0.0]
In file included from src/llvm/ext/llvm_ext.cc:1:
In file included from /private/var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/include/llvm/IR/DIBuilder.h:17:
In file included from /private/var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/include/llvm/ADT/ArrayRef.h:12:
In file included from /private/var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/include/llvm/ADT/Hashing.h:47:
In file included from /private/var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/include/llvm/Support/DataTypes.h:16:
In file included from /private/var/cache/omnibus/src/llvm_bin/llvm-10.0.0-3/include/llvm-c/DataTypes.h:28:
In file included from /Applications/Xcode-12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:304:
In file included from /Applications/Xcode-12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/math.h:301:
In file included from /Applications/Xcode-12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/stdlib.h:20:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/stdlib.h:62:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
#error Unsupported architecture
 ^
In file included from src/llvm/ext/llvm_ext.cc:1:
maxfierke commented 3 years ago

@bcardiff I'm wondering if it's using Clang++ from XCode, instead of our LLVM for compiling that piece (in which case it might take issue with the "aarch64" part of the "target" parameter because XCode's LLVM insists on calling it "arm64"). Maybe we could try setting the CXX path to our LLVM's clang++ to ensure it uses our LLVM? It looks like PATH is set okay, but wondering if CXX gets set by something outside of Omnibus (e.g. CircleCI's XCode stuff)

Didn't run into this locally, so I'm curious if something from Homebrew is making its way into the process too (e.g. re: the earlier issue, I have automake, autoconf, and binutils installed from Homebrew, which might explain why I didn't run into issues there)

bcardiff commented 3 years ago

The llvm_bin packages we are using does not include clang. And the CI in circle_ci has not been provisioned with homebrew so far in order to keep some degree of reproducibility.

I'll see if using clang++ from https://releases.llvm.org/download.html works. Probably 11.0.0 since I see that arm64 is listed as a supported target there. llvm 10.0.0 does not support clang++ -print-targets so I am not sure if it will work.

Using clang++ from brew is not an option since it will be updated at any given time.

Just in case, what is the output of /usr/local/opt/llvm/bin/clang++ --version for you @maxfierke ?

maxfierke commented 3 years ago

The llvm_bin packages we are using does not include clang. And the CI in circle_ci has not been provisioned with homebrew so far in order to keep some degree of reproducibility.

I'll see if using clang++ from https://releases.llvm.org/download.html works. Probably 11.0.0 since I see that arm64 is listed as a supported target there. llvm 10.0.0 does not support clang++ -print-targets so I am not sure if it will work.

Using clang++ from brew is not an option since it will be updated at any given time.

Just in case, what is the output of /usr/local/opt/llvm/bin/clang++ --version for you @maxfierke ?

ahhh you know what, I'm also using clang from Xcode because Homebrew's LLVM's clang isn't linked into /usr/local/bin

Xcode clang:

$ clang++ --version
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
maxfierke commented 3 years ago

@bcardiff I think this might have more to do with the wrong macOS SDK being used via isysroot?. XCode 12.4 is new enough to target arm64 & x86_64 universal builds, but the 10.15 SDK being referenced is not. It should be pointing to something like: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk or maybe /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk

xcrun --sdk macosx --show-sdk-path should point to the correct SDK, I believe? doesn't have to be MacOSX11.3.sdk specifically, I guess, but an SDK for macOS 11.

Alternative, it sounds like their XCode 12.5.1 executor is running Big Sur, which should have the right SDK by default

maxfierke commented 3 years ago

@bcardiff Figured it out. -isysroot was the right direction, but there were a few more spots that needed it specified, and then some target triples tweaked for Apple's clang. Updated to include setting SDKROOT, which does the same but also covers some spots where the wrong Command Line Tools SDK would make its way into the linker arguments and blow up.

My laptop running Big Sur was clearly too smooth sailing, so I tested it a spare machine with a close-to-clean install of macOS 10.15.7 and XCode 12.4 to match CircleCI. I need to test the generated builds still to check they're still working as expected, but I think it should all build under the CircleCI runner now

straight-shoota commented 2 years ago

I started a test run on circle, but the dist_darwin job fails: https://app.circleci.com/pipelines/github/crystal-lang/crystal/6809/workflows/4129662e-0d99-4925-b2aa-457622bc0c45/jobs/637832 (unknown Ruby: 2.7.3). Not sure what's going on there. Latest nightly build worked, and I just merged master to avoid any synchronization issues. Does this PR require a newer Ruby version? We're using 2.5.6 in circleci apparently.

maxfierke commented 2 years ago

@straight-shoota would check out @bcardiff's earlier branch: https://github.com/crystal-lang/crystal/tree/ci/univ you'll def need to bump Ruby in crystal-lang/crystal and maybe bump Xcode there to 12.4 if its specified there too

straight-shoota commented 2 years ago

Oops, apparently I missed that.

I've updated the ci/univ branch, CI is running now: https://app.circleci.com/pipelines/github/crystal-lang/crystal/6818/workflows/295bbb0f-fdf0-4b09-a020-b5387cddf42e/jobs/63799

bcardiff commented 2 years ago

It seems we have a winner :-)

The first gen of the compiler can be downloaded from https://63809-6887813-gh.circle-artifacts.com/0/dist_packages/crystal-ci-univ-dev-1-darwin-universal.tar.gz

A second gen built in https://github.com/crystal-lang/crystal/commit/d7f25e0c605c19d6c325d6306fb24715e8930989 / https://app.circleci.com/pipelines/github/crystal-lang/crystal/6833/workflows/47d9c257-1d3d-4e66-a0c5-7de40458c28f can be downloaded from https://63866-6887813-gh.circle-artifacts.com/0/dist_packages/crystal-ci-univ2nd-dev-1-darwin-universal.tar.gz

While testing it locally (on x86_64) I've found no warnings as the one @jhass mentioned in https://github.com/crystal-lang/distribution-scripts/pull/104#issuecomment-885623876 . I would be interested if someone with M1 can check if the compiler generate those warnings there.

In order to use the universal compiler for crystal development, and to keep using openssl from homebrew I used the following env variables setup with the compiler's Makefile:

$ cd ~/crystal/working/copy
$ PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig" \
  LLVM_CONFIG=... \
  CRYSTAL=~/Downloads/crystal-ci-univ-dev-1/bin/crystal \
  make ...

Something to have in mind is that after this is merged and released some of the changes in https://github.com/crystal-lang/crystal/commit/d7f25e0c605c19d6c325d6306fb24715e8930989 needs to be merged in master and the install-crystal github action will need to be updated in order to look up for darwin-universal.tar.gz in https://github.com/crystal-lang/install-crystal/blob/80e31f4d35ac2d489a8b9843f8ef3875a05e51b7/index.js#L128. I'm not sure if the path is exposed somewhere else. If we want to keep the -darwin-x86_64.tar.gz we can, but we need to untar/rename/tar the package.

jhass commented 2 years ago

No warnings on M1 with the second gen compiler now! 🎉

Major pain point still is finding libraries, worked around that with an explicit CRYSTAL_LIBRARY_PATH=$(brew --prefix)/lib this time, I think the default of -L/usr/local/lib will bite us on these setups. But again, totally out of scope here :)

bcardiff commented 2 years ago

I think the pain point @jhass is because we are using a tar.gz that know nothing of brew. In brew the path you expect is defined in their wrapper https://github.com/Homebrew/homebrew-core/blob/d0d92be271cc47bbb3cff3222e3aec105f5a805c/Formula/crystal.rb#L102

jhass commented 2 years ago

Well as far as I remember the entire point of adding /usr/local/lib as a default in the first place is to make the tarball work out of the box on macOS. No other system benefits from that. And now for macOS M1 machines it's actively harmful by putting the wrong kind of architecture libraries into the path leading to more confusing error messages.

maxfierke commented 2 years ago

Can confirm the builds from CI look good on the M1. Thanks for your help w/ the CI-side of things @bcardiff and @straight-shoota !

bcardiff commented 2 years ago

I see @jhass. I was missing that /usr/local/lib is not a good default for M1. We can iterate the wrapper script and detect arch if needed I guess. Thanks!

beta-ziliani commented 2 years ago

Awesome work! 🚀