Apple Silicon toolchain support

[jez]

I just got an M1 MacBook Pro today, and am looking into how to use this project to generate arm64 binaries (for the record: everything works fine using this project to generate x86_64 binaries, which then run under Rosetta).

In the Apple Developer Documentation, they make it out to be as simple as passing a -target flag to the C compiler, though I'm sure it'll be more work to do the same thing in Bazel.

https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary

Have anyone put thought or time into how this project might be extended to support generating arm64 binaries on M1 Macs? I'm probably going to be spending some time getting this working, and I'd love any tips, ideas, or places to start.

[rrbutani]

Actually, I'm going to open a new issue for this.

135

[vinistock]

I worked through some more investigations on this and thought I'd bring up another thing I'm seeing. I'm trying to change a dependency of a package by matching on a config_setting that uses platform constraint values. Basically, we need a different dependency on darwin-arm64.

The code I have on a Sorbet branch is equivalent to this:

# BUILD file

platform(
    name = "darwin_arm",
    constraint_values = [
        "@platforms//cpu:arm64",
        "@platforms//os:osx",
    ]
)

platform(
    name = "darwin_x86",
    constraint_values = [
        "@platforms//cpu:x86_64",
        "@platforms//os:osx",
    ],
)

config_setting(
    name = "darwin",
    constraint_values = [
        "@platforms//cpu:x86_64",
        "@platforms//os:osx",
    ]
)

config_setting(
    name = "darwin_arm64",
    constraint_values = [
        "@platforms//cpu:arm64",
        "@platforms//os:osx",
    ]
)

# Library BUILD file
cc_library(
  ...
  deps = select({
    "//:darwin": ["x86_specific_dependency"],
    "//:darwin_arm64": ["portable_version_dependency"],
    "//conditions:default": ["portable_version_dependency"]
  })
)

However, we always match //:darwin even when selecting the platform to be --platforms=@//:darwin_arm. If I remove the entry for darwin from the select, it falls under default and compiles successfully. Could this be related to the new toolchains?

This is the Sorbet branch I'm currently playing with if that helps.

[sluongng]

@vinistock make sure that you are using a bazel arm64 release and not bazel amd64 running on rosetta. I think there is also a flag that help debug toolchain resolution that can be passed to bazel.

[vinistock]

@sluongng you are right! I was indeed running an x86 version of bazel. Upgrading to 5.0.0 with an ARM64 executable seems to fix not only the issue with conditionally compiling the dependency, but also the -framework flag problem!!

I will spend some time validating it, but it looks like that was the issue all along.

[vinistock]

I can confirm that using the ARM64 version of Bazel fixes the -framework issue and now the config_settings match properly. In the branch I'm working on, it's already possible to compile Sorbet in debug and release mode, as well as running some tests.

I am, however, having the same issue I was having before of the custom Ruby build selecting the wrong toolchains when using find_cpp_toolchain. This time, I'm sure I am invoking llvm_register_toolchains.

While debugging, I noticed that the arm64 toolchain is rejected by using the --toolchain_resolution_debug flag.

INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform @local_config_platform//:host: Rejected toolchain //:clang-darwin-arm64-toolchain; mismatching values: x86_64

I was also able to confirm that changing the exec_compatible_with configuration in the toolchain from x86 to arm64 fixes the resolution and selects the right toolchain for the custom Ruby build. However, that breaks the regular Sorbet build.

toolchain(
    name = "clang-darwin-arm64",
    exec_compatible_with = [
        "@platforms//cpu:x86_64", # If this value is x86_64, the toolchain is rejected. If we use arm64, the toolchain is selected
        "@platforms//os:osx",
    ],
    target_compatible_with = [
        "@platforms//cpu:arm64",
        "@platforms//os:osx",
    ],
    toolchain = ":clang-darwin-arm64-toolchain",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

I'm not sure why, when using exec_compatible_with x86, the right toolchain is selected for Sorbet, but not for the Ruby build. Any ideas on what could be happening?

[Kernald]

Hi! Just wondering if any progress has been made on that since February?

[totnine]

LLVM 15.0.0 was released, which contains 'clang+llvm-15.0.0-arm64-apple-darwin21.0.tar.xz'.

[omerbenamram]

I checked it out and it works really nice! (https://github.com/omerbenamram/bazel-toolchain/tree/llvm-15.0.0) Unfortunately there are still no binaries for x86 linux (which is quite strage) for llvm 15 - so I guess we can't upstream this yet.

[iamricard]

I checked it out and it works really nice! (https://github.com/omerbenamram/bazel-toolchain/tree/llvm-15.0.0) Unfortunately there are still no binaries for x86 linux (which is quite strage) for llvm 15 - so I guess we can't upstream this yet.

15.0.2 does have a couple of x86 linux builds. i tested the RHEL on an ubuntu machine and it worked fine, so maybe we could just use that? more artifacts are listed in the discourse thread for the release.

off version 0.7.2 of this repo, a patch like this seems to work (props to @omerbenamram for basically putting this diff together):

diff --git a/toolchain/cc_toolchain_config.bzl b/toolchain/cc_toolchain_config.bzl
index 8785a8e..1ac65a5 100644
--- a/toolchain/cc_toolchain_config.bzl
+++ b/toolchain/cc_toolchain_config.bzl
@@ -68,6 +68,15 @@ def cc_toolchain_config(
             "clang",
             "darwin_x86_64",
             "darwin_x86_64",
+        ),
+         "darwin-aarch64": (
+            "clang-aarch64-darwin",
+            "aarch64-apple-macosx",
+            "darwin",
+            "macosx",
+            "clang",
+            "darwin_aarch64",
+            "darwin_aarch64",
         ),
         "linux-aarch64": (
             "clang-aarch64-linux",
diff --git a/toolchain/internal/common.bzl b/toolchain/internal/common.bzl
index 7493c64..53b3b53 100644
--- a/toolchain/internal/common.bzl
+++ b/toolchain/internal/common.bzl
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64")]
+SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64"), ("darwin", "aarch64")]

 host_tool_features = struct(
     SUPPORTS_ARG_FILE = "supports_arg_file",
@@ -68,7 +68,10 @@ def arch(rctx):
     ])
     if exec_result.return_code:
         fail("Failed to detect machine architecture: \n%s\n%s" % (exec_result.stdout, exec_result.stderr))
-    return exec_result.stdout.strip()
+    arch = exec_result.stdout.strip()
+    if arch == "arm64":
+        return "aarch64"
+    return arch

 def os_arch_pair(os, arch):
     return "{}-{}".format(os, arch)
diff --git a/toolchain/internal/llvm_distributions.bzl b/toolchain/internal/llvm_distributions.bzl
index 074ed84..43473ae 100644
--- a/toolchain/internal/llvm_distributions.bzl
+++ b/toolchain/internal/llvm_distributions.bzl
@@ -207,6 +207,12 @@ _llvm_distributions = {
     "clang+llvm-14.0.0-x86_64-apple-darwin.tar.xz": "cf5af0f32d78dcf4413ef6966abbfd5b1445fe80bba57f2ff8a08f77e672b9b3",
     "clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz": "61582215dafafb7b576ea30cc136be92c877ba1f1c31ddbbd372d6d65622fef5",
     "clang+llvm-14.0.0-x86_64-linux-sles12.4.tar.xz": "78f70cc94c3b6f562455b15cebb63e75571d50c3d488d53d9aa4cd9dded30627",
+
+    # 15.0.2
+    "clang+llvm-15.0.2-arm64-apple-darwin21.0.tar.xz": "8c33f807bca56568b7060d0474daf63c8c10ec521d8188ac76362354d313ec58",
+    "clang+llvm-15.0.2-x86_64-apple-darwin.tar.xz": "a37ec6204f555605fa11e9c0e139a251402590ead6e227fc72da193e03883882",
+    "clang+llvm-15.0.2-aarch64-linux-gnu.tar.xz": "527ed550784681f95ec7a1be8fbf5a24bd03d7da9bf31afb6523996f45670be3",
+    "clang+llvm-15.0.2-x86_64-unknown-linux-gnu-rhel86.tar.xz": "f48f479e91ee7297ed8306c9d4495015691237cd91cc5330d3e1ee057b0548bd",
 }

 # Note: Unlike the user-specified llvm_mirror attribute, the URL prefixes in
@@ -229,6 +235,7 @@ _llvm_distributions_base_url = {
     "13.0.0": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
     "13.0.1": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
     "14.0.0": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
+    "15.0.2": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
 }

 def _get_auth(ctx, urls):
diff --git a/toolchain/tools/llvm_release_name.py b/toolchain/tools/llvm_release_name.py
index 39505cc..3485d61 100755
--- a/toolchain/tools/llvm_release_name.py
+++ b/toolchain/tools/llvm_release_name.py
@@ -30,7 +30,12 @@ def _patch_llvm_version(llvm_version):

 def _darwin(llvm_version, arch):
     major_llvm_version = _major_llvm_version(llvm_version)
-    suffix = "darwin-apple" if major_llvm_version == 9 else "apple-darwin"
+    if major_llvm_version == 9:
+        suffix = "darwin-apple"
+    elif arch == "arm64":
+        suffix = "apple-darwin21.0"
+    else:
+        suffix = "apple-darwin"
     return "clang+llvm-{llvm_version}-{arch}-{suffix}.tar.xz".format(
         llvm_version=llvm_version, arch=arch, suffix=suffix)

@@ -86,6 +91,8 @@ def _linux(llvm_version, distname, version, arch):
     # If you find this mapping wrong, please send a Pull Request on Github.
     if arch in ["aarch64", "armv7a", "mips", "mipsel"]:
         os_name = "linux-gnu"
+    elif major_llvm_version == 15:
+        os_name = "unknown-linux-gnu-rhel86"
     elif distname == "freebsd":
         os_name = "unknown-freebsd-%s" % version
     elif distname == "suse":

[jbott]

It looks like 15.0.3 was recently released, but unfortunately still does not include any x86_64 linux binaries.

For now, I'm used the patch provided by @iamricard and @omerbenamram. It seems to work well, with the addition of an extra patch to fix the strip_prefix for the clang+llvm-15.0.2-x86_64-unknown-linux-gnu-rhel86.tar.xz file. It seems to be only clang+llvm-15.0.2-x86_64-unknown-linux-gnu, missing the -rhel86 suffix. I've only checked the darwin aarch64 and linux x86_64 builds though, so this may be the case with others.

I have an example standalone project using this toolchain here. It's focused on building cross-platform py3_image targets, so the use of this toolchain is really orthogonal to the purpose of the demo, but it may be useful to someone.

Full patch, in case that repo gets removed at some point:

diff --git a/toolchain/cc_toolchain_config.bzl b/toolchain/cc_toolchain_config.bzl
index 8785a8e..1ac65a5 100644
--- a/toolchain/cc_toolchain_config.bzl
+++ b/toolchain/cc_toolchain_config.bzl
@@ -68,6 +68,15 @@ def cc_toolchain_config(
             "clang",
             "darwin_x86_64",
             "darwin_x86_64",
+        ),
+         "darwin-aarch64": (
+            "clang-aarch64-darwin",
+            "aarch64-apple-macosx",
+            "darwin",
+            "macosx",
+            "clang",
+            "darwin_aarch64",
+            "darwin_aarch64",
         ),
         "linux-aarch64": (
             "clang-aarch64-linux",
diff --git a/toolchain/internal/common.bzl b/toolchain/internal/common.bzl
index 7493c64..53b3b53 100644
--- a/toolchain/internal/common.bzl
+++ b/toolchain/internal/common.bzl
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64")]
+SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64"), ("darwin", "aarch64")]

 host_tool_features = struct(
     SUPPORTS_ARG_FILE = "supports_arg_file",
@@ -68,7 +68,10 @@ def arch(rctx):
     ])
     if exec_result.return_code:
         fail("Failed to detect machine architecture: \n%s\n%s" % (exec_result.stdout, exec_result.stderr))
-    return exec_result.stdout.strip()
+    arch = exec_result.stdout.strip()
+    if arch == "arm64":
+        return "aarch64"
+    return arch

 def os_arch_pair(os, arch):
     return "{}-{}".format(os, arch)
diff --git a/toolchain/internal/llvm_distributions.bzl b/toolchain/internal/llvm_distributions.bzl
index 074ed84..62011f3 100644
--- a/toolchain/internal/llvm_distributions.bzl
+++ b/toolchain/internal/llvm_distributions.bzl
@@ -207,6 +207,12 @@ _llvm_distributions = {
     "clang+llvm-14.0.0-x86_64-apple-darwin.tar.xz": "cf5af0f32d78dcf4413ef6966abbfd5b1445fe80bba57f2ff8a08f77e672b9b3",
     "clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz": "61582215dafafb7b576ea30cc136be92c877ba1f1c31ddbbd372d6d65622fef5",
     "clang+llvm-14.0.0-x86_64-linux-sles12.4.tar.xz": "78f70cc94c3b6f562455b15cebb63e75571d50c3d488d53d9aa4cd9dded30627",
+
+    # 15.0.2
+    "clang+llvm-15.0.2-arm64-apple-darwin21.0.tar.xz": "8c33f807bca56568b7060d0474daf63c8c10ec521d8188ac76362354d313ec58",
+    "clang+llvm-15.0.2-x86_64-apple-darwin.tar.xz": "a37ec6204f555605fa11e9c0e139a251402590ead6e227fc72da193e03883882",
+    "clang+llvm-15.0.2-aarch64-linux-gnu.tar.xz": "527ed550784681f95ec7a1be8fbf5a24bd03d7da9bf31afb6523996f45670be3",
+    "clang+llvm-15.0.2-x86_64-unknown-linux-gnu-rhel86.tar.xz": "f48f479e91ee7297ed8306c9d4495015691237cd91cc5330d3e1ee057b0548bd",
 }

 # Note: Unlike the user-specified llvm_mirror attribute, the URL prefixes in
@@ -229,6 +235,7 @@ _llvm_distributions_base_url = {
     "13.0.0": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
     "13.0.1": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
     "14.0.0": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
+    "15.0.2": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
 }

 def _get_auth(ctx, urls):
@@ -310,6 +317,9 @@ def _distribution_urls(rctx):

     strip_prefix = basename[:(len(basename) - len(".tar.xz"))]

+    if strip_prefix == "clang+llvm-15.0.2-x86_64-unknown-linux-gnu-rhel86":
+      strip_prefix = "clang+llvm-15.0.2-x86_64-unknown-linux-gnu"
+
     return urls, sha256, strip_prefix

 def _host_os_key(rctx):
diff --git a/toolchain/tools/llvm_release_name.py b/toolchain/tools/llvm_release_name.py
index 39505cc..3485d61 100755
--- a/toolchain/tools/llvm_release_name.py
+++ b/toolchain/tools/llvm_release_name.py
@@ -30,7 +30,12 @@ def _patch_llvm_version(llvm_version):

 def _darwin(llvm_version, arch):
     major_llvm_version = _major_llvm_version(llvm_version)
-    suffix = "darwin-apple" if major_llvm_version == 9 else "apple-darwin"
+    if major_llvm_version == 9:
+        suffix = "darwin-apple"
+    elif arch == "arm64":
+        suffix = "apple-darwin21.0"
+    else:
+        suffix = "apple-darwin"
     return "clang+llvm-{llvm_version}-{arch}-{suffix}.tar.xz".format(
         llvm_version=llvm_version, arch=arch, suffix=suffix)

@@ -86,6 +91,8 @@ def _linux(llvm_version, distname, version, arch):
     # If you find this mapping wrong, please send a Pull Request on Github.
     if arch in ["aarch64", "armv7a", "mips", "mipsel"]:
         os_name = "linux-gnu"
+    elif major_llvm_version == 15:
+        os_name = "unknown-linux-gnu-rhel86"
     elif distname == "freebsd":
         os_name = "unknown-freebsd-%s" % version
     elif distname == "suse":

[junhyeokahn]

@jbott It looks like 15.05 with linux binaires has been released.

[meastham]

I rebased @omerbenamram's change on top of the recent commits from this repo supporting 15.06 if anybody wants to use it: https://github.com/meastham/bazel-toolchain/tree/m1_support

Seems to be working fine

[garymm]

@meastham can you open a PR?

[meastham]

Sure: #174