Support local OSX SDK

[sluongng]

Toolchain setup

http_archive(
    name = "bazel-zig-cc",
    sha256 = "e9f82bfb74b3df5ca0e67f4d4989e7f1f7ce3386c295fd7fda881ab91f83e509",
    strip_prefix = "bazel-zig-cc-v1.0.1",
    urls = ["https://github.com/uber/bazel-zig-cc/archive/v1.0.1.tar.gz"],
)

load("@bazel-zig-cc//toolchain:defs.bzl", zig_toolchains = "toolchains")

zig_toolchains()

register_toolchains(
    "@zig_sdk//toolchain:linux_amd64_gnu.2.34",
    "@zig_sdk//toolchain:linux_arm64_gnu.2.34",
    "@zig_sdk//toolchain:darwin_amd64",
    "@zig_sdk//toolchain:darwin_arm64",
)

Error outputs

> bazel build @com_github_mattn_go_ieproxy//...
ERROR: /private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/com_github_mattn_go_ieproxy/BUILD.bazel:57:8: GoCompilePkg external/com_github_mattn_go_ieproxy/go-ieproxy_test.internal.a failed: (Exit 1): builder failed: error executing command (from target @com_github_mattn_go_ieproxy//:go-ieproxy_test) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/go_sdk_darwin_arm64/builder_reset/builder compilepkg -sdk external/go_sdk_darwin_arm64 -installsuffix darwin_arm64 -src ... (remaining 73 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
buildbuddy/external/com_github_mattn_go_ieproxy/ieproxy_darwin.go:7:10: fatal error: 'CFNetwork/CFProxySupport.h' file not found
#include <CFNetwork/CFProxySupport.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
compilepkg: error running subcommand external/go_sdk_darwin_arm64/pkg/tool/darwin_arm64/cgo: exit status 2

Gazelle generated build files

> cat /private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/com_github_mattn_go_ieproxy/BUILD.bazel
load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")

go_library(
    name = "go-ieproxy",
    srcs = [
        "ieproxy.go",
        "ieproxy_darwin.go",
        "ieproxy_unix.go",
        "ieproxy_windows.go",
        "kernel32_data_windows.go",
        "pac_darwin.go",
        "pac_unix.go",
        "pac_windows.go",
        "proxy_middleman.go",
        "proxy_middleman_darwin.go",
        "proxy_middleman_unix.go",
        "proxy_middleman_windows.go",
        "utils.go",
        "winhttp_data_windows.go",
    ],
    cgo = True,
    clinkopts = select({
        "@io_bazel_rules_go//go/platform:darwin": [
            "-framework CFNetwork",
            "-framework CoreFoundation",
        ],
        "@io_bazel_rules_go//go/platform:ios": [
            "-framework CFNetwork",
            "-framework CoreFoundation",
        ],
        "//conditions:default": [],
    }),
    importpath = "github.com/mattn/go-ieproxy",
    visibility = ["//visibility:public"],
    deps = select({
        "@io_bazel_rules_go//go/platform:darwin": [
            "@org_golang_x_net//http/httpproxy",
        ],
        "@io_bazel_rules_go//go/platform:ios": [
            "@org_golang_x_net//http/httpproxy",
        ],
        "@io_bazel_rules_go//go/platform:windows": [
            "@org_golang_x_net//http/httpproxy",
            "@org_golang_x_sys//windows",
            "@org_golang_x_sys//windows/registry",
        ],
        "//conditions:default": [],
    }),
)

alias(
    name = "go_default_library",
    actual = ":go-ieproxy",
    visibility = ["//visibility:public"],
)

go_test(
    name = "go-ieproxy_test",
    srcs = [
        "darwin_test.go",
        "example_test.go",
        "utils_test.go",
        "windows_test.go",
    ],
    embed = [":go-ieproxy"],
)

Bazel version is 6.0.0 on M1 Mac.

Using Grail's toolchain, the sysroot is set to OSX SDK dynamically with https://github.com/grailbio/bazel-toolchain/blob/069ee4e20ec605a6c76c1798658e17175b2eb35e/toolchain/internal/sysroot.bzl#L24-L39

Ultimately this should affect cxx_builtin_include_directories and builtin_sysroot attribute when call cc_common. create_cc_toolchain_config_info()

[motiejus]

W are not planning to add support non-hermetic osx compilation with bazel-zig-cc. If we go with osx support, it needs to be hermetic. Zig supports linking MachO, but it needs the libc definitions (tbd files). That is the biggest problem, which are gray area (feel free to search for tbd in ziglang issues).

We are open, though, to support tbd files via a user-provided http_archive.

[shs96c]

If there were some way of providing the sysroot it would be possible to set this up.

[motiejus]

If there were some way of providing the sysroot it would be possible to set this up.

I have no idea. It's probably not too difficult, but I never tried. We did not get to the point to try it at Uber too.

If someone in the community needs it and implements support for this sysroot, we will review/accept the PR.

[sluongng]

My research led me to rules_apple where cc_library could be linked like this https://github.com/bazelbuild/rules_apple/blob/master/test/starlark_tests/resources/kext_resources/BUILD

I have not tested whether this copt could be added to rules_go just yet, but if it could then that would solve my problem.

@motiejus how do you solve building application with transitive deps depending on MacOS libraries like this? Do you simply block these deps from being used at Uber entirely?

[motiejus]

@motiejus how do you solve building application with transitive deps depending on MacOS libraries like this? Do you simply block these deps from being used at Uber entirely?

I have only (and by extension Uber) used zig c++ for building Linux targets; I have not built anything non-trivial for OSX with Darwin except the launcher, which does not need any more frameworks except CoreFoundation (thus trivial).

As for how to compile more OSX applications, here are the rough steps:

1. Background reading: https://github.com/ziglang/zig/issues/1349#issuecomment-1028332574
2. Implement ability to specify a Darwin sysroot (via http_archive?) and attach it to a particular Darwin toolchain.
3. Configure the Darwin toolchain to specify the right path to the sysroot, when invoked.

You are welcome to wait (Uber may get to it, but I can't promise anything; someone might as well do it too), implement this yourself or ask/pay someone in the community to implement it for you. Like I said before, if we get a patch that adds a .tbd sysroot support for Darwin targets, we will happily merge and maintain it.

[fparga]

[...] except the launcher, which does not need any more frameworks except CoreFoundation (thus trivial).

Is there something to do to have access to CoreFoundation? We recently updated our dependency on protobuf that since v22.0 depends on abseil that itself internally depends on CoreFoundation:

https://github.com/abseil/abseil-cpp/blob/4ff6968df3833b11f9ab063675126b1562b8b73b/absl/time/internal/cctz/BUILD.bazel#L56-L60

Since then, building any go_library that depends on google libraries that ship protos on a mac fails with that error:

/private/var/tmp/_bazel_user/b36a059997bad77ac57dde217c426f28/external/com_google_absl/absl/time/internal/cctz/BUILD.bazel:32:11: Compiling absl/time/internal/cctz/src/time_zone_lookup.cc [for tool] failed: (Exit 1): sandbox-exec failed: error executing command
  (cd /private/var/tmp/_bazel_user/b36a059997bad77ac57dde217c426f28/sandbox/darwin-sandbox/503/execroot/project && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/var/folders/jh/bh0m7pbj2nqc2b8f5v98sl900000gp/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_user/b36a059997bad77ac57dde217c426f28/sandbox/darwin-sandbox/503/sandbox.sb /var/tmp/_bazel_user/install/202e556d99f024ca401d77f7be1827ad/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/private/var/tmp/_bazel_user/b36a059997bad77ac57dde217c426f28/sandbox/darwin-sandbox/503/stats.out' external/zig_sdk/tools/aarch64-macos-none/c++ -MD -MF bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_absl/absl/time/internal/cctz/_objs/time_zone/time_zone_lookup.d '-frandom-seed=bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_absl/absl/time/internal/cctz/_objs/time_zone/time_zone_lookup.o' '-DBAZEL_CURRENT_REPOSITORY="com_google_absl"' -iquote external/com_google_absl -iquote bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_absl -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-mcpu=apple_m1' -O2 -DNDEBUG -g0 -Werror -w -g0 -c external/com_google_absl/absl/time/internal/cctz/src/time_zone_lookup.cc -o bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_absl/absl/time/internal/cctz/_objs/time_zone/time_zone_lookup.o)
external/com_google_absl/absl/time/internal/cctz/src/time_zone_lookup.cc:26:10: fatal error: 'CoreFoundation/CFTimeZone.h' file not found
#include <CoreFoundation/CFTimeZone.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Our zig_toolchain is configured following the Use case: always compile with zig cc instructions, so with a list of unconditionally added toolchains, and with BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1.

Is it possible to have such a setup working? Or is zig-cc not an option anymore?

I'm posting this question here, because it's related to the original issue, but please let me know if I should create a new one.

[motiejus]

Is it possible to have such a setup working? Or is zig-cc not an option anymore?

I am honestly surprised you were able to compile a real binary to osx using hermetic_cc_toolchain. :)

I'm posting this question here, because it's related to the original issue, but please let me know if I should create a new one.

I'm afraid it won't be actionable, since osx target is not supported in hermetic_cc_toolchain until someone does the work and implements it.

[fparga]

I am honestly surprised you were able to compile a real binary to osx using hermetic_cc_toolchain. :)

Yes it worked perfectly for several month, up until that dependency on abseil got added. I guess I got lucky and my use case was quite trivial, haha.

I'm afraid it won't be actionable, since osx target is not supported in hermetic_cc_toolchain until someone does the work and implements it.

Ok that was my understanding, I was just curious if somehow it was different if the only dependency on OSX SDK was CoreFoundation, but I guess not.

Thank you @motiejus, that's some incredible work you've done there!

[ghost]

This comes up often. Reopening for visibility. Also see my comment in https://github.com/uber/hermetic_cc_toolchain/issues/104#issuecomment-1643816202

[steeve]

I have some updates on that front: https://github.com/ziglang/zig/issues/10299#issuecomment-2045669073

Also, here is a repo rule to download and unpack it:

load(
    "@bazel_tools//tools/build_defs/repo:utils.bzl",
    "get_auth",
    "patch",
    "workspace_and_buildfile",
)

def _http_pkg_archive_impl(rctx):
    if rctx.attr.build_file and rctx.attr.build_file_content:
        fail("Only one of build_file and build_file_content can be provided.")
    rctx.download(
        url = rctx.attr.urls,
        output = ".downloaded.pkg",
        sha256 = rctx.attr.sha256,
        canonical_id = " ".join(rctx.attr.urls),
        auth = get_auth(rctx, rctx.attr.urls),
    )

    res = rctx.execute(["/usr/sbin/pkgutil", "--expand-full", ".downloaded.pkg", "tmp"])
    if res.return_code != 0:
        fail("Failed to extract package: {}".format(res.stdout))
    rctx.delete(".downloaded.pkg")

    strip_prefix = "tmp/Payload/"
    if rctx.attr.strip_prefix:
        strip_prefix += rctx.attr.strip_prefix + "/"
    extracted = rctx.path(strip_prefix)
    if not extracted.is_dir or not extracted.exists:
        fail("Extracted package does not contain expected directory: {}".format(strip_prefix))
    entries = extracted.readdir(watch = "no")
    for entry in entries:
        rctx.execute(["mv", str(entry), entry.basename])
    rctx.delete(extracted)
    workspace_and_buildfile(rctx)
    patch(rctx)

http_pkg_archive = repository_rule(
    _http_pkg_archive_impl,
    attrs = {
        "urls": attr.string_list(mandatory = True),
        "sha256": attr.string(mandatory = True),
        "strip_prefix": attr.string(),
        "build_file": attr.label(allow_single_file = True),
        "build_file_content": attr.string(),
        "workspace_file": attr.label(allow_single_file = True),
        "workspace_file_content": attr.string(),
    },
)