search

google / ml-metadata

For recording and retrieving metadata associated with ML developer and data scientist workflows.
https://www.tensorflow.org/tfx/guide/mlmd
Apache License 2.0
627 stars 148 forks source link

Add support for M1 macs #143

Open sandeshbhatjr opened 2 years ago

sandeshbhatjr commented 2 years ago

When trying to install tfx on a ARM-based M1 Mac, I run into problems with installing ml-metadata which is a dependency. The errors is as follows:

ERROR: Could not find a version that satisfies the requirement ml-metadata==1.6.0 (from versions: 0.12.0.dev0, 0.13.0.dev0, 0.13.1.dev0)
ERROR: No matching distribution found for ml-metadata==1.6.0

Using pip index versions ml-metadata shows no matching distribution. I guess this is because there are no wheels distributed for the ARM based Macs, so I decided to install it from source as suggested on the home page as follows: python setup.py bdist_wheel, with bazel version 5.0.0. This, however, leads to the following error:

INFO: Reading rc options for 'run' from /Users/jogisararamachandrasandeshbhat/Projects/ml-projects/ml-metadata/.bazelrc:
  Inherited 'build' options: --cxxopt=-std=c++17 --incompatible_restrict_string_escapes=false --incompatible_require_linker_input_cc_api=false
ERROR: --incompatible_restrict_string_escapes=false :: Unrecognized option: --incompatible_restrict_string_escapes=false

Is this a bazel version incompatibility?

-- System details: Hardware: MacBook Pro (14-inch, 2021), Apple M1 Pro, 32 GB RAM OS: macOS Moneterey 12.1 Bazel: 5.0.0 Python: 3.8.12 pip: 21.3.1

paulelvers commented 2 years ago

Havin just the same issue. I use miniforge conda.

BrianSong commented 2 years ago

Hi Sandesh,

The recommend bazel version for ml-metadata is 3.7.2. I wonder could you downgrade your bazel version and give it another try to see if the error persists?

sandeshbhatjr commented 2 years ago

This is rather tricky cause bazel itself started supporting darwin-arm64 from v4.2.1. With bazelisk, trying the following USE_BAZEL_VERSION=3.7.2 python setup.py bdist_wheel fails because of the following issue: https://github.com/bazelbuild/bazelisk/issues/245. Basically, there is no arm64 version of Bazel < 4.2.1.

There is a probably way to build bazel from source, but anyway- since the main error from last time was the missing option incompatible_restrict_string_escapes and I think this option is being deprecated in later versions of bazel, I tried using the oldest version of bazel I can, i.e, 4.2.1. Doing USE_BAZEL_VERSION=4.2.1 python setup.py bdist_wheel, the build still fails with a whole bunch of errors as follows:

running bdist_wheel
running build
running bazel_build
INFO: SHA256 (https://github.com/pybind/pybind11/archive/v2.4.3.tar.gz) = 1eed57bc6863190e35637290f97a20c81cfe4d9090ac0a24f3bbf08f265eb71d
DEBUG: Rule 'pybind11' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "1eed57bc6863190e35637290f97a20c81cfe4d9090ac0a24f3bbf08f265eb71d"
DEBUG: Repository pybind11 instantiated at:
  /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/WORKSPACE:118:13: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
WARNING: /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/ml_metadata/proto/BUILD:21:26: in cc_library rule //ml_metadata/proto:metadata_store_proto: target '//ml_metadata/proto:metadata_store_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/ml_metadata/util/BUILD:24:11: in cc_library rule //ml_metadata/util:struct_utils: target '//ml_metadata/util:struct_utils' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/public/BUILD:2020:11: in cc_library rule @com_google_zetasql//zetasql/public:language_options: target '@com_google_zetasql//zetasql/public:language_options' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/ml_metadata/metadata_store/BUILD:56:11: in cc_library rule //ml_metadata/metadata_store:rdbms_metadata_access_object: target '//ml_metadata/metadata_store:rdbms_metadata_access_object' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/common/BUILD:184:11: in cc_library rule @com_google_zetasql//zetasql/common:proto_helper: target '@com_google_zetasql//zetasql/common:proto_helper' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
DEBUG: Rule 'libmysqlclient' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "34f8887af03d022416dd6593de91d0706e57f46b", shallow_since = "1545150882 +0100" and dropping ["tag"]
DEBUG: Repository libmysqlclient instantiated at:
  /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/WORKSPACE:248:22: in <toplevel>
  /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/ml_metadata/workspace.bzl:22:20: in ml_metadata_workspace
  /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/ml_metadata/mysql_configure.bzl:24:23: in mysql_configure
Repository rule new_git_repository defined at:
  /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/bazel_tools/tools/build_defs/repo/git.bzl:186:37: in <toplevel>
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/public/types/BUILD:19:11: in cc_library rule @com_google_zetasql//zetasql/public/types:types: target '@com_google_zetasql//zetasql/public/types:types' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/public/BUILD:65:11: in cc_library rule @com_google_zetasql//zetasql/public:type: target '@com_google_zetasql//zetasql/public:type' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/public/BUILD:1164:11: in cc_library rule @com_google_zetasql//zetasql/public:proto_value_conversion: target '@com_google_zetasql//zetasql/public:proto_value_conversion' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/common/BUILD:336:11: in cc_library rule @com_google_zetasql//zetasql/common:builtin_function_internal: target '@com_google_zetasql//zetasql/common:builtin_function_internal' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/public/BUILD:278:11: in cc_library rule @com_google_zetasql//zetasql/public:simple_catalog: target '@com_google_zetasql//zetasql/public:simple_catalog' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/zetasql/analyzer/BUILD:162:11: in cc_library rule @com_google_zetasql//zetasql/analyzer:resolver: target '@com_google_zetasql//zetasql/analyzer:resolver' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/ml_metadata/metadata_store/BUILD:111:11: in cc_library rule //ml_metadata/metadata_store:query_config_executor: target '//ml_metadata/metadata_store:query_config_executor' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
INFO: Analyzed target //ml_metadata:move_generated_files (95 packages loaded, 9578 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/external/com_google_zetasql/bazel/BUILD:40:15: CcConfigureMakeRule external/com_google_zetasql/bazel/m4/include [for host] failed: (Exit 1): bash failed: error executing command 
  (cd /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/sandbox/darwin-sandbox/12/execroot/ml_metadata && \
  exec env - \
    PATH=/Users/jogisararamachandrasandeshbhat/Library/Caches/bazelisk/downloads/bazelbuild/bazel-4.2.1-darwin-arm64/bin:/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/bin:/Users/jogisararamachandrasandeshbhat/opt/anaconda3/bin:/Users/jogisararamachandrasandeshbhat/opt/anaconda3/condabin:/Applications/google-cloud-sdk/bin:/Users/jogisararamachandrasandeshbhat/.nvm/versions/node/v16.13.1/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin \
  /bin/bash -c 'function cleanup_function() {
local ecode=$?
if [ $ecode -eq 0 ]; then
cleanup_on_success
else
cleanup_on_failure
fi
}
set -e
function cleanup_on_success() {
printf "rules_foreign_cc: Cleaning temp directories"
  rm -rf $BUILD_TMPDIR $EXT_BUILD_DEPS
}
function cleanup_on_failure() {
printf "\nrules_foreign_cc: Build failed!\nrules_foreign_cc: Keeping temp build directory $BUILD_TMPDIR and dependencies directory $EXT_BUILD_DEPS for debug.\nrules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify '\''--sandbox_debug'\'' Bazel command line flag.\n\nrules_foreign_cc: Printing build logs:\n\n_____ BEGIN BUILD LOGS _____\n"
cat "$BUILD_LOG"
printf "\n_____ END BUILD LOGS _____\n"
printf "rules_foreign_cc: Build script location: $BUILD_SCRIPT\n"
printf "rules_foreign_cc: Build log location: $BUILD_LOG\n\n"
}
trap "cleanup_function" EXIT
export BUILD_SCRIPT="bazel-out/host/bin/external/com_google_zetasql/bazel/m4/logs/Configure_script.sh"
export BUILD_LOG="bazel-out/host/bin/external/com_google_zetasql/bazel/m4/logs/Configure.log"
touch $BUILD_LOG
$BUILD_SCRIPT &> $BUILD_LOG')
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox

rules_foreign_cc: Build failed!
rules_foreign_cc: Keeping temp build directory  and dependencies directory  for debug.
rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify '--sandbox_debug' Bazel command line flag.

rules_foreign_cc: Printing build logs:

_____ BEGIN BUILD LOGS _____
Bazel external C/C++ Rules #0.0.8. Building library 'm4'
Environment:______________TMPDIR=/var/folders/z9/qn33rq1d7b9cvzxdsldn_y080000gn/T/
EXT_BUILD_ROOT=/private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/sandbox/darwin-sandbox/12/execroot/ml_metadata
XCODE_VERSION_OVERRIDE=13.2.1.13C100
INSTALLDIR=/private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/sandbox/darwin-sandbox/12/execroot/ml_metadata/bazel-out/host/bin/external/com_google_zetasql/bazel/m4
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x2
PATH=/private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/sandbox/darwin-sandbox/12/execroot/ml_metadata:/Users/jogisararamachandrasandeshbhat/Library/Caches/bazelisk/downloads/bazelbuild/bazel-4.2.1-darwin-arm64/bin:/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/bin:/Users/jogisararamachandrasandeshbhat/opt/anaconda3/bin:/Users/jogisararamachandrasandeshbhat/opt/anaconda3/condabin:/Applications/google-cloud-sdk/bin:/Users/jogisararamachandrasandeshbhat/.nvm/versions/node/v16.13.1/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
BUILD_TMPDIR=/var/folders/z9/qn33rq1d7b9cvzxdsldn_y080000gn/T/tmp.go2FcLzX
PWD=/private/var/tmp/_bazel_jogisararamachandrasandeshbhat/db807437c60e300e8932281d2dba72cf/sandbox/darwin-sandbox/12/execroot/ml_metadata
EXT_BUILD_DEPS=/var/folders/z9/qn33rq1d7b9cvzxdsldn_y080000gn/T/tmp.vpjLcQ4t
SHLVL=2
BUILD_LOG=bazel-out/host/bin/external/com_google_zetasql/bazel/m4/logs/Configure.log
BUILD_SCRIPT=bazel-out/host/bin/external/com_google_zetasql/bazel/m4/logs/Configure_script.sh
APPLE_SDK_PLATFORM=MacOSX
APPLE_SDK_VERSION_OVERRIDE=12.1
_=/usr/bin/env
__________________________ln: illegal option -- t
usage: ln [-Ffhinsv] source_file [link_name]
       ln [-Ffhinsv] source_file ... linkname_dir
       link source_file link_name

_____ END BUILD LOGS _____
rules_foreign_cc: Build script location: bazel-out/host/bin/external/com_google_zetasql/bazel/m4/logs/Configure_script.sh
rules_foreign_cc: Build log location: bazel-out/host/bin/external/com_google_zetasql/bazel/m4/logs/Configure.log

Target //ml_metadata:move_generated_files failed to build
INFO: Elapsed time: 12.859s, Critical Path: 0.34s
INFO: 333 processes: 320 internal, 13 darwin-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Traceback (most recent call last):
  File "setup.py", line 120, in <module>
    setup(
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/__init__.py", line 155, in setup
    return distutils.core.setup(**attrs)
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/jogisararamachandrasandeshbhat/Projects/experimental/ml-metadata/.python_env_files/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "setup.py", line 98, in run
    subprocess.check_call(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/jogisararamachandrasandeshbhat/.nvm/versions/node/v16.13.1/bin/bazel', 'run', '--compilation_mode', 'opt', '--define', 'grpc_no_ares=true', '--verbose_failures', '--macos_minimum_os=10.14', '//ml_metadata:move_generated_files']' returned non-zero exit status 1
sandeshbhatjr commented 2 years ago

Based on the error log above, I tried building zetasql from their repo and it fails. Perhaps that is the culprit? There's an issue for it here: https://github.com/google/zetasql/issues/80

BrianSong commented 2 years ago

Thanks for the investigation, sandesh!

Internally, we are discussing about the priority for support M1 Mac. I will come back with more details.

sandeshbhatjr commented 2 years ago

Thanks. If it helps, the main use case for me is to use with TFX to deploy pipelines on Vortex/AI platform.

nearKim commented 2 years ago

Unfortunately, the problem persists after the last commit. The error messages does not appear to have chaged from what @sandeshbhatjr reported. I'm using bazel version 4.2.1.

BrianSong commented 2 years ago

The entire TF community is current working on support with M1 Mac.

This code change(https://github.com/google/ml-metadata/commit/719202beca5eb5f72b0e4784c29720952995896d) is initial and we are currently reviewing it with Apple team. Will update this issue as long as we heard back from Apple team. Thanks!

stfines-clgx commented 2 years ago

Is there a status update for this?

BrianSong commented 2 years ago

The POC internally would contact with Apple in June 17. After receiving his updates, I would update here.

safoinme commented 2 years ago

@BrianSong Any update on this, please?

htahir1 commented 2 years ago

Would be awesome to get an update on this @BrianSong :-)

BrianSong commented 2 years ago

Thanks for following this up! @safoinme @htahir1

I reached out to the POC in late June for his updates regarding the interaction with Apple team. Unfortunately, he said this specific topic was not mentioned/addressed as he planned. This issue is now reassigned to a new TPM who is currently out of office until next week. I will reach out to her as soon as she is back and update any progress I have.

Thanks for being patient with me.

dvaldivia commented 2 years ago

problem persist with Bazel 5.2.0

➜ bazel --version
bazel 5.2.0
BrianSong commented 2 years ago

I am back with some updates.

Our new TPM is working closely with Apple team and the TensorFlow community is working together for the M1 Mac support. In terms of prioritization, it seems that TensorFlow would be landed first and then it would be MLMD/TFX. But our TPM is scheduling a possible working session with Apple team for answering questions and review code change like https://github.com/google/ml-metadata/commit/719202beca5eb5f72b0e4784c29720952995896d.

Will update this thread if we have any new progress. Thanks for being patient!

htahir1 commented 2 years ago

I am wondering whether there has been an update on this issue. So struggling with Mac M1 here

amerberg commented 2 years ago

I've also been trying to make this work. I was able to work around the error in https://github.com/google/ml-metadata/issues/143#issuecomment-1028183742 by rewriting the calls to ln without -t (which isn't supported by MacOS). However, doing so gives me this error:

ERROR: /private/var/tmp/_bazel_equilibrium/30c23ea65bc29cf1417ebea094fdb8f6/external/zlib/BUILD.bazel:31:11: Compiling adler32.c failed: undeclared inclusion(s) in rule '@zlib//:zlib':
this rule is missing dependency declarations for the following files included by 'adler32.c':
  '/Library/Developer/CommandLineTools/usr/lib/clang/14.0.0/include/stddef.h'
  '/Library/Developer/CommandLineTools/usr/lib/clang/14.0.0/include/__stddef_max_align_t.h'
  '/Library/Developer/CommandLineTools/usr/lib/clang/14.0.0/include/limits.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/limits.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_symbol_aliasing.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_posix_availability.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/limits.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/limits.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/_limits.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/syslimits.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/appleapiopts.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_int8_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_int16_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_int32_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_int64_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_u_int8_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_u_int16_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_u_int32_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_u_int64_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_intptr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uintptr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/endian.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/endian.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_endian.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/libkern/_OSByteOrder.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/libkern/arm/OSByteOrder.h'
  '/Library/Developer/CommandLineTools/usr/lib/clang/14.0.0/include/stdint.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdint.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types/_uint8_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types/_uint16_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types/_uint32_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types/_uint64_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types/_intmax_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types/_uintmax_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/arch.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_u_char.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_u_short.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_u_int.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_caddr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_dev_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_blkcnt_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_blksize_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_gid_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_in_addr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_in_port_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_ino_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_ino64_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_key_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_mode_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_nlink_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_id_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_pid_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_off_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_uid_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_clock_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_size_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_ssize_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_time_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_useconds_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_suseconds_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_rsize_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_errno_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fd_def.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/Availability.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/AvailabilityVersions.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/AvailabilityInternal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fd_setsize.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fd_set.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fd_clr.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fd_zero.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fd_isset.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fd_copy.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_attr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_cond_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_condattr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_mutex_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_mutexattr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_once_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_rwlock_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_rwlockattr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_pthread/_pthread_key_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fsblkcnt_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_fsfilcnt_t.h'
  '/Library/Developer/CommandLineTools/usr/lib/clang/14.0.0/include/stdarg.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/string.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_null.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/strings.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_strings.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_common.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdlib.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/wait.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/signal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/signal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/signal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/machine/_mcontext.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/arm/_mcontext.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/mach/machine/_structs.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/mach/arm/_structs.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_sigaltstack.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_ucontext.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_sigset_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/resource.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_timeval.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/alloca.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_ct_rune_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_rune_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_wchar_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/malloc/_malloc.h'
Target //ml_metadata:move_generated_files failed to build
BrianSong commented 2 years ago

Hi, our TPM is still working on scheduling the working session with Apple team regarding the M1 Mac support for MLMD. I have reached out to her again and will update the thread as long as I hear any update from her.

dvaldivia commented 2 years ago

@BrianSong what is missing and what can be contributed?

htahir1 commented 2 years ago

I'd also be interested to contribute. @BrianSong would be happy to hear whats missing

BrianSong commented 2 years ago

Thanks for the offering! The missing part is the review and testing for the code change: https://github.com/google/ml-metadata/commit/719202beca5eb5f72b0e4784c29720952995896d. This code change mimics the Apple team patch for supporting M1 Mac in tfx-bsl

However, since the internal team now does not have M1 Mac to test and verify it, we are replying on the Apple team's help.

htahir1 commented 2 years ago

@BrianSong I'm not quite sure I understand - that commit seems already to be merged in, so what is there to test?

At ZenML, we have many a few Mac M1s you could use to test and verify this, and scheduling a working session with us would be easier. If you're interested please let me know at hamza@zenml.io and we can schedule it :-). We are very keen to get this supported asap

BrianSong commented 2 years ago

Hi @htahir1, thanks for offering this! The review, or I should say, the guidance for how to add M1 Mac Support under current infrastructure is what we need from the Apple team. I have followed up with our TPM again and hope she could push this to happen soon.

nicholasjng commented 2 years ago

Just my two cents: I have (unsuccessfully) tried building ml_metadata with a contemporary Bazel (5.2/3), and the various failures that I saw there indicate to me that:

Bazel fully supports M1 with the latest build defs, so maybe it's time to migrate some of those dependencies (most of them are 16-18 months old) to newer versions if you want to support M1.

htahir1 commented 2 years ago

@BrianSong Happy to test out @nicholasjng 's suggestions on our mac m1's ;-)! If you would be willing to schedule a meeting

nicholasjng commented 2 years ago

Good news - I have successfully built M1 wheels locally for both Python 3.9 and 3.10 on my machine, using Bazel 5.3.2. (Shoutout to @aliabbasjaffri for help with debugging.)

The changes needed to accomplish this are mostly portable - I did have to modify a few build files in the process to build a valid metadata_store_extension.so object, and also patch out TLS1.3 cipher suites from the mariadb-connector-c repo.

The wheels are untested, but imports in the Python interpreter are working properly, which is a good first sign. I bumped the following dependencies in the process:

absl lts_20210324.2 -> lts_20220623.1
grpc 1.33.2 -> 1.50.0
pybind11 2.4.3 -> 2.10.0
pybind11_bazel -> current HEAD
zetasql -> current HEAD
mariadb-c-connector v3.0.8-release -> v3.3.2

I eliminated protobuf and zlib from the main workspace, as grpc pulls them in transitively. So these deps implicitly get pinned to their respective values in the chosen grpc version, which made sense to me.

I will write up the process here soonish when I get the time, and PR my changes if there's interest.

BrianSong commented 2 years ago

Thanks @nicholasjng, this sounds great! Please feel free to send out PR for review. I will also meet with Apple team next week to get recommendations on build & test configuration, bazelrc config / setup.py settings and "future proofing" of requirements for compatibility.

htahir1 commented 2 years ago

This is amazing work @nicholasjng thank you for your help!

nicholasjng commented 2 years ago

I sent in my changes in #166. Users should be able to build on their own machines by doing the following, in their local repos (if forked, use upstream as the remote instead)

git fetch origin pull/166/head:m1wheels
git checkout m1wheels
python3 -m venv venv --system-site-packages --upgrade-deps  # for wheel and setuptools
source venv/bin/activate
python setup.py bdist_wheel # builds move_generated_files.sh target

NOTE: If you attempt multiple wheel builds for multiple different Python versions in succession (e.g. first 3.9, then 3.10), use bazel clean --expunge between builds, otherwise something or someone (probably pybind11_bazel) gets cached and proceeds to use the wrong Python headers for the second build, resulting in ABI mismatches and errors upon import.

rcrowe-google commented 2 years ago

Thank you @nicholasjng , this is amazing work!

htahir1 commented 2 years ago

@BrianSong Since #166 has been up for a week now, I was wondering whether we could get an update from your meeting with the Apple team?

BrianSong commented 2 years ago

@htahir1 As a result of the meeting, I can use Apple's remote M1 Mac for the testing. In the meanwhile, I am also testing #166 internally as we have some internal tests around OSS code.

However, given other internal projects going on, this would not be a top priority for our team. Does patching #166 work for you as a short term solution?

htahir1 commented 2 years ago

@BrianSong Thank you for the update. While patching mlmd might work for me personally, we are using MLMD as a dependency in our own package and our downstream users cannot possibly patch it the same way in a reasonable manner.

Would it be possible to get a timeline for M1 support if you have it? I do understand that your internal priorities take precedence, but only an estimate would be useful to appease downstream users

kevinnowland commented 2 years ago

I was able to build the wheel using the instructions provided by @nicholasjng (after also installing wheel via pip) and then installed the built wheel into the same virtual environment. However, I am getting symbol not found errors upon import:

In [1]: from ml_metadata.metadata_store.metadata_store import MetadataStore
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In [1], line 1
----> 1 from ml_metadata.metadata_store.metadata_store import MetadataStore

File ~/Code/ml-metadata/ml_metadata/metadata_store/__init__.py:15
      1 # Copyright 2020 Google LLC.
      2 #
      3 # Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
     14 """Init module for ML Metadata."""
---> 15 from ml_metadata.metadata_store.metadata_store import downgrade_schema
     16 from ml_metadata.metadata_store.metadata_store import ListOptions
     17 from ml_metadata.metadata_store.metadata_store import MetadataStore

File ~/Code/ml-metadata/ml_metadata/metadata_store/metadata_store.py:31
     29 from ml_metadata import errors
     30 from ml_metadata import proto
---> 31 from ml_metadata.metadata_store.pywrap.metadata_store_extension import metadata_store as metadata_store_serialized
     32 from ml_metadata.proto import metadata_store_pb2
     33 from ml_metadata.proto import metadata_store_service_pb2

ImportError: dlopen(/Users/kevinnowland/Code/ml-metadata/ml_metadata/metadata_store/pywrap/metadata_store_extension.so, 0x0002): symbol not found in flat namespace (_aes_hw_cbc_encrypt)

Bazel 5.2.0 (installed manually, not via homebrew) python 3.9 (installed using pyenv) Mac OS 12.5.1 (with M1 chip)

Any help getting this installed would be great. Any updates on timeline for the related PR would also be appreciated if there are any.

Thank you!

nicholasjng commented 2 years ago

Hey! I had these errors too when I first succeeded in building the MLMD wheel - only with _zlib_client_plugin instead of _aes_hw_cbc_encrypt.

In my case, it was a symbol in mariadb-c-connector which was undefined due to the source missing in the Bazel build command. Adding the corresponding source file to the build configuration solved the issue.

A quick Google search gives me boringssl as the source of aes_hw_cbc_encrypt (the leading underscore gets added by MLMD's Bazel plugin for the pybind extension in the following):

https://github.com/google/ml-metadata/blob/ccf26c84689db4d3846aa57d72ccd23a5dea0154/ml_metadata/ml_metadata.bzl#L225

Perhaps an include from boringssl is missing somewhere?

kevinnowland commented 2 years ago

Thanks for the help @nicholasjng ! Unfortunately, I'm not really sure I'm capable to use that information successfully as I'm not really familiar with bazel / c / c++... I tried the relatively simple step of updating the commit hash and corresponding sha in the WORKSPACE file to the latest boringssl commit on the master-with-bazel branch of that repo. Predictably, the same error occurred. I'm assuming there's probably some edit in the ml_metadata/third_party/systemlibs/boringssl.BUILD file that would be needed, but I'm not sure what it would be. I do see the function here:

https://github.com/google/boringssl/blob/b3fb5fdaaa30f8b6e659a1f6e0d6adc2234a8aef/src/crypto/fipsmodule/aes/internal.h#L81

but do not know what to do with that information.

nicholasjng commented 2 years ago

I see. The file is there on current master, and the function is guarded behind the HWAES preprocessor definition - but from what I see in the file, that should be set on Apple M1.

Could you see if the produced metadata_store_extension.so file is still in your source tree, and if so, share the output of nm -gC ml_metadata/metadata_store/pywrap/metadata_store_extension.so?

kevinnowland commented 2 years ago

That file is present. Running the command produced the attached 605 line file which includes the line U _aes_hw_cbc_encrypt.

output.txt

nicholasjng commented 2 years ago

Thanks! This is the diff to my (working) SO file:

➜ git diff output.txt out2.txt
diff --git a/output.txt b/out2.txt
index 279557f..e020cad 100644
--- a/output.txt
+++ b/out2.txt
@@ -4,7 +4,6 @@
                  U _CFStringGetMaximumSizeForEncoding
                  U _CFTimeZoneCopyDefault
                  U _CFTimeZoneGetName
-                 U _ChaCha20_ctr32
                  U _PyBaseObject_Type
                  U _PyBuffer_Release
                  U _PyByteArray_AsString
@@ -14,8 +13,8 @@
                  U _PyBytes_AsStringAndSize
                  U _PyBytes_FromStringAndSize
                  U _PyBytes_Size
-                 U _PyCFunction_NewEx
                  U _PyCFunction_Type
+                 U _PyCMethod_New
                  U _PyCapsule_GetContext
                  U _PyCapsule_GetName
                  U _PyCapsule_GetPointer
@@ -39,7 +38,6 @@
                  U _PyErr_WriteUnraisable
                  U _PyEval_AcquireThread
                  U _PyEval_GetBuiltins
-                 U _PyEval_InitThreads
                  U _PyEval_SaveThread
                  U _PyExc_BufferError
                  U _PyExc_ImportError
@@ -53,12 +51,14 @@
                  U _PyException_SetCause
                  U _PyException_SetContext
                  U _PyException_SetTraceback
+                 U _PyFrame_GetBack
+                 U _PyFrame_GetCode
                  U _PyFrame_GetLineNumber
                  U _PyGILState_Ensure
                  U _PyGILState_GetThisThreadState
                  U _PyGILState_Release
                  U _PyImport_AddModule
-0000000000004ae0 T _PyInit_metadata_store_extension
+00000000000032c0 T _PyInit_metadata_store_extension
                  U _PyInstanceMethod_New
                  U _PyInstanceMethod_Type
                  U _PyList_GetItem
@@ -366,23 +366,13 @@
                  U __tlv_bootstrap
                  U _abort
                  U _access
-                 U _aes_hw_cbc_encrypt
-                 U _aes_hw_ctr32_encrypt_blocks
-                 U _aes_hw_decrypt
-                 U _PyCFunction_NewEx
                  U _PyCFunction_Type
+                 U _PyCMethod_New
diff --git a/output.txt b/out2.txt
index 279557f..e020cad 100644
--- a/output.txt
+++ b/out2.txt
@@ -4,7 +4,6 @@
                  U _CFStringGetMaximumSizeForEncoding
                  U _CFTimeZoneCopyDefault
                  U _CFTimeZoneGetName
-                 U _ChaCha20_ctr32
                  U _PyBaseObject_Type
                  U _PyBuffer_Release
                  U _PyByteArray_AsString
@@ -14,8 +13,8 @@
                  U _PyBytes_AsStringAndSize
                  U _PyBytes_FromStringAndSize
                  U _PyBytes_Size
-                 U _PyCFunction_NewEx
                  U _PyCFunction_Type
+                 U _PyCMethod_New
                  U _PyCapsule_GetContext
                  U _PyCapsule_GetName
                  U _PyCapsule_GetPointer
@@ -39,7 +38,6 @@
                  U _PyErr_WriteUnraisable
                  U _PyEval_AcquireThread
                  U _PyEval_GetBuiltins
-                 U _PyEval_InitThreads
                  U _PyEval_SaveThread
                  U _PyExc_BufferError
                  U _PyExc_ImportError
@@ -53,12 +51,14 @@
                  U _PyException_SetCause
                  U _PyException_SetContext
                  U _PyException_SetTraceback
+                 U _PyFrame_GetBack
+                 U _PyFrame_GetCode
                  U _PyFrame_GetLineNumber
                  U _PyGILState_Ensure
                  U _PyGILState_GetThisThreadState
                  U _PyGILState_Release
                  U _PyImport_AddModule
-0000000000004ae0 T _PyInit_metadata_store_extension
+00000000000032c0 T _PyInit_metadata_store_extension
                  U _PyInstanceMethod_New
                  U _PyInstanceMethod_Type
                  U _PyList_GetItem
@@ -366,23 +366,13 @@
                  U __tlv_bootstrap
                  U _abort
                  U _access
-                 U _aes_hw_cbc_encrypt
-                 U _aes_hw_ctr32_encrypt_blocks
-                 U _aes_hw_decrypt
-                 U _aes_hw_encrypt
-                 U _aes_hw_set_decrypt_key
-                 U _aes_hw_set_encrypt_key
                  U _atoi
                  U _backtrace
                  U _backtrace_symbols
-                 U _beeu_mod_inverse_vartime
                  U _bind
-                 U _bn_mul_mont
                  U _bsearch
                  U _bzero
                  U _calloc
-                 U _chacha20_poly1305_open
-                 U _chacha20_poly1305_seal
                  U _clock_gettime
                  U _close
                  U _closedir
@@ -392,16 +382,6 @@
                  U _dlerror
                  U _dlopen
                  U _dlsym
-                 U _ecp_nistz256_mul_mont
-                 U _ecp_nistz256_neg
-                 U _ecp_nistz256_ord_mul_mont
-                 U _ecp_nistz256_ord_sqr_mont
-                 U _ecp_nistz256_point_add
-                 U _ecp_nistz256_point_add_affine
-                 U _ecp_nistz256_point_double
-                 U _ecp_nistz256_select_w5
-                 U _ecp_nistz256_select_w7
-                 U _ecp_nistz256_sqr_mont
                  U _exit
                  U _fchmod
                  U _fchown
@@ -436,9 +416,6 @@
                  U _futimes
                  U _fwrite
                  U _gai_strerror
-                 U _gcm_ghash_v8
-                 U _gcm_gmult_v8
-                 U _gcm_init_v8
                  U _getaddrinfo
                  U _getcontext
                  U _getcwd
@@ -546,9 +523,6 @@
                  U _setcontext
                  U _setlocale
                  U _setsockopt
-                 U _sha1_block_data_order
-                 U _sha256_block_data_order
-                 U _sha512_block_data_order
                  U _shutdown
                  U _signal
                  U _sleep

All red-marked symbols are undefined in your SO file. Looks like the culprit is boringssl, like suspected - the other missing functions also look like they come from there. I would advise the following steps:

Let me know if that works out.

kevinnowland commented 2 years ago

Unfortunately, I still can't get this to work. Going to commit 2a3f948 initially failed due to the version of bazel I was using being too high. I switched to version 3.7.2 as indicated in ml_metadata/bazelversion (HEAD recommends 5.3.2, which I've also tried at HEAD). This failure was due to the flag --incompatible_restrict_string_escapes in .bazelrc, which I believe no longer exists in newer versions of bazel. Replacing this .bazelrc with the version from HEAD and using the newer version of bazel did not work, as there are some string escapes that are bad with the new version.

Upon switching to 3.7.2 at 2a3f948, the build no longer completes. Errors seemed to me to initially be around OpenSSL versions, so I linked a brew installed version and set the OPENSSL_ROOT_DIR and OPENSSL_LIBRARIES environment variables which changed the error message to be something about INCONV_LIBRARIES but did not get me through the build.

Example error logs from cmake:

I cannot find the openssl root directory for the system version. I would like to set the environment variables appropriately for it to see what happens.

BrianSong commented 1 year ago

FYI, I have transferred to another team. Please reach out to @ryanpeters-google for further questions as he is the owner for MLMD OSS now.

gerilya commented 1 year ago

I sent in my changes in #166. Users should be able to build on their own machines by doing the following, in their local repos (if forked, use upstream as the remote instead)

git fetch origin pull/166/head:m1wheels
git checkout m1wheels
python3 -m venv venv --system-site-packages --upgrade-deps  # for wheel and setuptools
source venv/bin/activate
python setup.py bdist_wheel # builds move_generated_files.sh target

NOTE: If you attempt multiple wheel builds for multiple different Python versions in succession (e.g. first 3.9, then 3.10), use bazel clean --expunge between builds, otherwise something or someone (probably pybind11_bazel) gets cached and proceeds to use the wrong Python headers for the second build, resulting in ABI mismatches and errors upon import.

Just tried this on my new M2 Mac and it fails miserably:

git clone git@github.com:google/ml-metadata.git
cd ml-metadata
git fetch origin pull/166/head:m1wheels
git checkout m1wheels

conda create -n tfx-build python=3.9
conda activate tfx-build
python setup.py bdist_wheel

Getting this error:

/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py:509: InformationOnly: Normalizing '1.11.0.dev' to '1.11.0.dev0'
  self.metadata.version = self._normalize_version(
running bdist_wheel
running build
running bazel_build
ERROR: /private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/bazel_tools/platforms/BUILD:89:6: in alias rule @bazel_tools//platforms:windows: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.
ERROR: /private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/bazel_tools/platforms/BUILD:89:6: Analysis of target '@bazel_tools//platforms:windows' failed
ERROR: /Users/USER/work/tfs-workshop/ml-metadata/ml_metadata/BUILD:54:10: errors encountered resolving select() keys for //ml_metadata:move_generated_files
ERROR: Analysis of target '//ml_metadata:move_generated_files' failed; build aborted:
INFO: Elapsed time: 0.093s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
ERROR: Build failed. Not running target
Traceback (most recent call last):
  File "/Users/USER/work/tfs-workshop/ml-metadata/setup.py", line 126, in <module>
    setup(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 364, in run
    self.run_command("build")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
    self.run_command(cmd_name)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/Users/USER/work/tfs-workshop/ml-metadata/setup.py", line 102, in run
    subprocess.check_call(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/bazel', 'run', '--compilation_mode', 'opt', '--define', 'grpc_no_ares=true', '--verbose_failures', '--macos_minimum_os=11.0', '--config=macos_arm64', '//ml_metadata:move_generated_files']' returned non-zero exit status 1.
gerilya commented 1 year ago

Also tried with bazel 5.3.2 and 5.4.1 versions:

bazel clean --expunge
export USE_BAZEL_VERSION=5.3.2
python setup.py bdist_wheel

which return a different error:

/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py:509: InformationOnly: Normalizing '1.11.0.dev' to '1.11.0.dev0'
  self.metadata.version = self._normalize_version(
running bdist_wheel
running build
running bazel_build
Starting local Bazel server and connecting to it...
INFO: Repository go_sdk instantiated at:
  /Users/USER/work/tfs-workshop/ml-metadata/WORKSPACE:187:23: in <toplevel>
  /private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/io_bazel_rules_go/go/private/sdk.bzl:446:24: in go_register_toolchains
  /private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/io_bazel_rules_go/go/private/sdk.bzl:42:17: in go_host_sdk
Repository rule _go_host_sdk defined at:
  /private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/io_bazel_rules_go/go/private/sdk.bzl:36:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'go_sdk':
   Traceback (most recent call last):
    File "/private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/io_bazel_rules_go/go/private/sdk.bzl", line 31, column 30, in _go_host_sdk_impl
        goroot = _detect_host_sdk(ctx)
    File "/private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/io_bazel_rules_go/go/private/sdk.bzl", line 276, column 13, in _detect_host_sdk
        fail("Could not detect host go version")
Error in fail: Could not detect host go version
ERROR: /Users/USER/work/tfs-workshop/ml-metadata/WORKSPACE:187:23: fetching _go_host_sdk rule //external:go_sdk: Traceback (most recent call last):
    File "/private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/io_bazel_rules_go/go/private/sdk.bzl", line 31, column 30, in _go_host_sdk_impl
        goroot = _detect_host_sdk(ctx)
    File "/private/var/tmp/_bazel_USER/6f81b3df1db13b8e693b00f10d79b1b5/external/io_bazel_rules_go/go/private/sdk.bzl", line 276, column 13, in _detect_host_sdk
        fail("Could not detect host go version")
Error in fail: Could not detect host go version
ERROR: Analysis of target '//ml_metadata:move_generated_files' failed; build aborted: Could not detect host go version
INFO: Elapsed time: 5.704s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (46 packages loaded, 21 targets configured)
FAILED: Build did NOT complete successfully (46 packages loaded, 21 targets configured)
    Fetching @local_config_cc_toolchains; Building xcode-locator
Traceback (most recent call last):
  File "/Users/USER/work/tfs-workshop/ml-metadata/setup.py", line 126, in <module>
    setup(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 364, in run
    self.run_command("build")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
    self.run_command(cmd_name)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/Users/USER/work/tfs-workshop/ml-metadata/setup.py", line 102, in run
    subprocess.check_call(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/tfx-build/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/bazel', 'run', '--compilation_mode', 'opt', '--define', 'grpc_no_ares=true', '--verbose_failures', '--macos_minimum_os=11.0', '--config=macos_arm64', '//ml_metadata:move_generated_files']' returned non-zero exit status 1.
ozadi3 commented 1 year ago

@gerilya followed your steps and on my M1 the build fails as well

gerilya commented 1 year ago

Well, I am new to bazel, but I have experience fixing broken make builds so I thought I would give it a try. It is available here.

I took a wonderful @nicholasjng 's PR and applied it to ml-metadata v1.14.0. I got errors all over the place and after some trial and error, I figured I should pin bazel version to 5.x so I pinned it to 5.3.2. Then I got a bunch of errors which looked like this:

ERROR: /private/var/tmp/_bazel_USER/ff62e4e8ab4b9f2f8a8a3cefc7e2f5d0/external/bazel_tools/platforms/BUILD:89:6: in alias rule @bazel_tools//platforms:windows: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.

I fixed them by following suggestions from PR8622 referenced above. Then I got some more errors which I was able to google the fix for.

The (seemingly last) part I am struggling with is a postgresql build. v1.14 added Postgres support for its metastore and it generates an error:

INFO: From Linking external/com_google_absl/absl/types/libbad_optional_access.a:
warning: /Library/Developer/CommandLineTools/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_absl/absl/types/libbad_optional_access.a the table of contents is empty (no object file members in the library define global symbols)
ERROR: /private/var/tmp/_bazel_USER/ff62e4e8ab4b9f2f8a8a3cefc7e2f5d0/external/postgresql/BUILD.bazel:10:11: Compiling src/port/pg_bitutils.c failed: (Exit 1): cc_wrapper.sh failed: error executing command
  (cd /private/var/tmp/_bazel_USER/ff62e4e8ab4b9f2f8a8a3cefc7e2f5d0/sandbox/darwin-sandbox/262/execroot/ml_metadata && \
  exec env - \
    PATH='/Users/USER/Library/Caches/bazelisk/downloads/sha256/d3ce116379b835e2a0ae16b924a485c48330db434b192f090c8f875238e584cb/bin:/opt/homebrew/share/google-cloud-sdk/bin:/Users/USER/bin:/opt/homebrew/Caskroom/miniconda/base/bin:/opt/homebrew/Caskroom/miniconda/base/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/usr/local/munki:/Applications/Visual Studio Code.app/Contents/Resources/app/bin' \
    PWD=/proc/self/cwd \
  external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/darwin_arm64-opt/bin/external/postgresql/_objs/postgresql/pg_bitutils.pic.d '-frandom-seed=bazel-out/darwin_arm64-opt/bin/external/postgresql/_objs/postgresql/pg_bitutils.pic.o' -fPIC -DFRONTEND '-DHAVE_DECL_STRLCPY=1' '-DHAVE_STRLCPY=1' '-DHAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN=1' '-DHAVE_SYS_UCRED_H=1' -iquote external/postgresql -iquote bazel-out/darwin_arm64-opt/bin/external/postgresql -isystem external/postgresql/config -isystem bazel-out/darwin_arm64-opt/bin/external/postgresql/config -isystem external/postgresql/src/include -isystem bazel-out/darwin_arm64-opt/bin/external/postgresql/src/include -isystem external/postgresql/src/interfaces/libpq -isystem bazel-out/darwin_arm64-opt/bin/external/postgresql/src/interfaces/libpq -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/postgresql/src/port/pg_bitutils.c -o bazel-out/darwin_arm64-opt/bin/external/postgresql/_objs/postgresql/pg_bitutils.pic.o)
# Configuration: 82c9cbc8e4f15c6d82088e6f261ca8145bd41e3909ff5a57fd6a0547d856f851
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from external/postgresql/src/port/pg_bitutils.c:16:
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/cpuid.h:14:2: error: this header is for x86 only
#error this header is for x86 only
 ^
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/cpuid.h:288:5: error: invalid output constraint '=a' in asm
    __cpuid(__leaf, __eax, __ebx, __ecx, __edx);
    ^
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/cpuid.h:253:11: note: expanded from macro '__cpuid'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/cpuid.h:303:5: error: invalid output constraint '=a' in asm
    __cpuid(__leaf, *__eax, *__ebx, *__ecx, *__edx);
    ^
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/cpuid.h:253:11: note: expanded from macro '__cpuid'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/cpuid.h:317:5: error: invalid output constraint '=a' in asm
    __cpuid_count(__leaf, __subleaf, *__eax, *__ebx, *__ecx, *__edx);
    ^
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/cpuid.h:260:11: note: expanded from macro '__cpuid_count'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
external/postgresql/src/port/pg_bitutils.c:206:41: error: invalid output constraint '=q' in asm
__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc");
                                        ^
external/postgresql/src/port/pg_bitutils.c:219:41: error: invalid output constraint '=q' in asm
__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc");
                                        ^
6 errors generated.
Target //ml_metadata:move_generated_files failed to build
INFO: Elapsed time: 56.043s, Critical Path: 24.50s
INFO: 1695 processes: 1439 internal, 256 darwin-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Traceback (most recent call last):
  File "/Users/USER/work/tfs-workshop/ml-metadata-gerilya/setup.py", line 130, in <module>
    setup(
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 325, in run
    self.run_command("build")
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
    self.run_command(cmd_name)
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/Users/USER/work/tfs-workshop/ml-metadata-gerilya/setup.py", line 107, in run
    subprocess.check_call(
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/bazel', 'run', '--compilation_mode', 'opt', '--define', 'grpc_no_ares=true', '--verbose_failures', '--worker_sandboxing', '--macos_minimum_os=11.0', '--config=macos_arm64', '//ml_metadata:move_generated_files']' returned non-zero exit status 1.
tangm commented 11 months ago

Have made a pull request to the original patch by @nicholasjng at https://github.com/nicholasjng/ml-metadata/pull/1 . Also ran into the same postgres issue that @gerilya did, but I've tried to use some select conditions to get the right defines for apple silicons in place. Hope https://github.com/tensorflow/tfx-bsl/issues/48 gets resolved soon too!

thesuperzapper commented 11 months ago

While it's not specifically for MacOS (its actually for Linux ARM), I needed to build gcr.io/tfx-oss-public/ml_metadata_store_server for ARM64, so I made a PR https://github.com/google/ml-metadata/pull/188 that allows that to work (including getting the PostgreSQL extensions to build).

I am interested to know if it also fixes other people's problems building on MacOS ARM.

gerilya commented 11 months ago

@tangm @nicholasjng thank you very much guys for the great work. I just tried to build the updated m1fix branch on my Mac M2 and hit errors again:

git clone git@github.com:nicholasjng/ml-metadata.git
cd ml-metadata
git checkout m1fix
conda create -n m1fix-build python=3.9
conda activate m1fix-build
python setup.py bdist_wheel

Here are the errors:

/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/dist.py:509: InformationOnly: Normalizing '1.15.0.dev' to '1.15.0.dev0'
  self.metadata.version = self._normalize_version(
running bdist_wheel
running build
running bazel_build
ERROR: Traceback (most recent call last):
    File "/private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/internal/rule_support.bzl", line 221, column 36, in <toplevel>
        deps_cfg = apple_common.multi_arch_split,
Error: 'apple_common' value has no field or method 'multi_arch_split'
INFO: Repository build_bazel_rules_swift instantiated at:
  /Users/USER/work/tfs-workshop/ML-METADATA/ml-metadata/WORKSPACE:161:16: in <toplevel>
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/com_github_grpc_grpc/bazel/grpc_extra_deps.bzl:63:29: in grpc_extra_deps
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/repositories.bzl:120:11: in apple_rules_dependencies
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/repositories.bzl:86:14: in _maybe
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
INFO: Repository rules_cc~0.0.9 instantiated at:
  <builtin>: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
INFO: Repository rules_java instantiated at:
  /Users/USER/work/tfs-workshop/ML-METADATA/ml-metadata/WORKSPACE:161:16: in <toplevel>
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/com_github_grpc_grpc/bazel/grpc_extra_deps.bzl:49:18: in grpc_extra_deps
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/com_google_protobuf/protobuf_deps.bzl:65:24: in protobuf_deps
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/com_google_protobuf/protobuf_deps.bzl:19:17: in _github_archive
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: error loading package '@@com_github_grpc_grpc//src/compiler': at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:34:6: at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/ios.bzl:26:5: at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/internal/testing/ios_rules.bzl:22:5: at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/internal/testing/apple_test_bundle_support.bzl:67:5: initialization of module 'apple/internal/rule_support.bzl' failed
ERROR: /Users/USER/work/tfs-workshop/ML-METADATA/ml-metadata/WORKSPACE:164:5: error loading package '@@com_github_grpc_grpc//src/compiler': at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:34:6: at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/ios.bzl:26:5: at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/internal/testing/ios_rules.bzl:22:5: at /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/build_bazel_rules_apple/apple/internal/testing/apple_test_bundle_support.bzl:67:5: initialization of module 'apple/internal/rule_support.bzl' failed and referenced by '//external:grpc_python_plugin'
ERROR: Analysis of target '//ml_metadata:move_generated_files' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.788s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED:
    Fetching repository @@go_sdk; starting
    Fetching repository @@rules_java~7.1.0~toolchains~local_jdk; starting
    Fetching repository @@local_jdk; starting
    Fetching repository @@rules_pkg; starting
    Fetching /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/rules_pkg; Extracting rules_pkg-0.7.0.tar.gz
    Fetching https://github.com/bazelbuild/rules_cc/releases/download/0.0.9/rules_cc-0.0.9.tar.gz
    Fetching https://github.com/bazelbuild/rules_java/archive/981f06c3d2bd10225e85209904090eb7b5fb26bd.zip
Traceback (most recent call last):
  File "/Users/USER/work/tfs-workshop/ML-METADATA/ml-metadata/setup.py", line 131, in <module>
    setup(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 364, in run
    self.run_command("build")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
    self.run_command(cmd_name)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/Users/USER/work/tfs-workshop/ML-METADATA/ml-metadata/setup.py", line 107, in run
    subprocess.check_call(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/bazel', 'run', '--compilation_mode', 'opt', '--define', 'grpc_no_ares=true', '--verbose_failures', '--macos_minimum_os=11.0', '--config=macos_arm64', '//ml_metadata:move_generated_files']' returned non-zero exit status 1.
gerilya commented 11 months ago 
bazel clean --expunge
export USE_BAZEL_VERSION=6.4.0
python setup.py bdist_wheel
external/com_google_zetasql/zetasql/analyzer/rewriters/registration.cc:50:10: warning: passing variable 'registration_order_' by reference requires holding mutex 'mu_' [-Wthread-safety-reference]
  return registration_order_;
         ^
1 warning generated.
INFO: From Compiling ml_metadata/metadata_store/rdbms_metadata_access_object.cc:
ml_metadata/metadata_store/rdbms_metadata_access_object.cc:1178:7: warning: variable 'num_changed_properties' set but not used [-Wunused-but-set-variable]
  int num_changed_properties = 0;
      ^
ml_metadata/metadata_store/rdbms_metadata_access_object.cc:1183:7: warning: variable 'num_changed_custom_properties' set but not used [-Wunused-but-set-variable]
  int num_changed_custom_properties = 0;
      ^
2 warnings generated.
ERROR: /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/rules_foreign_cc/toolchains/BUILD.bazel:130:10: BootstrapGNUMake external/rules_foreign_cc/toolchains/make [for tool] failed: (Exit 1): bash failed: error executing command (from target @rules_foreign_cc//toolchains:make_tool)
  (cd /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata && \
  exec env - \
    PATH='/Users/USER/Library/Caches/bazelisk/downloads/sha256/574d54bfb7e84adf3fe29bc2a0d13214048c1b5e295f826fced3fb94fba282ee/bin:/opt/homebrew/share/google-cloud-sdk/bin:/Users/USER/bin:/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/bin:/opt/homebrew/Caskroom/miniconda/base/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/usr/local/munki:/Applications/Visual Studio Code.app/Contents/Resources/app/bin' \
  /bin/bash -c bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make_tool_foreign_cc/wrapper_build_script.sh)
# Configuration: 2c27fb3c76ce23569653752137cb59fbe91df8b3f0717bc04d0f6be5cbaef63d
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
rules_foreign_cc: Build failed!
rules_foreign_cc: Keeping temp build directory and dependencies directory for debug.
rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify --sandbox_debug Bazel command line flag.
rules_foreign_cc: Printing build logs:
_____ BEGIN BUILD LOGS _____
+ AR=/usr/bin/libtool
+ ARFLAGS='-static -s -o'
+ CC=/private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh
+ CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted'
+ LD=/private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh
+ LDFLAGS='-undefined dynamic_lookup -headerpad_max_install_names -lc++ -lm -undefined error'
+ ./configure --without-guile --with-guile=no --disable-dependency-tracking --prefix=/private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... build-aux/install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh accepts -g... yes
checking for /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh option to accept ISO C89... none needed
checking whether /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh understands -c and -o together... yes
checking dependency style of /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh... none
checking how to run the C preprocessor... /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking whether _XOPEN_SOURCE should be defined... no
checking for gcc... (cached) /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh
checking whether we are using the GNU C compiler... (cached) yes
checking whether /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh accepts -g... (cached) yes
checking for /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh option to accept ISO C89... (cached) none needed
checking whether /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh understands -c and -o together... (cached) yes
checking dependency style of /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh... (cached) none
checking for Minix Amsterdam compiler... no
checking for ar... /usr/bin/libtool
checking for ranlib... ranlib
checking build system type... arm-apple-darwin23.1.0
checking host system type... arm-apple-darwin23.1.0
checking for size_t... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for _set_invalid_parameter_handler... yes
checking for fcntl... yes
checking for symlink... yes
checking for getdtablesize... yes
checking for getprogname... yes
checking for getexecname... yes
checking for sys/socket.h... yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for limits.h... yes
checking for wchar.h... yes
checking for stdint.h... (cached) yes
checking whether // is distinct from /... no
checking whether the preprocessor supports include_next... yes
checking whether system header files limit the line length... no
checking for complete errno.h... yes
checking whether strerror_r is declared... yes
checking for strerror_r... yes
checking whether strerror_r returns char *... no
checking for working fcntl.h... yes
checking for pid_t... yes
checking for mode_t... yes
checking for eaccess... yes
checking whether getdtablesize is declared... yes
checking host CPU and C ABI... arm64
checking whether limits.h has LLONG_MAX, WORD_BIT, ULLONG_WIDTH etc.... no
checking whether malloc, realloc, calloc are POSIX compliant... yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
checking for wchar_t... yes
checking for wint_t... yes
checking whether wint_t is too small... no
checking whether stdint.h conforms to C99... yes
checking whether stdint.h predates C++11... no
checking whether stdint.h has UINTMAX_WIDTH etc.... no
checking for C/C++ restrict keyword... __restrict
checking whether strerror(0) succeeds... no
checking for alloca as a compiler built-in... yes
checking whether // is distinct from /... (cached) no
checking whether dup2 works... yes
checking for error_at_line... no
checking whether fcntl handles F_DUPFD correctly... yes
checking whether fcntl understands F_DUPFD_CLOEXEC... yes
checking whether getdtablesize works... yes
checking for getloadavg... yes
checking sys/loadavg.h usability... no
checking sys/loadavg.h presence... no
checking for sys/loadavg.h... no
checking whether getloadavg is declared... yes
checking whether program_invocation_name is declared... no
checking whether program_invocation_short_name is declared... no
checking whether __argv is declared... no
checking whether __progname is defined in default libraries... yes
checking if system libc has GNU glob... no
checking for ssize_t... yes
checking for good max_align_t... yes
checking whether NULL can be used in arbitrary expressions... yes
checking which flavor of printf attribute matches inttypes macros... system
checking for stpcpy... yes
checking for ranlib... (cached) ranlib
checking how to run the C preprocessor... /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh -E
checking for ar... /usr/bin/libtool
checking for perl... perl
checking for library containing strerror... none required
checking whether byte ordering is bigendian... no
checking for a sed that does not truncate output... /usr/bin/sed
checking whether NLS is requested... yes
checking for msgfmt... /opt/homebrew/bin/msgfmt
checking for gmsgfmt... /opt/homebrew/bin/msgfmt
checking for xgettext... /opt/homebrew/bin/xgettext
checking for msgmerge... /opt/homebrew/bin/msgmerge
checking for ld used by /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh... /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh
checking if the linker (/private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh) is GNU ld... no
checking for shared library run path origin... done
checking for CFPreferencesCopyAppValue... yes
checking for CFLocaleCopyCurrent... yes
checking for GNU gettext in libc... no
checking for iconv... yes
checking for working iconv... no
checking for GNU gettext in libintl... no
checking whether to use NLS... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for library containing getpwnam... none required
checking for ANSI C header files... (cached) yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking whether stat file-mode macros are broken... no
checking whether time.h and sys/time.h may both be included... yes
checking for stdlib.h... (cached) yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking for unistd.h... (cached) yes
checking for limits.h... (cached) yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking for string.h... (cached) yes
checking for memory.h... (cached) yes
checking for sys/param.h... (cached) yes
checking sys/resource.h usability... yes
checking sys/resource.h presence... yes
checking for sys/resource.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking sys/timeb.h usability... yes
checking sys/timeb.h presence... yes
checking for sys/timeb.h... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/file.h usability... yes
checking sys/file.h presence... yes
checking for sys/file.h... yes
checking spawn.h usability... yes
checking spawn.h presence... yes
checking for spawn.h... yes
checking for an ANSI C-conforming const... yes
checking return type of signal handlers... void
checking for uid_t in sys/types.h... yes
checking for pid_t... (cached) yes
checking for off_t... yes
checking for size_t... (cached) yes
checking for ssize_t... yes
checking for unsigned long long int... yes
checking for uintmax_t... yes
checking for nanoseconds field of struct stat... st_mtimespec.tv_nsec
checking whether to use high resolution file timestamps... yes
checking for library containing clock_gettime... none required
checking whether system uses MSDOS-style paths... no
checking for standard gettimeofday... yes
checking for strdup... yes
checking for strndup... yes
checking for memrchr... yes
checking for umask... yes
checking for mkstemp... yes
checking for mktemp... yes
checking for fdopen... yes
checking for dup... yes
checking for dup2... yes
checking for getcwd... yes
checking for realpath... yes
checking for sigsetmask... yes
checking for sigaction... yes
checking for getgroups... yes
checking for seteuid... yes
checking for setegid... yes
checking for setlinebuf... yes
checking for setreuid... yes
checking for setregid... yes
checking for getrlimit... yes
checking for setrlimit... yes
checking for setvbuf... yes
checking for pipe... yes
checking for strsignal... yes
checking for lstat... yes
checking for readlink... yes
checking for atexit... yes
checking for isatty... yes
checking for ttyname... yes
checking for pselect... yes
checking for posix_spawn... yes
checking for posix_spawnattr_setsigmask... yes
checking whether bsd_signal is declared... yes
checking vfork.h usability... no
checking vfork.h presence... no
checking for vfork.h... no
checking for fork... yes
checking for vfork... yes
checking for working fork... yes
checking for working vfork... (cached) yes
checking for strcasecmp... yes
checking for strncasecmp... yes
checking for strcmpi... yes
checking for strncmpi... yes
checking for stricmp... yes
checking for strnicmp... yes
checking for working strcoll... yes
checking whether closedir returns void... no
checking for struct dirent.d_type... yes
checking for pkg-config... no
checking whether sys_siglist is declared... yes
checking whether _sys_siglist is declared... no
checking whether __sys_siglist is declared... no
checking for sys/wait.h... yes
checking for waitpid... yes
checking for wait3... yes
checking for union wait... no
checking for SA_RESTART... yes
checking whether dlopen is declared... yes
checking whether dlsym is declared... yes
checking whether dlerror is declared... yes
checking for library containing dlopen... none required
checking If the linker accepts -Wl,--export-dynamic... no
checking If the linker accepts -rdynamic... yes
checking for posix_spawn that fails synchronously... yes
checking for location of SCCS get command... get
checking if malloc debugging is wanted... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating build.cfg
config.status: creating lib/Makefile
config.status: creating po/Makefile.in
config.status: creating doc/Makefile
config.status: creating tests/config-flags.pm
config.status: creating src/config.h
config.status: executing depfiles commands
config.status: executing po-directories commands
config.status: creating po/POTFILES
config.status: creating po/Makefile
+ cat build.cfg
# Configuration for building GNU Make in the absence of any 'make' program.
# build.cfg.  Generated from build.cfg.in by configure.

# Copyright (C) 1993-2020 Free Software Foundation, Inc.
# This file is part of GNU Make.
#
# GNU Make is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software
# Foundation; either version 3 of the License, or (at your option) any later
# version.
#
# GNU Make is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
# details.
#
# You should have received a copy of the GNU General Public License along with
# this program.  If not, see <http://www.gnu.org/licenses/>.

# See Makefile.in for comments describing these variables.

top_srcdir='.'

prefix='/private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make'
exec_prefix=`eval echo ${prefix}`

CC='/private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata/external/local_config_cc/cc_wrapper.sh'
AR='/usr/bin/libtool'
CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted '
CPPFLAGS=''
DEFS='-DHAVE_CONFIG_H'
ARFLAGS='-static -s -o'
LDFLAGS='-rdynamic -undefined dynamic_lookup -headerpad_max_install_names -lc++ -lm -undefined error'
ALLOCA=''
LOADLIBES='  '
REMOTE='stub'
OBJEXT='o'
EXEEXT=''
+ ./build.sh
config.status: creating ./lib/alloca.h__
config.status: creating ./lib/fcntl.h__
config.status: creating ./lib/limits.h__
config.status: creating ./lib/fnmatch.h__
config.status: creating ./lib/glob.h__
config.status: creating ./lib/stdint.h__
config.status: creating ./lib/stdio.h__
config.status: creating ./lib/stdlib.h__
config.status: creating ./lib/string.h__
config.status: creating ./lib/sys/types.h__
config.status: creating ./lib/unistd.h__
compiling lib/concat-filename.c...
compiling lib/dirname-lgpl.c...
compiling lib/basename-lgpl.c...
compiling lib/stripslash.c...
compiling lib/exitfail.c...
compiling lib/fd-hook.c...
compiling lib/findprog-in.c...
./lib/findprog-in.c:137:25: error: call to undeclared function 'eaccess'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
                    if (eaccess (progpathname, X_OK) == 0)
                        ^
./lib/findprog-in.c:137:25: note: did you mean 'access'?
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:431:6: note: 'access' declared here
int      access(const char *, int);
         ^
./lib/findprog-in.c:211:21: error: call to undeclared function 'eaccess'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
                if (eaccess (progpathname, X_OK) == 0)
                    ^
2 errors generated.
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make_tool_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make_tool_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make_tool_foreign_cc/BootstrapGNUMake.log

Target //ml_metadata:move_generated_files failed to build
ERROR: /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/external/com_google_zetasql/zetasql/parser/BUILD:223:11 Middleman _middlemen/@com_Ugoogle_Uzetasql_S_Szetasql_Sparser_Cbison_Uparser_Ugenerated_Ulib-cc_library-compile failed: (Exit 1): bash failed: error executing command (from target @rules_foreign_cc//toolchains:make_tool)
  (cd /private/var/tmp/_bazel_USER/1aa13aee38960e7ec7f87d23e1054a31/sandbox/darwin-sandbox/25/execroot/ml_metadata && \
  exec env - \
    PATH='/Users/USER/Library/Caches/bazelisk/downloads/sha256/574d54bfb7e84adf3fe29bc2a0d13214048c1b5e295f826fced3fb94fba282ee/bin:/opt/homebrew/share/google-cloud-sdk/bin:/Users/USER/bin:/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/bin:/opt/homebrew/Caskroom/miniconda/base/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/usr/local/munki:/Applications/Visual Studio Code.app/Contents/Resources/app/bin' \
  /bin/bash -c bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make_tool_foreign_cc/wrapper_build_script.sh)
# Configuration: 2c27fb3c76ce23569653752137cb59fbe91df8b3f0717bc04d0f6be5cbaef63d
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 91.707s, Critical Path: 57.96s
INFO: 2489 processes: 1334 internal, 1154 darwin-sandbox, 1 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
Traceback (most recent call last):
  File "/Users/USER/work/tfs-workshop/ML-METADATA/ml-metadata/setup.py", line 131, in <module>
    setup(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 364, in run
    self.run_command("build")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
    self.run_command(cmd_name)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/Users/USER/work/tfs-workshop/ML-METADATA/ml-metadata/setup.py", line 107, in run
    subprocess.check_call(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/m1fix-build/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/bazel', 'run', '--compilation_mode', 'opt', '--define', 'grpc_no_ares=true', '--verbose_failures', '--macos_minimum_os=11.0', '--config=macos_arm64', '//ml_metadata:move_generated_files']' returned non-zero exit status 1.
nicholasjng commented 11 months ago

This is less about the Bazel version, and more about a dependency issue. Could you bazel clean --expunge and try again? (You could also git pull origin m1fix again just to be sure)

Otherwise, @tangm can you share your environment info?