search

google-deepmind / reverb

Reverb is an efficient and easy-to-use data storage and transport system designed for machine learning research
Apache License 2.0
700 stars 93 forks source link

Building wheel on Apple M1 #31

Closed markub3327 closed 3 years ago

markub3327 commented 3 years ago

I tried to compile wheel direct on macOS Big Sur, but I get an error:

Starting local Bazel server and connecting to it... INFO: Repository python_includes instantiated at: /Users/martin/Projects/reverb/WORKSPACE:82:16: in /Users/martin/Projects/reverb/reverb/cc/platform/default/repo.bzl:300:25: in cc_tf_configure Repository rule unexported__python_includes_repo_impl defined at: /Users/martin/Projects/reverb/WORKSPACE:82:16: in /Users/martin/Projects/reverb/reverb/cc/platform/default/repo.bzl:297:43: in cc_tf_configure ERROR: An error occurred during the fetch of repository 'python_includes': Traceback (most recent call last): File "/Users/martin/Projects/reverb/reverb/cc/platform/default/repo.bzl", line 237, column 43, in _python_includes_repo_impl python_solib = _find_python_solib_path(repo_ctx) File "/Users/martin/Projects/reverb/reverb/cc/platform/default/repo.bzl", line 98, column 13, in _find_python_solib_path fail("Could not locate python shared library path:\n{}" Error in fail: Could not locate python shared library path: src/main/tools/process-wrapper-legacy.cc:80: "execvp(python3.8-config, ...)": No such file or directory ERROR: Error fetching repository: Traceback (most recent call last): File "/Users/martin/Projects/reverb/reverb/cc/platform/default/repo.bzl", line 237, column 43, in _python_includes_repo_impl python_solib = _find_python_solib_path(repo_ctx) File "/Users/martin/Projects/reverb/reverb/cc/platform/default/repo.bzl", line 98, column 13, in _find_python_solib_path fail("Could not locate python shared library path:\n{}" Error in fail: Could not locate python shared library path: src/main/tools/process-wrapper-legacy.cc:80: "execvp(python3.8-config, ...)": No such file or directory INFO: Repository bazel_skylib instantiated at: /Users/martin/Projects/reverb/WORKSPACE:58:10: in /private/var/tmp/_bazel_martin/79641e1633fab9ab92513b178e080b75/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:211:21: in grpc_deps Repository rule http_archive defined at: /private/var/tmp/_bazel_martin/79641e1633fab9ab92513b178e080b75/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in INFO: Repository build_bazel_rules_swift instantiated at: /Users/martin/Projects/reverb/WORKSPACE:68:25: in /private/var/tmp/_bazel_martin/79641e1633fab9ab92513b178e080b75/external/build_bazel_rules_apple/apple/repositories.bzl:117:11: in apple_rules_dependencies /private/var/tmp/_bazel_martin/79641e1633fab9ab92513b178e080b75/external/build_bazel_rules_apple/apple/repositories.bzl:84:14: in _maybe Repository rule http_archive defined at: /private/var/tmp/_bazel_martin/79641e1633fab9ab92513b178e080b75/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in ERROR: /private/var/tmp/_bazel_martin/79641e1633fab9ab92513b178e080b75/external/tensorflow_solib/BUILD:2:11: @tensorflow_solib//:framework_lib depends on @python_includes//:python_includes in repository @python_includes which failed to fetch. no such package '@python_includes//': Could not locate python shared library path: src/main/tools/process-wrapper-legacy.cc:80: "execvp(python3.8-config, ...)": No such file or directory ERROR: /private/var/tmp/_bazel_martin/79641e1633fab9ab92513b178e080b75/external/tensorflow_solib/BUILD:2:11: @tensorflow_solib//:framework_lib depends on @python_includes//:numpy_includes in repository @python_includes which failed to fetch. no such package '@python_includes//': Could not locate python shared library path: src/main/tools/process-wrapper-legacy.cc:80: "execvp(python3.8-config, ...)": No such file or directory ERROR: Analysis of target '//reverb/pip_package:build_pip_package' failed; build aborted: Analysis failed INFO: Elapsed time: 29.788s INFO: 0 processes. FAILED: Build did NOT complete successfully (46 packages loaded, 253 targets configured) currently loading: @com_github_grpc_grpc// Fetching @protobuf_archive; fetching Fetching @eigen_archive; fetching

Using

Xcode CLI Python 3.8.2 Numpy 1.18.5 Mac-optimized TensorFlow r2.4rc0 bazel 4.0.0-homebrew (installed by HomeBrew)

I need a wheel natively for Apple M1 chip. Thanks.

acassirer commented 3 years ago

Hey,

Reverb currently only support Linux.

Please see https://github.com/deepmind/reverb/issues/11 for more details.

markub3327 commented 3 years ago

@acassirer When will Reverb supports the macOS (because it's Linux like system)? Thanks.

acassirer commented 3 years ago

I'm afraid that support for MacOS is not something that on the roadmap for Reverb. If the community contributes a solution (e.g #24) then we're happy to work with the author to incorporate the functionality but I'm unable to give you a date to look forward to.

markub3327 commented 3 years ago

@acassirer Can I have a question about support Ubuntu 20.04.2 LTS running on Raspberry Pi 4 aarch64? Is this not on your roadmap too?

I get an error:

ubuntu@ubuntu:~$ pip3 install dm-reverb ERROR: Could not find a version that satisfies the requirement dm-reverb (from versions: none) ERROR: No matching distribution found for dm-reverb

acassirer commented 3 years ago

Hey,

No we definitely have no intentions of explicitly supporting raspberry PI and its distributions of Linux.

That being said, we are building the wheel with manylinux2010_x86_64. @ebrevdo, do you know if manylinux2014 is possible? Looks like aarch64 should be covered then.

ebrevdo commented 3 years ago

We need to build using the same crosstool toolchain as TF stable / tf nightly. +Amit Patankar @.***> what are tf plans in this regard? Are we behind TF?

On Fri, Mar 12, 2021, 1:30 AM Albin Cassirer @.***> wrote:

Hey,

No we definitely have no intentions of explicitly supporting raspberry PI and its distributions of Linux.

That being said, we are building the wheel with manylinux2010_x86_64. @ebrevdo https://github.com/ebrevdo, do you know if manylinux2014 is possible? Looks like aarch64 should be covered https://www.python.org/dev/peps/pep-0599 then.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deepmind/reverb/issues/31#issuecomment-797362285, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANWFG2JHW6ZXFJJ7IDT2C3TDHNL5ANCNFSM4YZFK3PQ .

ebrevdo commented 3 years ago

+tensorflow-devinfra-team @.***> AFAIK we have no plans to migrate to manylinux2014 anytime soon. We still build with manylinux 2010. The SIG Build discussion group https://github.com/tensorflow/build#community may have more knowledge regarding this transition.

-Amit

On Fri, Mar 12, 2021 at 8:43 AM Eugene Brevdo @.***> wrote:

We need to build using the same crosstool toolchain as TF stable / tf nightly. +Amit Patankar @.***> what are tf plans in this regard? Are we behind TF?

On Fri, Mar 12, 2021, 1:30 AM Albin Cassirer @.***> wrote:

Hey,

No we definitely have no intentions of explicitly supporting raspberry PI and its distributions of Linux.

That being said, we are building the wheel with manylinux2010_x86_64. @ebrevdo https://github.com/ebrevdo, do you know if manylinux2014 is possible? Looks like aarch64 should be covered https://www.python.org/dev/peps/pep-0599 then.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deepmind/reverb/issues/31#issuecomment-797362285, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANWFG2JHW6ZXFJJ7IDT2C3TDHNL5ANCNFSM4YZFK3PQ .

cemlyn007 commented 1 year ago

Good morning, was wondering if this stance might have changed?

cemlyn007 commented 1 year ago

I have got the latest version of Reverb to compile for Mac x86_64 and arm64. I could do an MR if people are interested?