Can not build `tensorflow-io` from source on Apple M1

[sun1638650145]

1. First I tried using python setup.py bdist_wheel, which builds wheel, but the installation prompts me

   ERROR: Could not find a version that satisfies the requirement tensorflow-io-gcs-filesystem==0.24.0 (from tensorflow-io) (from versions: none)
   ERROR: No matching distribution found for tensorflow-io-gcs-filesystem==0.24.0

   So I changed it to install pip install dist/tensorflow_io-0.24.0-cp38-cp38-macosx_11_0_arm64.whl --no-deps without dependencies, so although it can be installed normally, it will prompt me that there is no dynamic library when running

   /Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
   caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
    warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
   /Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io.so']
   caused by: ["dlopen(/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/usr/local/lib/libtensorflow_io.so' (no such file), '/usr/lib/libtensorflow_io.so' (no such file)"]
    warnings.warn(f"file system plugins are not loaded: {e}")

2. After the above method fails, I refer to the build method in the documentation, unfortunately, run this command bazel build -s --verbose_failures $BAZEL_OPTIMIZATION //tensorflow_io/... //tensorflow_io_gcs_filesystem/... will produce the following error:

   ERROR: While resolving toolchains for target //tensorflow_io/core:golang_ops: no matching toolchains found for types @io_bazel_rules_go//go:toolchain
   ERROR: Analysis of target '//tensorflow_io/core:golang_ops' failed; build aborted: no matching toolchains found for types @io_bazel_rules_go//go:toolchain

   So, what should I do?

[sun1638650145]

There is some information about the development environment.

[ZaibanAli]

First, goto to the tensorflow/io root folder.

Second, do:

rm -r dist/*
export MACOSX_DEPLOYMENT_TARGET="$(sw_vers -productVersion)"

Third, on the same bash shell do:

python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
pip install --no-dependencies dist/tensorflow_io_gcs_filesystem-0.24.0-cp38-cp38-macosx_12_0_arm64.whl

Fourth,

python3 setup.py -q bdist_wheel
pip install --no-dependencies dist/tensorflow_io-0.24.0-cp38-cp38-macosx_12_0_arm64.whl

What I know currently is that Bazel build is not supported for mac m1 (for tensorflow/io).

[sun1638650145]

@ZaibanAli Thank you very much for your help, but still there is no dynamic library error.

[ZaibanAli]

@sun1638650145 Okay so I tested it again and I am getting this error too when importing tensorflow_io or when using object detection api.

>>> import tensorflow_io
/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ["dlopen(/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/usr/local/lib/libtensorflow_io.so' (no such file), '/usr/lib/libtensorflow_io.so' (no such file)"]
  warnings.warn(f"file system plugins are not loaded: {e}")
>>> print(tensorflow_io.__version__)
0.24.0

[yongtang]

The tensorflow-io will need to have .so file complied from C++ code in order for file system plugins to work. But in order to compile C++ code it will need to have tensorflow installed. @ZaibanAli @sun1638650145 Which tensorflow package are you using on M1?

[ZaibanAli]

@yongtang I am using this https://github.com/apple/tensorflow_macos/releases/tag/v0.1alpha3

All other things are working correctly, the issues only occurs when I try to use tensorlfow-io. (in my case it is required by tensorflow object detection api)

[sun1638650145]

@yongtang I'm using the stable version of tensorflow_macos.

[ZaibanAli]

Update: I am able to run object detection api even with this warning, the problem was that I had previously installed the old TensorFlow addon package. Updating the TensorFlow addon package to the newest resolved the issue.

[yongtang]

@ZaibanAli @sun1638650145 My recent M1 laptop is having a firmware issue and is with apple repair now. I may only be able to take a look at building tensorflow-io after the laptop is back.

Before the firmware issue I was using pip to install tensorflow_macos though I remember I had an issue with hdf5 dependency. Do you encounter the same hdf5 dependencies?

[ZaibanAli]

@yongtang no dependency issues on my side.

[sun1638650145]

@yongtang hdf5 build depends on numpy, luckily, numpy has provided precompiled packages since 1.21.0, and building from source is also very easy.

[dvaldivia]

trying the steps suggested to compile fails, using python 3.9, is it needed to use python 3.8?

(base)  io git:(master) ✗ 
➜ pip install --no-dependencies dist/tensorflow_io_gcs_filesystem-0.24.0-cp39-cp39-macosx_12_2_arm64.whl 
ERROR: tensorflow_io_gcs_filesystem-0.24.0-cp39-cp39-macosx_12_2_arm64.whl is not a supported wheel on this platform.
(base)  io git:(master) ✗ 
➜ pip install --no-dependencies dist/tensorflow_io-0.24.0-cp39-cp39-macosx_12_2_arm64.whl               
ERROR: tensorflow_io-0.24.0-cp39-cp39-macosx_12_2_arm64.whl is not a supported wheel on this platform.

[ZaibanAli]

@dvaldivia I had the same issue with python 3.9. Try it with python 3.8.

The reason I found out was that TensorFlow .whl files are available for cp38 only for macos (tensorflow_macos/v0.1alpha3).

[yongtang]

I am able to get my M1 laptop back. However, while I successfully installed tensorflow_macos, during the bazel build the following issues popping up:

ERROR: /private/var/tmp/_bazel_yongtang/c33235605c366b16ba32c7f8a25ff251/external/local_config_cc/BUILD:48:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'darwin_arm64'
ERROR: /private/var/tmp/_bazel_yongtang/c33235605c366b16ba32c7f8a25ff251/external/local_config_cc/BUILD:48:19: Analysis of target '@local_config_cc//:toolchain' failed
INFO: Repository curl instantiated at:

It looks like there is still some compatibility issue from bazel, either due to bazel itself, or due to some third party libraries (e.g., tensorflow)'s bazel BUILD. Will have to dig further.

[sun1638650145]

@yongtang I also had type problems when building with bazel, but now bazel has native support for M1, and can build tf, tf-addons, tf-text perfectly. So why is this?

[yongtang]

The only difference I can see, is that I haven't installed XCode (only use command line tools). I can try install xcode.

[leochoo]

@yongtang Any update yet? I'm running into this issue as well. It does not let me use tfio.audio as a result.

What would be a workaround you recommend?

[npanpaliya]

The tensorflow-io will need to have .so file complied from C++ code in order for file system plugins to work. But in order to compile C++ code it will need to have tensorflow installed. @ZaibanAli @sun1638650145 Which tensorflow package are you using on M1?

Hello, I'm also facing the same issue while importing tensorflow-io 0.27.0. What exactly needs to be done to resolve these warnings?

[guites]

Hello, I'm also facing the same issue while importing tensorflow-io 0.27.0. What exactly needs to be done to resolve these warnings?

I'm receiving the same warnings

(tfod-api-cpu) guites@macos io % python                   
Python 3.9.15 (main, Nov 24 2022, 08:28:41) 
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow_io
~/another-test/io/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['~/another-test/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '~/another-test/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
~/another-test/io/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ["dlopen(~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/guites/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file)"]
  warnings.warn(f"file system plugins are not loaded: {e}")
>>> tensorflow_io.__version__
'0.26.0'

Tried using an arm64 tensorflow_addons wheel as provided by @sun1638650145, but the warnings persist.

still, I am able to run objecte detection jobs (https://www.tensorflow.org/hub/tutorials/tf2_object_detection).

[cmditch]

I'm seeing the same error you're seeing @guites. This is happening inside a docker container (dev container) on an M1 mac.

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow_io
/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: cannot open shared object file: No such file or directory']
  warnings.warn(f"file system plugins are not loaded: {e}")
>>> tensorflow_io.__version__
'0.27.0'

[ALEEF02]

Just wanted to chime in that I am also having these issues but on the base tensorflow. Seems to run fine, checking tensorflow.__version__ yields 2.11.0.

[sun1638650145]

I built tensorflow from source, without using tensorflow-macos, but I'm still having issues building tensorflow-io.

[gopal-tathe]

Hi, any update on this issue, getting same error.

[LeoBlackCat]

the same error...

[TiagoGouvea]

I'm trying to make it fully work for three days, but still have those errors.

[TiagoGouvea]

Looking at many questions on stackoverflow, with 5k view, 3 answers, and 6 up votes on the best answer, make me convinced that it just doesn't work for the major of devs trying.

[kulinseth]

I built tensorflow from source, without using tensorflow-macos, but I'm still having issues building tensorflow-io.

Hi @sun1638650145 and @TiagoGouvea , can you please paste the build error ?

[TiagoGouvea]

I'm getting this message on every run, and it takes some like 5 seconds. After that the script follows.

/Users/tiagogouvea/python/io/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/Users/tiagogouvea/python/io/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ["dlopen(/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file)"]
  warnings.warn(f"file system plugins are not loaded: {e}")

Using a conda environment, this is my build and install results:

> python setup.py build
Project: tensorflow-io
Exclude: ['tests', 'tests.*', 'tensorflow_io_gcs_filesystem', 'tensorflow_io_gcs_filesystem.*']
Install Requires: ['tensorflow-io-gcs-filesystem==0.32.0']
Project Rootpath: tensorflow_io
running build
running build_py
running build_ext
> ython setup.py install
Project: tensorflow-io
Exclude: ['tests', 'tests.*', 'tensorflow_io_gcs_filesystem', 'tensorflow_io_gcs_filesystem.*']
Install Requires: ['tensorflow-io-gcs-filesystem==0.32.0']
Project Rootpath: tensorflow_io
running install
/Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
running bdist_egg
running egg_info
writing tensorflow_io.egg-info/PKG-INFO
writing dependency_links to tensorflow_io.egg-info/dependency_links.txt
writing requirements to tensorflow_io.egg-info/requires.txt
writing top-level names to tensorflow_io.egg-info/top_level.txt
reading manifest file 'tensorflow_io.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'tensorflow_io.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-11.1-arm64/egg
running install_lib
running build_py
running build_ext
creating build/bdist.macosx-11.1-arm64/egg
....
copying tensorflow_io.egg-info/top_level.txt -> build/bdist.macosx-11.1-arm64/egg/EGG-INFO
creating 'dist/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg' and adding 'build/bdist.macosx-11.1-arm64/egg' to it
removing 'build/bdist.macosx-11.1-arm64/egg' (and everything under it)
Processing tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg
removing '/Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg' (and everything under it)
creating /Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg
Extracting tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg to /Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages
Adding tensorflow-io 0.32.0 to easy-install.pth file

Installed /Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg
Processing dependencies for tensorflow-io==0.32.0
Searching for tensorflow-io-gcs-filesystem==0.32.0
Reading https://pypi.org/simple/tensorflow-io-gcs-filesystem/
No local packages or working download links found for tensorflow-io-gcs-filesystem==0.32.0
error: Could not find suitable distribution for Requirement.parse('tensorflow-io-gcs-filesystem==0.32.0')

Thank you for your attention @kulinseth

[sun1638650145]

Now, io can be built perfectly on Apple silicon. If anyone needs it, you can find the tutorial here. I hope it can be helpful to everyone.

[flippinroo2]

here

Any chance we can re-open this until a MR is made to fix this for anyone who has a MacBook after 2020...?

I really don't want to have to add anymore new brew packages and then modify a few lines in the source code too :/

[sun1638650145]

@flippinroo2 tensorflow does not seem to be planning official support for Apple silicon. I have provided several tensorflow libraries whl files, and if you need them, you can download them directly from here.

[flippinroo2]

@flippinroo2 tensorflow does not seem to be planning official support for Apple silicon. I have provided several tensorflow libraries whl files, and if you need them, you can download them directly from here.

WOW! Thanks so much dude. This helps A LOT!