Use tensorflow compression with tensorflow federated on apple silicon

[yingding]

Is your feature request related to a problem? Please describe. During the installation of tensorflow-federated with pip on a MBP with apple silicon

python3 -m pip install tensorflow-macos==2.9.2 tensorflow-metal==0.6.0 tensorflow-federated==0.37.0

I got the error

Collecting tensorflow-federated==0.37.0
  Using cached tensorflow_federated-0.37.0-py2.py3-none-any.whl (903 kB)
ERROR: Could not find a version that satisfies the requirement tensorflow-compression==2.9.2 (from versions: none)
ERROR: No matching distribution found for tensorflow-compression==2.9.2

Describe the solution you'd like how can i get the tensorflow-compression installed on apple silicon? It would be really great if we can have a pre-compiled package also for apple silicon.

Describe alternatives you've considered I am open for any suggestions, would really like to use tensorflow-federated with metal and GPU on M1 apple silicon.

[jonarchist]

Hi! Thanks for reaching out.

Our automated build system currently doesn't support building for M1 (from what I can tell, it might become available Q1 2023), but it would be great to know if you encounter any issues with building it on your system.

Have you tried to follow the instructions for building pip packages under Darwin?

Thanks!

[yingding]

I followed the instruction to build pip packages under Darwin on Apple silicon, need to change the requirements.txt

# tensorflow ~= 2.10.0
tensorflow-macos ~= 2.10.0

Run a test with python3 -m tensorflow_compression.all_tests got

...
[ RUN      ] tensorflow_compression.python.entropy_models.continuous_indexed_test.ContinuousIndexedEntropyModelTest.test_compression_works_in_tf_function
2022-11-02 22:47:00.146878: W tensorflow/core/framework/op_kernel.cc:1780] OP_REQUIRES failed at functional_ops.cc:374 : INTERNAL: No function library
...
Ran 341 tests in 677.321s
FAILED (errors=5, skipped=31)

Imported tensorflow-compression in my python3.10 venv import tensorflow_compression as tfc and got the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/__init__.py", line 17, in <module>
    from tensorflow_compression.python import distributions
  File "/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/python/distributions/__init__.py", line 19, in <module>
    from tensorflow_compression.python.distributions.round_adapters import *
  File "/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/python/distributions/round_adapters.py", line 22, in <module>
    from tensorflow_compression.python.ops import round_ops
  File "/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/python/ops/__init__.py", line 17, in <module>
    from tensorflow_compression.python.ops.gen_ops import *
  File "/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/python/ops/gen_ops.py", line 20, in <module>
    ops = load_library.load_op_library(resource_loader.get_path_to_datafile(
  File "/Users/yingding/VENV/metal3.10/lib/python3.10/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: dlopen(/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/python/ops/../../cc/libtensorflow_compression.so, 0x0006): tried: '/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/python/ops/../../cc/libtensorflow_compression.so' (no such file), '/Users/yingding/VCS/build/tensorflow_compression/tensorflow_compression/cc/libtensorflow_compression.so' (no such file)

Any thoughts on this? Would be really great to have pre-compiled pages for apple silicon.

[jonarchist]

If you didn't get an error building the package, then that seems promising!

It looks like it's trying to access libtensorflow_compression.so, but in Darwin, the name should end in .dylib. Perhaps under M1, Darwin uses a different OS identifier here (instead of mac): https://github.com/tensorflow/compression/blob/master/workspace.bzl#L24

You might want to patch that file to use .dylib. Can you see if that helps?

Also, we recently needed to update this to read .2.dylib. In 2.10.0 it might still just be .dylib.

[ghost]

Did anyone get it built on apple silicon?