apple / tensorflow_macos

TensorFlow for macOS 11.0+ accelerated using Apple's ML Compute framework.
Other
3.67k stars 309 forks source link

how to install opencv2 in M1 #67

Open vivek-ram opened 3 years ago

vivek-ram commented 3 years ago

was getting an error when installing this directly with pip commend

jjbenes commented 3 years ago

I assume you have Xcode and tensorflow_macos. You should also turn off Anaconda if you're using it. You need to compile cv2 from source.

  1. Stay in the virtual environment for tensorflow_macos. a. Make sure that you're not also in a Conda environment. b. Make sure that file $(which python) says that python is a universal binary.
  2. You need your own homebrew in /opt/homebrew (not the one in /usr/local/bin.) See Alternative Installs here.
  3. Get cmake to compile cv2. /opt/homebrew/bin/brew install cmake
  4. Compile cv2 from source. Assuming that your tensorflow_macos virtual environment is at ~/tensorflow_macos_venv, follow the steps here except run cmake like this:
    cmake -DCMAKE_BUILD_TYPE=Release \
    -DPYTHON3_EXECUTABLE=`which python` \
    -DPYTHON3_NUMPY_INCLUDE_DIR=~/tensorflow_macos_venv/lib/python3.8/site-packages/numpy/core/include \
    -DPYTHON3_INCLUDE_DIR=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Headers \
    -DBUILD_opencv_python2=OFF \
    -DBUILD_opencv_python3=ON \
    -DCMAKE_INSTALL_PREFIX=~/tensorflow_macos_venv \
    <your OpenCV source directory>
  5. Run make or make -j7 as the link suggests.
  6. Run make install. cv2 should now be in ~/tensorflow_macos_venv/lib/python3.8/site-packages/cv2 for python to import.
vivek-ram commented 3 years ago

my virtual environment for Tensorflow_macos is from Conda.

jjbenes commented 3 years ago

Apple isn't ready to support tensorflow_macos with Conda yet. For the time being, if you don't mind, turn off Conda. I ran into problems when I got my Conda and TF virtual environment mixed up. For now, I move my ~/.bash_profile aside (because it sets up Conda) and open a new shell for the TensorFlow virtual environment. I assume this will all get straightened out in due time.

vivek-ram commented 3 years ago

kinda confusing bro mind u can message me in discord ?? I got few issues when I doing this environment thingy coz python 3.9 don't support it right. here is my discord vivek#2200

vivek-ram commented 3 years ago

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

getting this error when trying to install Tensorflow_mac

jjbenes commented 3 years ago

Not supposed to be Python 3.9. Apple wants us to use Python 3.8 under Xcode. Don't use Google's installation instructions since you're installing an alpha version of M1 TensorFlow. (If not, then you posted your question in the wrong place.) Did you install TF for M1 by following the steps from Apple here?

BTW, you'll want to get Xcode from the App Store if you don't have it yet. After you get Xcode, you'll need to sign a license (sudo xcodebuild -license).

Unfortunately, the software packages are not all ready for developers. You should be able to install cv2 for the M1 TF if you follow the 6 steps I outlined above.

vivek-ram commented 3 years ago

yeah I installed Tensorflow_mac in Conda environment but from that environment I can't install cv2

vivek-ram commented 3 years ago

can u give the steps for Tensorflow_mac from scratch coz I was getting the error when testing the Tensorflow.

Traceback (most recent call last): File "/Users/vivekd/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 64, in from tensorflow.python._pywrap_tensorflow_internal import * ImportError: dlopen(/Users/vivekd/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image found. Did find: /Users/vivekd/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture /Users/vivekd/tensorflow_macos_venv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture

During handling of the above exception, another exception occurred:

vivek-ram commented 3 years ago

is this for M1 Mac or I installed for intel ?? /Users/vivekd/tensorflow_macos_venv/bin/python: Mach-O 64-bit executable x86_64

jjbenes commented 3 years ago

It says x86, so just for Intel and the wrong version. You want to use the one from Xcode. See file $(which python) from step 1b above. The correct Python 3.8 returns:

Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]...

Mine came from /Library/Developer/CommandLineTools/usr/bin/python3. I expect you have the same one, too, if you have Xcode.

vivek-ram commented 3 years ago

mine is coming from /Users/vivekd/opt/anaconda3/bin/python: Mach-O 64-bit executable x86_64

vivek-ram commented 3 years ago

@jjbenes bro manage to install it but getting this issue

Screenshot 2020-12-14 at 9 13 53 am
yeedo-sec commented 3 years ago

I assume you have Xcode and tensorflow_macos. You should also turn off Anaconda if you're using it. You need to compile cv2 from source.

  1. Stay in the virtual environment for tensorflow_macos. a. Make sure that you're not also in a Conda environment. b. Make sure that file $(which python) says that python is a universal binary.
  2. You need your own homebrew in /opt/homebrew (not the one in /usr/local/bin.) See Alternative Installs here.
  3. Get cmake to compile cv2. /opt/homebrew/bin/brew install cmake
  4. Compile cv2 from source. Assuming that your tensorflow_macos virtual environment is at ~/tensorflow_macos_venv, follow the steps here except run cmake like this:
cmake -DCMAKE_BUILD_TYPE=Release \
-DPYTHON3_EXECUTABLE=`which python` \
-DPYTHON3_NUMPY_INCLUDE_DIR=~/tensorflow_macos_venv/lib/python3.8/site-packages/numpy/core/include \
-DPYTHON3_INCLUDE_DIR=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Headers \
-DBUILD_opencv_python2=OFF \
-DBUILD_opencv_python3=ON \
-DCMAKE_INSTALL_PREFIX=~/tensorflow_macos_venv \
<your OpenCV source directory>
  1. Run make or make -j7 as the link suggests.
  2. Run make install. cv2 should now be in ~/tensorflow_macos_venv/lib/python3.8/site-packages/cv2 for python to import.

i finished the step 6 but when i try to import cv2,No module named 'cv2' and i checked my ~/tensorflow_macos_venv/lib/python3.8/site-packages cv2 is not here

jjbenes commented 3 years ago

@Lucife5604 Did cv2 somehow get into /usr/local/lib/python3.8/site-packages/cv2? If so, make a link from ~/tensorflow_macos_venv/lib/python3.8/site-packages to it. Also, check where you're getting cmake from. which cmake should return /opt/homebrew/bin/cmake, the one you installed in step 3.

yeedo-sec commented 3 years ago

@Lucife5604 Did cv2 somehow get into /usr/local/lib/python3.8/site-packages/cv2? If so, make a link from ~/tensorflow_macos_venv/lib/python3.8/site-packages to it. Also, check where you're getting cmake from. which cmake should return /opt/homebrew/bin/cmake, the one you installed in step 3.

i checked the /usr/local/lib/python3.8/site-packages but nothing there,and the cmake is correct located in opt/homebrew/bin

vivek-ram commented 3 years ago

can we install opencv in Conda environment ?

beraekoklu commented 3 years ago

I also have this problem. cv2 is somehow in ~/tensorflow_macos_venv/lib/python3.8, not in site packages.

bjajoh commented 3 years ago

I got opencv and conda working correctly. I documented my procedure here and will add more in the future. https://github.com/bjajoh/cv2-tf-scipy-for-arm-mac/blob/main/README.md

This is what I came up with:

cmake -DCMAKE_BUILD_TYPE=Release \
-DBUILD_opencv_python2=OFF \
-DBUILD_opencv_python3=ON \
-DPYTHON3_EXECUTABLE=`which python3` \
-DPYTHON3_INCLUDE_DIR=~/miniforge3/envs/python38/include/python3.8 \
-DPYTHON3_PACKAGES_PATH=~/miniforge3/envs/python38/lib/python3.8 \
-DPYTHON3_NUMPY_INCLUDE_DIR=~/miniforge3/envs/python38/lib/python3.8/site-packages/numpy/core \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DOPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-DOPENCV_ENABLE_NONFREE=ON \
../opencv
lokinfey commented 3 years ago

I got opencv and conda working correctly. I documented my procedure here and will add more in the future. https://github.com/bjajoh/cv2-tf-scipy-for-arm-mac/blob/main/README.md

This is what I came up with:

cmake -DCMAKE_BUILD_TYPE=Release \
-DBUILD_opencv_python2=OFF \
-DBUILD_opencv_python3=ON \
-DPYTHON3_EXECUTABLE=`which python3` \
-DPYTHON3_INCLUDE_DIR=~/miniforge3/envs/python38/include/python3.8 \
-DPYTHON3_PACKAGES_PATH=~/miniforge3/envs/python38/lib/python3.8 \
-DPYTHON3_NUMPY_INCLUDE_DIR=~/miniforge3/envs/python38/lib/python3.8/site-packages/numpy/core \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DOPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-DOPENCV_ENABLE_NONFREE=ON \
../opencv

I try your method but if show me

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed. Compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ Build flags: Id flags:

The output was: 1 ld: library not found for -lc++ clang: error: linker command failed with exit code 1 (use -v to see invocation)

Compiling the C compiler identification source file "CMakeCCompilerId.c" failed. Compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc Build flags: Id flags:

The output was: 1 ld: library not found for -lSystem clang: error: linker command failed with exit code 1 (use -v to see invocation)

isuruf commented 3 years ago

You can now install opencv using conda. conda install opencv

jessicaadong commented 3 years ago

@Lucife5604 Did cv2 somehow get into /usr/local/lib/python3.8/site-packages/cv2? If so, make a link from ~/tensorflow_macos_venv/lib/python3.8/site-packages to it. Also, check where you're getting cmake from. which cmake should return /opt/homebrew/bin/cmake, the one you installed in step 3.

I replaced cmake with /opt/homebrew/bin/cmake in your instruction to build opencv. It built successfully but the library was installed at /usr/local/lib/python3.8/site-packages/cv2/cv2.cpython-38-darwin.so. I created a symbolic link to it at ~/tensorflow_macos_venv/lib/python3.8/site-packages, but I have:

(tensorflow_macos_venv) qingdong@Qings-MacBook-Pro site-packages % python
Python 3.8.2 (default, Nov  4 2020, 21:23:28) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/qingdong/tensorflow_macos_venv/lib/python3.8/site-packages/cv2.so, 2): no suitable image found.  Did find:
    /Users/qingdong/tensorflow_macos_venv/lib/python3.8/site-packages/cv2.so: mach-o, but wrong architecture
    /usr/local/lib/python3.8/site-packages/cv2/cv2.cpython-38-darwin.so: mach-o, but wrong architecture

Any idea what is happening and how to fix it?

hannesa2 commented 3 years ago
(tensorflow_venv) ➜  city-simulator git:(master) ✗ python
Python 3.8.8 (default, Feb 26 2021, 09:10:27) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

When I install open-cv with pip3 install opencv-python I run into following

(tensorflow_venv) ➜  city-simulator git:(master) ✗ pip3 install opencv-python                  
Collecting opencv-python
  Using cached opencv-python-4.5.1.48.tar.gz (88.3 MB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/hannes/git/car-ML/city-simulator/tensorflow_venv/bin/python3 /Users/hannes/git/car-ML/city-simulator/tensorflow_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/np/7ckckk8s5fd169kvnnfhjkk40000gr/T/pip-build-env-z_h51af_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel scikit-build cmake pip 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_machine != '"'"'aarch64'"'"'' 'numpy==1.19.3; python_version>='"'"'3.6'"'"' and sys_platform == '"'"'linux'"'"' and platform_machine == '"'"'aarch64'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_machine != '"'"'aarch64'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_machine != '"'"'aarch64'"'"'' 'numpy==1.19.3; python_version>='"'"'3.9'"'"' and platform_machine != '"'"'aarch64'"'"''
       cwd: None
  Complete output (556 lines):
  Ignoring numpy: markers 'python_version == "3.6" and platform_machine != "aarch64"' don't match your environment
  Ignoring numpy: markers 'python_version >= "3.6" and sys_platform == "linux" and platform_machine == "aarch64"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.7" and platform_machine != "aarch64"' don't match your environment
  Ignoring numpy: markers 'python_version >= "3.9" and platform_machine != "aarch64"' don't match your environment
  Collecting setuptools
    Using cached setuptools-54.1.2-py3-none-any.whl (785 kB)
  Collecting wheel
    Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
  Collecting scikit-build
    Using cached scikit_build-0.11.1-py2.py3-none-any.whl (72 kB)
  Collecting cmake
    Using cached cmake-3.18.4.post1.tar.gz (28 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting pip
    Using cached pip-21.0.1-py3-none-any.whl (1.5 MB)
  Collecting numpy==1.17.3
    Using cached numpy-1.17.3.zip (6.4 MB)
  Collecting distro
    Using cached distro-1.5.0-py2.py3-none-any.whl (18 kB)
  Collecting packaging
    Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
  Collecting pyparsing>=2.0.2
    Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
  Building wheels for collected packages: numpy, cmake
    Building wheel for numpy (setup.py): started
    Building wheel for numpy (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/hannes/git/car-ML/city-simulator/tensorflow_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/np/7ckckk8s5fd169kvnnfhjkk40000gr/T/pip-install-2ugu5tsn/numpy_ba03cd416255443d84b89beb97693a9c/setup.py'"'"'; __file__='"'"'/private/var/folders/np/7ckckk8s5fd169kvnnfhjkk40000gr/T/pip-install-2ugu5tsn/numpy_ba03cd416255443d84b89beb97693a9c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/np/7ckckk8s5fd169kvnnfhjkk40000gr/T/pip-wheel-lps1mtiu
         cwd: /private/var/folders/np/7ckckk8s5fd169kvnnfhjkk40000gr/T/pip-install-2ugu5tsn/numpy_ba03cd416255443d84b89beb97693a9c/
    Complete output (239 lines):
    Running from numpy source directory.
    blas_opt_info:
    blas_mkl_info:
    customize UnixCCompiler
      libraries mkl_rt not found in ['/Users/hannes/git/car-ML/city-simulator/tensorflow_venv/lib', '/usr/local/lib', '/usr/lib']
      NOT AVAILABLE

and a lot of more lines....

Is someone able to do that ? Or how to install it right ?

kcamachom commented 3 years ago

I managed to install opencv but I get this error when trying to import cv2:

ImportError: dlopen(/Users/aaaa/miniforge3/envs/apple_tensorflow/lib/python3.8/site-packages/cv2/cv2.cpython-38-darwin.so, 2): Symbol not found: _ippicviAddProduct_16u32f_C1IMR
  Referenced from: /Users/aaaa/miniforge3/envs/apple_tensorflow/lib/python3.8/site-packages/cv2/cv2.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/aaaa/miniforge3/envs/apple_tensorflow/lib/python3.8/site-packages/cv2/cv2.cpython-38-darwin.so

any idea how to fix it? Thanks

isuruf commented 3 years ago

@kcamachom, did you use conda install opencv?

kcamachom commented 3 years ago

@kcamachom, did you use conda install opencv?

Yes. I tried first with pip install opencv-python but it didn't work.

hannesa2 commented 3 years ago

Here are the commands to make OpenCV work with this Tensorflow https://github.com/apple/tensorflow_macos/issues/153#issuecomment-803575622

aravindrx123 commented 3 years ago

I got opencv and conda working correctly. I documented my procedure here and will add more in the future. https://github.com/bjajoh/cv2-tf-scipy-for-arm-mac/blob/main/README.md

This is what I came up with:

cmake -DCMAKE_BUILD_TYPE=Release \
-DBUILD_opencv_python2=OFF \
-DBUILD_opencv_python3=ON \
-DPYTHON3_EXECUTABLE=`which python3` \
-DPYTHON3_INCLUDE_DIR=~/miniforge3/envs/python38/include/python3.8 \
-DPYTHON3_PACKAGES_PATH=~/miniforge3/envs/python38/lib/python3.8 \
-DPYTHON3_NUMPY_INCLUDE_DIR=~/miniforge3/envs/python38/lib/python3.8/site-packages/numpy/core \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DOPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-DOPENCV_ENABLE_NONFREE=ON \
../opencv

I set -DCMAKE_INSTALL_PREFIX=~/miniforge3/envs/python38. It works for me