[Bug] (macOS) Linking problem? symbol not found in flat namespace '__ZN2at3mps9MPSStream6commitEb'

debugmenot commented 11 months ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmcv).

Environment

MacOS 13.6 (Same issue on 14.0 Sonoma) Apple Silicon M1 Ultra / 64G PyTorch 2.1 (release)

Impossible to get info from mmcv because of symbol not found error.

Reproduces the problem - code sample

Installed by official instructions.

If building MMCV from source with PyTorch 2.1 problem still exists...

There are no such issues with PyTorch 2.0

Reproduces the problem - command or script

~/o/mmdetection (main|✔) $ mim download mmdet --config rtmdet_tiny_8xb32-300e_coco --dest . (openmmlab) processing rtmdet_tiny_8xb32-300e_coco... downloading ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.9/54.9 MiB 851.5 kB/s eta 0:00:00 Successfully downloaded rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth to /Users/madcode/openmmlab/mmdetection Successfully dumped rtmdet_tiny_8xb32-300e_coco.py to /Users/madcode/openmmlab/mmdetection ~/o/mmdetection (main|✔) $ python demo/image_demo.py demo/demo.jpg rtmdet_tiny_8xb32-300e_coco.py --weights rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth --device cpu

Reproduces the problem - error message

~/o/mmdetection (main|✔) $ mim download mmdet --config rtmdet_tiny_8xb32-300e_coco --dest . (openmmlab) processing rtmdet_tiny_8xb32-300e_coco... downloading ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.9/54.9 MiB 851.5 kB/s eta 0:00:00 Successfully downloaded rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth to /Users/madcode/openmmlab/mmdetection Successfully dumped rtmdet_tiny_8xb32-300e_coco.py to /Users/madcode/openmmlab/mmdetection ~/o/mmdetection (main|✔) $ python demo/image_demo.py demo/demo.jpg rtmdet_tiny_8xb32-300e_coco.py --weights rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth --device cpu Traceback (most recent call last): File "demo/image_demo.py", line 43, in from mmdet.apis import DetInferencer File "/Users/madcode/openmmlab/mmdetection/mmdet/apis/init.py", line 2, in from .det_inferencer import DetInferencer File "/Users/madcode/openmmlab/mmdetection/mmdet/apis/det_inferencer.py", line 21, in from mmdet.evaluation import INSTANCE_OFFSET File "/Users/madcode/openmmlab/mmdetection/mmdet/evaluation/init.py", line 3, in from .metrics import * # noqa: F401,F403 File "/Users/madcode/openmmlab/mmdetection/mmdet/evaluation/metrics/init.py", line 5, in from .coco_metric import CocoMetric File "/Users/madcode/openmmlab/mmdetection/mmdet/evaluation/metrics/coco_metric.py", line 16, in from mmdet.datasets.api_wrappers import COCO, COCOeval File "/Users/madcode/openmmlab/mmdetection/mmdet/datasets/init.py", line 26, in from .utils import get_loading_pipeline File "/Users/madcode/openmmlab/mmdetection/mmdet/datasets/utils.py", line 5, in from mmdet.datasets.transforms import LoadAnnotations, LoadPanopticAnnotations File "/Users/madcode/openmmlab/mmdetection/mmdet/datasets/transforms/init.py", line 6, in from .formatting import (ImageToTensor, PackDetInputs, PackReIDInputs, File "/Users/madcode/openmmlab/mmdetection/mmdet/datasets/transforms/formatting.py", line 11, in from mmdet.structures.bbox import BaseBoxes File "/Users/madcode/openmmlab/mmdetection/mmdet/structures/bbox/init.py", line 2, in from .base_boxes import BaseBoxes File "/Users/madcode/openmmlab/mmdetection/mmdet/structures/bbox/base_boxes.py", line 9, in from mmdet.structures.mask.structures import BitmapMasks, PolygonMasks File "/Users/madcode/openmmlab/mmdetection/mmdet/structures/mask/init.py", line 3, in from .structures import (BaseInstanceMasks, BitmapMasks, PolygonMasks, File "/Users/madcode/openmmlab/mmdetection/mmdet/structures/mask/structures.py", line 12, in from mmcv.ops.roi_align import roi_align File "/opt/homebrew/Caskroom/miniconda/base/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/init.py", line 2, in from .active_rotated_filter import active_rotated_filter File "/opt/homebrew/Caskroom/miniconda/base/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in ext_module = ext_loader.load_ext( File "/opt/homebrew/Caskroom/miniconda/base/envs/openmmlab/lib/python3.8/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext ext = importlib.import_module('mmcv.' + name) File "/opt/homebrew/Caskroom/miniconda/base/envs/openmmlab/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: dlopen(/opt/homebrew/Caskroom/miniconda/base/envs/openmmlab/lib/python3.8/site-packages/mmcv/_ext.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '__ZN2at3mps9MPSStream6commitEb'

Additional information

No response

debugmenot commented 11 months ago

Building from source:

~/openmmlab $ conda create --name openmmlab_2 python=3.8 -y
Collecting package metadata (current_repodata.json): done Solving environment: done

(...)

ca-certificates pkgs/main/osx-arm64::ca-certificates-2023.08.22-hca03da5_0 libcxx pkgs/main/osx-arm64::libcxx-14.0.6-h848a8c0_0 libffi pkgs/main/osx-arm64::libffi-3.4.4-hca03da5_0 ncurses pkgs/main/osx-arm64::ncurses-6.4-h313beb8_0 openssl pkgs/main/osx-arm64::openssl-3.0.11-h1a28f6b_2 pip pkgs/main/osx-arm64::pip-23.2.1-py38hca03da5_0 python pkgs/main/osx-arm64::python-3.8.18-hb885b13_0 readline pkgs/main/osx-arm64::readline-8.2-h1a28f6b_0 setuptools pkgs/main/osx-arm64::setuptools-68.0.0-py38hca03da5_0 sqlite pkgs/main/osx-arm64::sqlite-3.41.2-h80987f9_0 tk pkgs/main/osx-arm64::tk-8.6.12-hb8d0fd4_0 wheel pkgs/main/osx-arm64::wheel-0.41.2-py38hca03da5_0 xz pkgs/main/osx-arm64::xz-5.4.2-h80987f9_0 zlib pkgs/main/osx-arm64::zlib-1.2.13-h5a0b063_0

(...)

~/openmmlab $ conda activate openmmlab_2 ~/openmmlab $ conda install pytorch torchvision cpuonly -c pytorch (openmmlab_2) Collecting package metadata (current_repodata.json): done Solving environment: done

(...)

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
cpuonly-2.0                |                0           2 KB  pytorch
pytorch-mutex-1.0          |              cpu           3 KB  pytorch
------------------------------------------------------------
                                       Total:           5 KB

The following NEW packages will be INSTALLED:

blas pkgs/main/osx-arm64::blas-1.0-openblas brotlipy pkgs/main/osx-arm64::brotlipy-0.7.0-py38h1a28f6b_1002 bzip2 pkgs/main/osx-arm64::bzip2-1.0.8-h620ffc9_4 certifi pkgs/main/osx-arm64::certifi-2023.7.22-py38hca03da5_0 cffi pkgs/main/osx-arm64::cffi-1.15.1-py38h80987f9_3 charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0 cpuonly pytorch/noarch::cpuonly-2.0-0 cryptography pkgs/main/osx-arm64::cryptography-41.0.3-py38hd4332d6_0 ffmpeg pkgs/main/osx-arm64::ffmpeg-4.2.2-h04105a8_0 filelock pkgs/main/osx-arm64::filelock-3.9.0-py38hca03da5_0 freetype pkgs/main/osx-arm64::freetype-2.12.1-h1192e45_0 gettext pkgs/main/osx-arm64::gettext-0.21.0-h13f89a0_1 giflib pkgs/main/osx-arm64::giflib-5.2.1-h80987f9_3 gmp pkgs/main/osx-arm64::gmp-6.2.1-hc377ac9_3 gmpy2 pkgs/main/osx-arm64::gmpy2-2.1.2-py38h8c48613_0 gnutls pkgs/main/osx-arm64::gnutls-3.6.15-h887c41c_0 icu pkgs/main/osx-arm64::icu-73.1-h313beb8_0 idna pkgs/main/osx-arm64::idna-3.4-py38hca03da5_0 jinja2 pkgs/main/osx-arm64::jinja2-3.1.2-py38hca03da5_0 jpeg pkgs/main/osx-arm64::jpeg-9e-h80987f9_1 lame pkgs/main/osx-arm64::lame-3.100-h1a28f6b_0 lcms2 pkgs/main/osx-arm64::lcms2-2.12-hba8e193_0 lerc pkgs/main/osx-arm64::lerc-3.0-hc377ac9_0 libdeflate pkgs/main/osx-arm64::libdeflate-1.17-h80987f9_1 libgfortran pkgs/main/osx-arm64::libgfortran-5.0.0-11_3_0_hca03da5_28 libgfortran5 pkgs/main/osx-arm64::libgfortran5-11.3.0-h009349e_28 libiconv pkgs/main/osx-arm64::libiconv-1.16-h1a28f6b_2 libidn2 pkgs/main/osx-arm64::libidn2-2.3.4-h80987f9_0 libjpeg-turbo pkgs/main/osx-arm64::libjpeg-turbo-2.0.0-h1a28f6b_0 libopenblas pkgs/main/osx-arm64::libopenblas-0.3.21-h269037a_0 libopus pkgs/main/osx-arm64::libopus-1.3-h1a28f6b_1 libpng pkgs/main/osx-arm64::libpng-1.6.39-h80987f9_0 libtasn1 pkgs/main/osx-arm64::libtasn1-4.19.0-h80987f9_0 libtiff pkgs/main/osx-arm64::libtiff-4.5.1-h313beb8_0 libunistring pkgs/main/osx-arm64::libunistring-0.9.10-h1a28f6b_0 libvpx pkgs/main/osx-arm64::libvpx-1.10.0-hc377ac9_0 libwebp pkgs/main/osx-arm64::libwebp-1.3.2-ha3663a8_0 libwebp-base pkgs/main/osx-arm64::libwebp-base-1.3.2-h80987f9_0 libxml2 pkgs/main/osx-arm64::libxml2-2.10.4-h0dcf63f_1 llvm-openmp pkgs/main/osx-arm64::llvm-openmp-14.0.6-hc6e5704_0 lz4-c pkgs/main/osx-arm64::lz4-c-1.9.4-h313beb8_0 markupsafe pkgs/main/osx-arm64::markupsafe-2.1.1-py38h1a28f6b_0 mpc pkgs/main/osx-arm64::mpc-1.1.0-h8c48613_1 mpfr pkgs/main/osx-arm64::mpfr-4.0.2-h695f6f0_1 mpmath pkgs/main/osx-arm64::mpmath-1.3.0-py38hca03da5_0 nettle pkgs/main/osx-arm64::nettle-3.7.3-h84b5d62_1 networkx pkgs/main/osx-arm64::networkx-3.1-py38hca03da5_0 numpy pkgs/main/osx-arm64::numpy-1.24.3-py38h1398885_0 numpy-base pkgs/main/osx-arm64::numpy-base-1.24.3-py38h90707a3_0 openh264 pkgs/main/osx-arm64::openh264-1.8.0-h98b2900_0 pillow pkgs/main/osx-arm64::pillow-9.4.0-py38h313beb8_1 pycparser pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0 pyopenssl pkgs/main/osx-arm64::pyopenssl-23.2.0-py38hca03da5_0 pysocks pkgs/main/osx-arm64::pysocks-1.7.1-py38hca03da5_0 pytorch pytorch/osx-arm64::pytorch-2.1.0-py3.8_0 pytorch-mutex pytorch/noarch::pytorch-mutex-1.0-cpu pyyaml pkgs/main/osx-arm64::pyyaml-6.0-py38h80987f9_1 requests pkgs/main/osx-arm64::requests-2.31.0-py38hca03da5_0 sympy pkgs/main/osx-arm64::sympy-1.11.1-py38hca03da5_0 torchvision pytorch/osx-arm64::torchvision-0.16.0-py38_cpu typing_extensions pkgs/main/osx-arm64::typing_extensions-4.7.1-py38hca03da5_0 urllib3 pkgs/main/osx-arm64::urllib3-1.26.16-py38hca03da5_0 x264 pkgs/main/osx-arm64::x264-1!152.20180806-h1a28f6b_0 yaml pkgs/main/osx-arm64::yaml-0.2.5-h1a28f6b_0 zstd pkgs/main/osx-arm64::zstd-1.5.5-hd90d995_0

(...)

~/openmmlab $ python -c 'import torch;print(torch.version)' (openmmlab_2) 2.1.0 ~/openmmlab $ git clone https://github.com/open-mmlab/mmcv.git ~/openmmlab $ cd mmcv ~/o/mmcv (main|✔) $ pip install -r requirements/optional.txt (openmmlab_2) Collecting ninja (from -r requirements/optional.txt (line 1)) Downloading ninja-1.11.1-py2.py3-none-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl (270 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 270.7/270.7 kB 1.2 MB/s eta 0:00:00 Collecting psutil (from -r requirements/optional.txt (line 2)) Downloading psutil-5.9.5-cp38-abi3-macosx_11_0_arm64.whl (246 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 246.1/246.1 kB 7.0 MB/s eta 0:00:00 Installing collected packages: ninja, psutil Successfully installed ninja-1.11.1 psutil-5.9.5 ~/o/mmcv (main|✔) $ gcc --version (openmmlab_2) Apple clang version 15.0.0 (clang-1500.0.40.1) Target: arm64-apple-darwin22.6.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin ~/o/mmcv (main|✔) $ pip install -e . -v

(…)

building output omitted. Only warnings are like:

ld: warning: object file (/Users/madcode/openmmlab/mmcv/build/temp.macosx-11.1-arm64-cpython-38/mmcv/ops/csrc/pytorch/voxelization.o) was built for newer 'macOS' version (13.0) than being linked (11.1) and so one...

Installed /Users/madcode/openmmlab/mmcv

Successfully installed addict-2.4.0 contourpy-1.1.1 cycler-0.12.0 fonttools-4.43.0 importlib-metadata-6.8.0 importlib-resources-6.1.0 kiwisolver-1.4.5 markdown-it-py-3.0.0 matplotlib-3.7.3 mdurl-0.1.2 mmcv-2.0.1 mmengine-0.8.5 opencv-python-4.8.1.78 packaging-23.2 platformdirs-3.11.0 pygments-2.16.1 pyparsing-3.1.1 python-dateutil-2.8.2 rich-13.6.0 six-1.16.0 termcolor-2.3.0 tomli-2.0.1 yapf-0.40.2 zipp-3.17.0

~/o/mmcv (main|✔) $ python .dev_scripts/check_installation.py (openmmlab_2) Traceback (most recent call last): File ".dev_scripts/check_installation.py", line 4, in from mmcv.ops import box_iou_rotated File "/Users/madcode/openmmlab/mmcv/mmcv/ops/init.py", line 3, in from .active_rotated_filter import active_rotated_filter File "/Users/madcode/openmmlab/mmcv/mmcv/ops/active_rotated_filter.py", line 10, in ext_module = ext_loader.load_ext( File "/Users/madcode/openmmlab/mmcv/mmcv/utils/ext_loader.py", line 13, in load_ext ext = importlib.import_module('mmcv.' + name) File "/opt/homebrew/Caskroom/miniconda/base/envs/openmmlab_2/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: dlopen(/Users/madcode/openmmlab/mmcv/mmcv/_ext.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '__ZN2at3mps9MPSStream6commitEb'

fazhang-master commented 11 months ago

zhouzaida commented 11 months ago

Hi @debugmenot , as a workaround, you can pull the main branch to fix this error (#2958) or downgrade your torch to 2.0.0 and we will fix this error for 2.1.0 in the future.

debugmenot commented 11 months ago

Hi! Any schedule to return back MPS support? 2.1.0 is already released but MPS is disabled :( There are lots of new MPS functions supported in Torch 2.1...

zhouzaida commented 10 months ago

Hi @debugmenot, the support for MPS backend is in our schedule but with a not high priority. By the way, you can still use the op supported in torch.

kojipy commented 8 months ago

@debugmenot I encountered sama error when I use python 3.10.1. but swithing to python3.9.6 resolved the issue. I'm not sure but using python3.9.x might potentialy fix this issue.