Open joshestein opened 4 months ago
NicolasHug
commented
4 months ago As a hack, can you check if the build works if you change the extension of all the .c files in https://github.com/pytorch/vision/tree/main/torchvision/csrc/io/image/cpu/giflib into .cpp (with MACOSX_DEPLOYMENT_TARGET=11.0)
joshestein
commented
4 months ago That worked! Thank you 😄
joshestein
commented
4 months ago Although now building successfully, all the tests in tests/test_image.py are now failing. I'm seeing this warning:
serWarning: Failed to load image Python extension: 'dlopen(/tmp/vision/torchvision/image.so, 0x0006): symbol not found in flat namespace '_DGifCloseFile''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?I did install libjpeg-turbo and libpng with conda before building from source.
I'm seeing a lot of failures due to AttributeError: '_OpNamespace' 'image' object has no attribute 'read_file' for the attributes read_file, write_file, decode_jpeg, etc.
lamikr
commented
2 months ago gif_lib C files breaks the tochvision build also least on Fedora 40 and Mageia 9 for the same 'error: invalid argument '-std=c++17' not allowed with 'C')' error.
To me it looks like that the cpp_extension.py from pytorch adds the "std=c++17" compilation flag blindly to all type of files, no matter whether the extension is C or C++. I got it now building with this hack, but something better would be needed.
[lamikr@halsey pytorch ((3489a558fc7...))]$ diff -Naur torch/utils/cpp_extension.py /opt/rocm_sdk_612/lib/python3.11/site-packages/torch/utils/cpp_extension.py
--- torch/utils/cpp_extension.py 2024-08-02 18:04:56.673120304 -0700
+++ /opt/rocm_sdk_612/lib/python3.11/site-packages/torch/utils/cpp_extension.py 2024-08-03 01:21:36.356596205 -0700
@@ -568,8 +568,8 @@
cpp_format_prefix = '/{}:' if self.compiler.compiler_type == 'msvc' else '-{}='
cpp_flag_prefix = cpp_format_prefix.format('std')
cpp_flag = cpp_flag_prefix + 'c++17'
- if not any(flag.startswith(cpp_flag_prefix) for flag in cflags):
- cflags.append(cpp_flag)
+ #if not any(flag.startswith(cpp_flag_prefix) for flag in cflags):
+ # cflags.append(cpp_flag)
def unix_cuda_flags(cflags):
cflags = (COMMON_NVCC_FLAGS +
scm-aiml
commented
1 month ago Did this ever get resolved? I tried running the exact same set of commands listed by @joshestein above, using the nightly current main branch of vision and whether I set MACOSX_DEPLOYMENT_TARGET= 10.9 or 11.0 I still get the error: invalid argument '-std=c++17' not allowed with 'C'
NicolasHug
commented
1 month ago Hi folks.
Looks like the mac compiler is unhappy compiling C files while passing C++ flags, while the other compilers just ignore the flags. Perhaps the fix just a matter of renaming the .c files into c++, or compiling the libgif files with separate flags.
I don't have access to a mac, so it'll be hard for me to debug or fix this one. I'm happy to review a PR though, if you'd like to submit one.
scm-aiml
commented
1 month ago More then happy to give it a go as this would make future contribution helps easier!
I'll give an initial go tonight be capture more detailed outputs for you to review on the PR since no Mac.
On question for you @NicolasHug, on the CONTRIBUTE.md file it calls out for Mac to set target for 10.9. Is there a specific comparability reason or is that just a dated piece of the file?
NicolasHug
commented
1 month ago That part hasn't been updated in a while, there's a good chance it's not needed anymore or needs to be updated!
ezredd
commented
1 month ago I had the same error and found out that ninja was not installed on my mac (there is a warning to that end in the attached build log above). after running
brew install ninja
and rerunning
python setup.py develop
the error disappears
🐛 Describe the bug
I am having difficulty building from source on macOS with an M1 chip. Here is what I've tried:
conda create -n .torchvision_venv python=3.11conda activate .torchvision_venvconda install pytorch-nightly::pytorch -c pytorch-nightlygit clone https://github.com/pytorch/vision.gitcd visionMACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py developThis results in the following:
Error output
```bash /private/tmp/vision/setup.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html from pkg_resources import DistributionNotFound, get_distribution, parse_version Building wheel torchvision-0.19.0a0+947ae1d Compiling extensions with following flags: FORCE_CUDA: False FORCE_MPS: False DEBUG: False TORCHVISION_USE_PNG: True TORCHVISION_USE_JPEG: True TORCHVISION_USE_NVJPEG: True TORCHVISION_USE_FFMPEG: True TORCHVISION_USE_VIDEO_CODEC: True NVCC_FLAGS: Compiling with debug mode OFF Found PNG library Building torchvision with PNG image support libpng version: 1.6.43 libpng include path: /opt/homebrew/Cellar/libpng/1.6.43/include/libpng16 Running build on conda-build: False Running build on conda: True Building torchvision without JPEG image support Building torchvision without NVJPEG image support Building torchvision without ffmpeg support Building torchvision without video codec support running develop /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated. !! ******************************************************************************** Please avoid running ``setup.py`` and ``easy_install``. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://github.com/pypa/setuptools/issues/917 for details. ******************************************************************************** !! easy_install.initialize_options(self) /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated. !! ******************************************************************************** Please avoid running ``setup.py`` directly. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details. ******************************************************************************** !! self.initialize_options() running egg_info creating torchvision.egg-info writing torchvision.egg-info/PKG-INFO writing dependency_links to torchvision.egg-info/dependency_links.txt writing requirements to torchvision.egg-info/requires.txt writing top-level names to torchvision.egg-info/top_level.txt writing manifest file 'torchvision.egg-info/SOURCES.txt' /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py:499: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend. warnings.warn(msg.format('we could not find ninja.')) reading manifest file 'torchvision.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no previously-included files matching '__pycache__' found under directory '*' warning: no previously-included files matching '*.py[co]' found under directory '*' adding license file 'LICENSE' writing manifest file 'torchvision.egg-info/SOURCES.txt' running build_ext building 'torchvision._C' extension creating build creating build/temp.macosx-11.1-arm64-cpython-311 creating build/temp.macosx-11.1-arm64-cpython-311/private creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops/autocast creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops/autograd creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops/cpu creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops/mps creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops/quantized creating build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops/quantized/cpu clang -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/miniconda3/envs/.torchvision_venv/include -arch arm64 -fPIC -O2 -isystem /opt/miniconda3/envs/.torchvision_venv/include -arch arm64 -I/private/tmp/vision/torchvision/csrc -I/opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include -I/opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/TH -I/opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/THC -I/opt/miniconda3/envs/.torchvision_venv/include/python3.11 -c /private/tmp/vision/torchvision/csrc/ops/autocast/deform_conv2d_kernel.cpp -o build/temp.macosx-11.1-arm64-cpython-311/private/tmp/vision/torchvision/csrc/ops/autocast/deform_conv2d_kernel.o -g0 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_clang\" -DPYBIND11_STDLIB=\"_libcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1002\" -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17 In file included from /private/tmp/vision/torchvision/csrc/ops/autocast/deform_conv2d_kernel.cpp:1: In file included from /private/tmp/vision/torchvision/csrc/ops/autocast/../deform_conv2d.h:3: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/ATen/ATen.h:7: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/ATen/Context.h:4: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/ATen/CPUGeneratorImpl.h:3: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/ATen/core/Generator.h:18: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/c10/core/GeneratorImpl.h:8: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/c10/core/TensorImpl.h:13: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/c10/core/Storage.h:6: In file included from /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/c10/core/StorageImpl.h:8: /opt/miniconda3/envs/.torchvision_venv/lib/python3.11/site-packages/torch/include/c10/core/impl/COWDeleter.h:36:50: error: 'shared_mutex' is unavailable: introduced in macOS 10.12 using NotLastReference = std::shared_lockIf I try to target a higher version of macOS (to combat the 'is unavailable: introduced in macOS'), e.g.
_TARGET=11.0:MACOSX_DEPLOYMENT_TARGET=11.0 CC=clang CXX=clang++ python setup.py developI get an error about 'error: invalid argument '-std=c++17' not allowed with 'C')' when trying to use 'dgif_lib.c':
```bashMACOSX_DEPLOYMENT_TARGET=11.0Error outputIf I try straight
python setup.py developI get a similar error
```bashpython setup.py developError outputI have tried a couple other flags (e.g.
USE_METAL=1,USE_CUDA=0) but am getting the same errors.Any help will be greatly appreciated!
Versions
Collecting environment information... PyTorch version: 2.4.0.dev20240510 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A
OS: macOS 14.4.1 (arm64) GCC version: Could not collect Clang version: 15.0.0 (clang-1500.3.9.4) CMake version: Could not collect Libc version: N/A
Python version: 3.11.9 (main, Apr 19 2024, 11:43:47) [Clang 14.0.6 ] (64-bit runtime) Python platform: macOS-14.4.1-arm64-arm-64bit Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True
CPU: Apple M1 Pro
Versions of relevant libraries: [pip3] torch==2.4.0.dev20240510 [conda] pytorch 2.4.0.dev20240510 py3.11_0 pytorch-nightly