NVIDIA / apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
BSD 3-Clause "New" or "Revised" License
8.16k stars 1.35k forks source link

fix building torch extension with glog #1791

Open petronny opened 3 months ago

petronny commented 3 months ago

To fix the following error when building python-apex-git on Arch Linux:

running build_ext
/usr/lib/python3.11/site-packages/torch/utils/cpp_extension.py:500: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
/usr/lib/python3.11/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no /opt/cuda/bin/g++ version bounds defined for CUDA version 12.4
  warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'apex_C' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/csrc
/opt/cuda/bin/gcc -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -flto=auto -fPIC -I/usr/lib/python3.11/site-packages/torch/include -I/usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/lib/python3.11/site-packages/torch/include/TH -I/usr/lib/python3.11/site-packages/torch/include/THC -I/usr/include/python3.11 -c csrc/flatten_unflatten.cpp -o build/temp.linux-x86_64-cpython-311/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1018\" -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++17
In file included from /usr/include/c10/util/logging_is_google_glog.h:50,
                 from /usr/include/c10/util/Logging.h:26,
                 from /usr/include/ATen/core/ivalue_inl.h:25,
                 from /usr/include/ATen/core/ivalue.h:1551,
                 from /usr/include/ATen/core/List_inl.h:4,
                 from /usr/include/ATen/core/List.h:490,
                 from /usr/include/ATen/core/IListRef_inl.h:3,
                 from /usr/include/ATen/core/IListRef.h:631,
                 from /usr/include/ATen/WrapDimUtils.h:3,
                 from /usr/include/ATen/TensorNames.h:3,
                 from /usr/include/ATen/NamedTensorUtils.h:3,
                 from /usr/include/torch/csrc/autograd/variable.h:11,
                 from /usr/include/torch/csrc/autograd/autograd.h:3,
                 from /usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:3,
                 from /usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /usr/include/torch/extension.h:5,
                 from csrc/flatten_unflatten.cpp:1:
/usr/include/glog/logging.h:60:4: error: #error <glog/logging.h> was not included correctly. See the documention for how to consume the library.
   60 | #  error <glog/logging.h> was not included correctly. See the documention for how to consume the library.
      |    ^~~~~

Full build log: https://github.com/arch4edu/cactus/actions/runs/8490889706/job/23262378384

Ping @hubutui

hubutui commented 3 months ago

If torch upstream could add a default macro GLOG_USE_GLOG_EXPORT, then all downstream who need to compile torch extension could work out-of-box.