search

pytorch / pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration
https://pytorch.org
Other
84.02k stars 22.65k forks source link

The Python module installs cmake files and headers under the Python's site-packages directory that can't be used #116478

Open yurivict opened 10 months ago

yurivict commented 10 months ago

🐛 Describe the bug

These cmake files can't be found by cmake because cmake doesn't look in this location:

        /usr/local/lib/python3.9/site-packages/torch/include/c10/macros/cmake_macros.h
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/ATen/ATenConfig.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Caffe2Targets-release.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Caffe2Targets.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/FindCUDAToolkit.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/FindCUSPARSELT.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/FindCUDA.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/FindCUDNN.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/CMakeInitializeConfigs.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/make2cmake.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/parse_cubin.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/run_nvcc.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/select_compute_arch.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindPackageHandleStandardArgs.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindPackageMessage.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/LoadHIP.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/gflags.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/glog.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/mkl.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/mkldnn.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/protobuf.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/utils.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Torch/TorchConfig.cmake
        /usr/local/lib/python3.9/site-packages/torch/share/cmake/Torch/TorchConfigVersion.cmake
        etc.

Likewise, these headers are also can't be easily found:

        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_flatten.h
        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_layouts.h
        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_list.h
        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_memoryformats.h
        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_new.h
        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_numpy.h
        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_qschemes.h
        /usr/local/lib/python3.9/site-packages/torch/include/torch/csrc/utils/tensor_types.h
        etc.

In order to satisfy the C++ API cmake files, headers and libraries should normally be installed under $PREFIX, for example under /usr/local/include, /usr/local/lib, etc.

They are actually properly installed when the cmake build is performed separately. So there is no need to install them with the Python module into the above paths.

Versions

n/a

cc @malfet @seemethere @jbschlosser

albanD commented 10 months ago

Hey!

I think there are quite a few different things there:

SomeoneSerge commented 8 months ago
  1. PEP 725 is tangentially related: https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/67
  2. Just to observe: one can always extend CMAKE_PREFIX_PATH with .../site-packages/torch ad hoc.
  3. In Nixpkgs we explicitly move the includes and the cmake export into a separate output (nixpkgs#python3Packages.torch.dev): https://github.com/NixOS/nixpkgs/blob/fccb6b42c3cad4e664236de94b0f9a0ea22b1d6e/pkgs/development/python-modules/torch/default.nix#L449-L451
    1. We normally wouldn't have to do this ad hoc like in the snippet above, because our cmake infra sets up the sensible installation paths automatically: https://github.com/NixOS/nixpkgs/blob/fccb6b42c3cad4e664236de94b0f9a0ea22b1d6e/pkgs/by-name/cm/cmake/setup-hook.sh#L91-L103. One of the things currently forcing us to shuffle files around manually is the entanglement between setup.py and the cmake build: we let the python script run cmake, so we lose some of the control over the flags.

cpp_extension

One noticeable inconvenience is that cpp_extension tries to guess flags instead of relying on cmake or pkg-config. I suspect this is the reason we keep a copy of the headers in site-packages in the expression linked above, and it can also be seen as the reason CUDAExtension does not support splayed CUDA installations