search

nv-legate / legate.core

The Foundation for All Legate Libraries
https://docs.nvidia.com/legate/24.06/
Apache License 2.0
189 stars 63 forks source link

[QST] Building / using on Ada #943

Closed gmarkall closed 6 months ago

gmarkall commented 6 months ago

Software versions

I couldn't build legate so I am not sure I have a legate-issue command - I can't find one, anwyay. Here's the environment file I generated, though:

name: legate-test
channels:
- nvidia/label/cuda-12.4.1
- conda-forge
dependencies:

  - python=3.10,!=3.9.7  # avoid https://bugs.python.org/issue45121

  # cuda
  - cuda-version=12.4
  - cutensor=1.7*=h*
  - nccl
  - pynvml
  - cuda-cccl
  - cuda-cudart-dev
  - cuda-cudart-static
  - cuda-driver-dev
  - cuda-nvml-dev
  - cuda-nvtx
  - libcublas-dev
  - libcufft-dev
  - libcurand-dev
  - libcusolver-dev
  - libcusparse-dev
  - libnvjitlink-dev
  - cuda-nvcc
  - gcc_linux-64=11.*
  - gxx_linux-64=11.*

  # build
  - c-compiler
  - cmake>=3.24,!=3.25.0
  - cxx-compiler
  - cython>=3
  - elfutils
  - git
  - libhwloc=*=*default*
  - make
  - ninja
  - numba
  - openssl
  - pkg-config
  - rust
  - scikit-build>=0.13.1
  - setuptools>=60
  - zlib

  # runtime
  - cffi
  - llvm-openmp
  - numpy>=1.22
  - libblas=*=*openblas*
  - openblas=*=*openmp*
  - openblas<=0.3.21
  - opt_einsum
  - scipy
  - typing_extensions
  - libhwloc=*=*default*

  # tests
  - clang-tools>=8
  - clang>=8
  - colorama
  - coverage
  - mock
  - mypy>=0.961
  - pre-commit
  - pytest-cov
  - pytest-lazy-fixture
  - pytest-mock
  - pytest<8
  - types-docutils
  - pynvml
  - tifffile
  - libhwloc=*=*default*

  # docs
  - pandoc
  - doxygen
  - ipython
  - jinja2
  - markdown<3.4.0
  - pydata-sphinx-theme>=0.13
  - myst-parser
  - nbsphinx
  - sphinx-copybutton
  - sphinx>=4.4.0

Jupyter notebook / Jupyter Lab version

N/A

Expected behavior

Installation to succeed.

Observed behavior

Output:

Verbose build is off
Using python lib and version: /home/gmarkall/miniforge3/envs/legatedev/lib/libpython3.10.so, 3.10.14
Obtaining file:///home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core
  Checking if build backend supports build_editable ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: legate-core
  Running setup.py develop for legate-core
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [193 lines of output]
        /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/setuptools/dist.py:318: InformationOnly: Normalizing '24.01.00.dev+37.g8f7bff6' to '24.1.0.dev0+37.g8f7bff6'
          self.metadata.version = self._normalize_version(self.metadata.version)
        /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
        !!

                ********************************************************************************
                Please avoid running ``setup.py`` and ``easy_install``.
                Instead, use pypa/build, pypa/installer or other
                standards-based tools.

                See https://github.com/pypa/setuptools/issues/917 for details.
                ********************************************************************************

        !!
          easy_install.initialize_options(self)

        --------------------------------------------------------------------------------
        -- Trying 'Ninja' generator
        --------------------------------
        ---------------------------
        ----------------------
        -----------------
        ------------
        -------
        --
        CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
          Compatibility with CMake < 3.5 will be removed from a future version of
          CMake.

          Update the VERSION argument <min> value or use a ...<max> suffix to tell
          CMake that the project does not need compatibility with older versions.

        Not searching for unused variables given on the command line.

        -- The C compiler identification is GNU 11.4.0
        -- Detecting C compiler ABI info
        -- Detecting C compiler ABI info - done
        -- Check for working C compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-cc - skipped
        -- Detecting C compile features
        -- Detecting C compile features - done
        -- The CXX compiler identification is GNU 11.4.0
        -- Detecting CXX compiler ABI info
        -- Detecting CXX compiler ABI info - done
        -- Check for working CXX compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-c++ - skipped
        -- Detecting CXX compile features
        -- Detecting CXX compile features - done
        -- Configuring done (0.3s)
        -- Generating done (0.0s)
        -- Build files have been written to: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_cmake_test_compile/build
        --
        -------
        ------------
        -----------------
        ----------------------
        ---------------------------
        --------------------------------
        -- Trying 'Ninja' generator - success
        --------------------------------------------------------------------------------

        Configuring Project
          Working directory:
            /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build
          Command:
            /home/gmarkall/miniforge3/envs/legatedev/bin/cmake /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.14 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/gmarkall/miniforge3/envs/legatedev/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev/include/python3.10 -DPYTHON_LIBRARY:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/libpython3.10.so -DPython_EXECUTABLE:PATH=/home/gmarkall/miniforge3/envs/legatedev/bin/python3 -DPython_ROOT_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev/include/python3.10 -DPython_NumPy_INCLUDE_DIRS:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/home/gmarkall/miniforge3/envs/legatedev/bin/python3 -DPython3_ROOT_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev/include/python3.10 -DPython3_NumPy_INCLUDE_DIRS:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/numpy/core/include -DCMAKE_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DLegion_MAX_DIM=4 -DLegion_MAX_FIELDS=256 -DLegion_SPY=OFF -DLegion_BOUNDS_CHECKS=OFF -DLegion_USE_CUDA=ON -DLegion_USE_OpenMP=OFF -DLegion_NETWORKS= -DLegion_USE_Python=ON -DLegion_Python_Version=3.10.14 -DLegion_BUILD_JUPYTER=ON '-DLegion_EMBED_GASNet_CONFIGURE_ARGS="--with-ibv-max-hcas=8"' -DBUILD_MARCH=haswell -DLegion_CUDA_ARCH=ada -DCPM_DOWNLOAD_Legion=ON '-DLegion_PYTHON_EXTRA_INSTALL_ARGS="--root;/;--prefix;/home/gmarkall/miniforge3/envs/legatedev"' -DCMAKE_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DLegion_MAX_DIM=4 -DLegion_MAX_FIELDS=256 -DLegion_SPY=OFF -DLegion_BOUNDS_CHECKS=OFF -DLegion_USE_CUDA=ON -DLegion_USE_OpenMP=OFF -DLegion_NETWORKS= -DLegion_USE_Python=ON -DLegion_Python_Version=3.10.14 -DLegion_BUILD_JUPYTER=ON -DLegion_EMBED_GASNet_CONFIGURE_ARGS=--with-ibv-max-hcas=8 -DBUILD_MARCH=haswell -DLegion_CUDA_ARCH=ada -DCPM_DOWNLOAD_Legion=ON '-DLegion_PYTHON_EXTRA_INSTALL_ARGS=--root;/;--prefix;/home/gmarkall/miniforge3/envs/legatedev'

        Not searching for unused variables given on the command line.
        -- The C compiler identification is GNU 11.4.0
        -- The CXX compiler identification is GNU 11.4.0
        -- Detecting C compiler ABI info
        -- Detecting C compiler ABI info - done
        -- Check for working C compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-cc - skipped
        -- Detecting C compile features
        -- Detecting C compile features - done
        -- Detecting CXX compiler ABI info
        -- Detecting CXX compiler ABI info - done
        -- Check for working CXX compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-c++ - skipped
        -- Detecting CXX compile features
        -- Detecting CXX compile features - done
        -- Found Git: /home/gmarkall/miniforge3/envs/legatedev/bin/git (found version "2.45.0")
        -- Found Python3: /home/gmarkall/miniforge3/envs/legatedev/bin/python3 (found version "3.10.14") found components: Interpreter Development Development.Module Development.Embed
        -- CPM: adding package Legion@24.1.0 (24.1.0)
        -- Performing Test COMPILER_SUPPORTS_MARCH
        -- Performing Test COMPILER_SUPPORTS_MARCH - Success
        -- Performing Test COMPILER_SUPPORTS_MALTIVEC
        -- Performing Test COMPILER_SUPPORTS_MALTIVEC - Failed
        -- Performing Test COMPILER_SUPPORTS_MABI_ALTIVEC
        -- Performing Test COMPILER_SUPPORTS_MABI_ALTIVEC - Failed
        -- Performing Test COMPILER_SUPPORTS_MVSX
        -- Performing Test COMPILER_SUPPORTS_MVSX - Failed
        -- Performing Test COMPILER_SUPPORTS_DEFCHECK
        -- Performing Test COMPILER_SUPPORTS_DEFCHECK - Failed
        -- Found Python3: /home/gmarkall/miniforge3/envs/legatedev/bin/python3 (found version "3.10.14") found components: Interpreter
        -- The CUDA compiler identification is NVIDIA 12.4.131
        -- Detecting CUDA compiler ABI info
        -- Detecting CUDA compiler ABI info - done
        -- Check for working CUDA compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/nvcc - skipped
        -- Detecting CUDA compile features
        -- Detecting CUDA compile features - done
        -- Found CUDAToolkit: /home/gmarkall/miniforge3/envs/legatedev/include (found version "12.4.131")
        -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
        -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
        -- Looking for pthread_create in pthreads
        -- Looking for pthread_create in pthreads - not found
        -- Looking for pthread_create in pthread
        -- Looking for pthread_create in pthread - found
        -- Found Threads: TRUE
        -- Found ZLIB: /home/gmarkall/miniforge3/envs/legatedev/lib/libz.so (found version "1.2.13")
        -- Found elfutils/libdwfl.h header: /home/gmarkall/miniforge3/envs/legatedev/include
        -- Found libdw library: /home/gmarkall/miniforge3/envs/legatedev/lib/libdw.so
        -- Looking for posix_fallocate64
        -- Looking for posix_fallocate64 - found
        -- Performing Test COMPILER_SUPORTS_OVERLOADED_VIRTUAL_WARNING
        -- Performing Test COMPILER_SUPORTS_OVERLOADED_VIRTUAL_WARNING - Success
        -- Performing Test COMPILER_SUPPORTS_CXX_WDEPRECATED
        -- Performing Test COMPILER_SUPPORTS_CXX_WDEPRECATED - Success
        -- Performing Test COMPILER_SUPPORTS_CXX_FORMAT_PEDENTIC
        -- Performing Test COMPILER_SUPPORTS_CXX_FORMAT_PEDENTIC - Failed
        -- Found NCCL: /home/gmarkall/miniforge3/envs/legatedev/lib/libnccl.so
        -- CPM: adding package Thrust@1.17.0.0 (1.17.0)
        CMake Deprecation Warning at _skbuild/linux-x86_64-3.10/cmake-build/_deps/thrust-src/CMakeLists.txt:9 (cmake_policy):
          The OLD behavior for policy CMP0104 will be removed from a future version
          of CMake.

          The cmake-policies(7) manual explains that the OLD behaviors of all
          policies are deprecated and that a policy should be set to OLD only under
          specific short-term circumstances.  Projects should be ported to the NEW
          behavior and not rely on setting a policy to OLD.

        -- Found Thrust: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/thrust-src/thrust/cmake/thrust-config.cmake (found version "1.17.0.0")
        -- rapids-cmake [Thrust]: applied diff reroot_support.diff to fix issue: 'Support conda-forge usage of CMake rerooting [https://github.com/NVIDIA/thrust/pull/1969]'
        -- rapids-cmake [Thrust]: applied diff transform_iter_with_reduce_by_key.diff to fix issue: 'Support transform iterator with reduce by key [https://github.com/NVIDIA/thrust/pull/1805]'
        -- rapids-cmake [Thrust]: applied diff install_rules.diff to fix issue: 'Thrust 1.X installs incorrect files [https://github.com/NVIDIA/thrust/issues/1790]'
        -- Found CUB: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/thrust-src/dependencies/cub/cub/cmake/cub-config.cmake (found suitable version "1.17.0.0", minimum required is "1.17.0.0")
        CMake Warning (dev) at /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:245 (find_package):
          Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
          are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
          the cmake_policy command to set the policy and suppress this warning.

        Call Stack (most recent call first):
          _skbuild/linux-x86_64-3.10/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cython/init.cmake:53 (find_package)
          legate_core_python.cmake:62 (rapids_cython_init)
          CMakeLists.txt:101 (include)
        This warning is for project developers.  Use -Wno-dev to suppress it.

        -- Found PythonInterp: /home/gmarkall/miniforge3/envs/legatedev/bin/python3 (found version "3.10.14")
        CMake Warning (dev) at /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:252 (find_package):
          Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
          are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
          the cmake_policy command to set the policy and suppress this warning.

        Call Stack (most recent call first):
          _skbuild/linux-x86_64-3.10/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cython/init.cmake:53 (find_package)
          legate_core_python.cmake:62 (rapids_cython_init)
          CMakeLists.txt:101 (include)
        This warning is for project developers.  Use -Wno-dev to suppress it.

        -- Found PythonLibs: /home/gmarkall/miniforge3/envs/legatedev/lib/libpython3.10.so (found version "3.10.14")
        -- Found Cython: /home/gmarkall/miniforge3/envs/legatedev/bin/cython
        -- Performing Test Weak Link MODULE -> SHARED (gnu_ld_ignore) - Success
        _modinit_prefix:PyInit_
        _modinit_prefix:PyInit_
        -- legate_core_python: cython_lib_dir='/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/legate-core-cpp/lib'
        -- Configuring done (15.6s)
        -- Generating done (0.0s)
        -- Build files have been written to: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build
        [1/217] Building CUDA object _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o
        FAILED: _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o
        /home/gmarkall/miniforge3/envs/legatedev/bin/nvcc -forward-unknown-to-host-compiler -DCUDA_FATBIN_COMPILATION -I/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/legion-src/runtime -I/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/legion-build/runtime -O2 -std=c++17 "--generate-code=arch=compute_ada,code=[compute_ada,sm_ada]" -Xcudafe=--diag_suppress=boolean_controlling_expr_is_constant --fatbin -Wno-deprecated-gpu-targets -MD -MT _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o -MF _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o.d -x cu -c /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/legion-src/runtime/realm/cuda/cuda_memcpy.cu -o _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o
        nvcc fatal   : Unsupported gpu architecture 'compute_ada'
        [2/217] Generate install_info.py
        [3/217] Building CXX object _deps/legion-build/runtime/CMakeFiles/realm_cuhook.dir/realm/cuda/cuda_hook.cc.o
        ninja: build stopped: subcommand failed.
        Traceback (most recent call last):
          File "/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
            cmkr.make(make_args, install_target=cmake_install_target, env=env)
          File "/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/cmaker.py", line 697, in make
            self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
          File "/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/cmaker.py", line 742, in make_impl
            raise SKBuildError(msg)

        An error occurred while building with CMake.
          Command:
            /home/gmarkall/miniforge3/envs/legatedev/bin/cmake --build . --target install --config Release -- -j20
          Install target:
            install
          Source directory:
            /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core
          Working directory:
            /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build
        Please check the install target is valid and see CMake's output for more information.

        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [193 lines of output]
    /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/setuptools/dist.py:318: InformationOnly: Normalizing '24.01.00.dev+37.g8f7bff6' to '24.1.0.dev0+37.g8f7bff6'
      self.metadata.version = self._normalize_version(self.metadata.version)
    /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` and ``easy_install``.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://github.com/pypa/setuptools/issues/917 for details.
            ********************************************************************************

    !!
      easy_install.initialize_options(self)

    --------------------------------------------------------------------------------
    -- Trying 'Ninja' generator
    --------------------------------
    ---------------------------
    ----------------------
    -----------------
    ------------
    -------
    --
    CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
      Compatibility with CMake < 3.5 will be removed from a future version of
      CMake.

      Update the VERSION argument <min> value or use a ...<max> suffix to tell
      CMake that the project does not need compatibility with older versions.

    Not searching for unused variables given on the command line.

    -- The C compiler identification is GNU 11.4.0
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-cc - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- The CXX compiler identification is GNU 11.4.0
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-c++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Configuring done (0.3s)
    -- Generating done (0.0s)
    -- Build files have been written to: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_cmake_test_compile/build
    --
    -------
    ------------
    -----------------
    ----------------------
    ---------------------------
    --------------------------------
    -- Trying 'Ninja' generator - success
    --------------------------------------------------------------------------------

    Configuring Project
      Working directory:
        /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build
      Command:
        /home/gmarkall/miniforge3/envs/legatedev/bin/cmake /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.14 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/gmarkall/miniforge3/envs/legatedev/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev/include/python3.10 -DPYTHON_LIBRARY:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/libpython3.10.so -DPython_EXECUTABLE:PATH=/home/gmarkall/miniforge3/envs/legatedev/bin/python3 -DPython_ROOT_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev/include/python3.10 -DPython_NumPy_INCLUDE_DIRS:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/home/gmarkall/miniforge3/envs/legatedev/bin/python3 -DPython3_ROOT_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/home/gmarkall/miniforge3/envs/legatedev/include/python3.10 -DPython3_NumPy_INCLUDE_DIRS:PATH=/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/numpy/core/include -DCMAKE_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DLegion_MAX_DIM=4 -DLegion_MAX_FIELDS=256 -DLegion_SPY=OFF -DLegion_BOUNDS_CHECKS=OFF -DLegion_USE_CUDA=ON -DLegion_USE_OpenMP=OFF -DLegion_NETWORKS= -DLegion_USE_Python=ON -DLegion_Python_Version=3.10.14 -DLegion_BUILD_JUPYTER=ON '-DLegion_EMBED_GASNet_CONFIGURE_ARGS="--with-ibv-max-hcas=8"' -DBUILD_MARCH=haswell -DLegion_CUDA_ARCH=ada -DCPM_DOWNLOAD_Legion=ON '-DLegion_PYTHON_EXTRA_INSTALL_ARGS="--root;/;--prefix;/home/gmarkall/miniforge3/envs/legatedev"' -DCMAKE_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DLegion_MAX_DIM=4 -DLegion_MAX_FIELDS=256 -DLegion_SPY=OFF -DLegion_BOUNDS_CHECKS=OFF -DLegion_USE_CUDA=ON -DLegion_USE_OpenMP=OFF -DLegion_NETWORKS= -DLegion_USE_Python=ON -DLegion_Python_Version=3.10.14 -DLegion_BUILD_JUPYTER=ON -DLegion_EMBED_GASNet_CONFIGURE_ARGS=--with-ibv-max-hcas=8 -DBUILD_MARCH=haswell -DLegion_CUDA_ARCH=ada -DCPM_DOWNLOAD_Legion=ON '-DLegion_PYTHON_EXTRA_INSTALL_ARGS=--root;/;--prefix;/home/gmarkall/miniforge3/envs/legatedev'

    Not searching for unused variables given on the command line.
    -- The C compiler identification is GNU 11.4.0
    -- The CXX compiler identification is GNU 11.4.0
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-cc - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/x86_64-conda-linux-gnu-c++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Found Git: /home/gmarkall/miniforge3/envs/legatedev/bin/git (found version "2.45.0")
    -- Found Python3: /home/gmarkall/miniforge3/envs/legatedev/bin/python3 (found version "3.10.14") found components: Interpreter Development Development.Module Development.Embed
    -- CPM: adding package Legion@24.1.0 (24.1.0)
    -- Performing Test COMPILER_SUPPORTS_MARCH
    -- Performing Test COMPILER_SUPPORTS_MARCH - Success
    -- Performing Test COMPILER_SUPPORTS_MALTIVEC
    -- Performing Test COMPILER_SUPPORTS_MALTIVEC - Failed
    -- Performing Test COMPILER_SUPPORTS_MABI_ALTIVEC
    -- Performing Test COMPILER_SUPPORTS_MABI_ALTIVEC - Failed
    -- Performing Test COMPILER_SUPPORTS_MVSX
    -- Performing Test COMPILER_SUPPORTS_MVSX - Failed
    -- Performing Test COMPILER_SUPPORTS_DEFCHECK
    -- Performing Test COMPILER_SUPPORTS_DEFCHECK - Failed
    -- Found Python3: /home/gmarkall/miniforge3/envs/legatedev/bin/python3 (found version "3.10.14") found components: Interpreter
    -- The CUDA compiler identification is NVIDIA 12.4.131
    -- Detecting CUDA compiler ABI info
    -- Detecting CUDA compiler ABI info - done
    -- Check for working CUDA compiler: /home/gmarkall/miniforge3/envs/legatedev/bin/nvcc - skipped
    -- Detecting CUDA compile features
    -- Detecting CUDA compile features - done
    -- Found CUDAToolkit: /home/gmarkall/miniforge3/envs/legatedev/include (found version "12.4.131")
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
    -- Looking for pthread_create in pthreads
    -- Looking for pthread_create in pthreads - not found
    -- Looking for pthread_create in pthread
    -- Looking for pthread_create in pthread - found
    -- Found Threads: TRUE
    -- Found ZLIB: /home/gmarkall/miniforge3/envs/legatedev/lib/libz.so (found version "1.2.13")
    -- Found elfutils/libdwfl.h header: /home/gmarkall/miniforge3/envs/legatedev/include
    -- Found libdw library: /home/gmarkall/miniforge3/envs/legatedev/lib/libdw.so
    -- Looking for posix_fallocate64
    -- Looking for posix_fallocate64 - found
    -- Performing Test COMPILER_SUPORTS_OVERLOADED_VIRTUAL_WARNING
    -- Performing Test COMPILER_SUPORTS_OVERLOADED_VIRTUAL_WARNING - Success
    -- Performing Test COMPILER_SUPPORTS_CXX_WDEPRECATED
    -- Performing Test COMPILER_SUPPORTS_CXX_WDEPRECATED - Success
    -- Performing Test COMPILER_SUPPORTS_CXX_FORMAT_PEDENTIC
    -- Performing Test COMPILER_SUPPORTS_CXX_FORMAT_PEDENTIC - Failed
    -- Found NCCL: /home/gmarkall/miniforge3/envs/legatedev/lib/libnccl.so
    -- CPM: adding package Thrust@1.17.0.0 (1.17.0)
    CMake Deprecation Warning at _skbuild/linux-x86_64-3.10/cmake-build/_deps/thrust-src/CMakeLists.txt:9 (cmake_policy):
      The OLD behavior for policy CMP0104 will be removed from a future version
      of CMake.

      The cmake-policies(7) manual explains that the OLD behaviors of all
      policies are deprecated and that a policy should be set to OLD only under
      specific short-term circumstances.  Projects should be ported to the NEW
      behavior and not rely on setting a policy to OLD.

    -- Found Thrust: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/thrust-src/thrust/cmake/thrust-config.cmake (found version "1.17.0.0")
    -- rapids-cmake [Thrust]: applied diff reroot_support.diff to fix issue: 'Support conda-forge usage of CMake rerooting [https://github.com/NVIDIA/thrust/pull/1969]'
    -- rapids-cmake [Thrust]: applied diff transform_iter_with_reduce_by_key.diff to fix issue: 'Support transform iterator with reduce by key [https://github.com/NVIDIA/thrust/pull/1805]'
    -- rapids-cmake [Thrust]: applied diff install_rules.diff to fix issue: 'Thrust 1.X installs incorrect files [https://github.com/NVIDIA/thrust/issues/1790]'
    -- Found CUB: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/thrust-src/dependencies/cub/cub/cmake/cub-config.cmake (found suitable version "1.17.0.0", minimum required is "1.17.0.0")
    CMake Warning (dev) at /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:245 (find_package):
      Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
      are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
      the cmake_policy command to set the policy and suppress this warning.

    Call Stack (most recent call first):
      _skbuild/linux-x86_64-3.10/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cython/init.cmake:53 (find_package)
      legate_core_python.cmake:62 (rapids_cython_init)
      CMakeLists.txt:101 (include)
    This warning is for project developers.  Use -Wno-dev to suppress it.

    -- Found PythonInterp: /home/gmarkall/miniforge3/envs/legatedev/bin/python3 (found version "3.10.14")
    CMake Warning (dev) at /home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:252 (find_package):
      Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
      are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
      the cmake_policy command to set the policy and suppress this warning.

    Call Stack (most recent call first):
      _skbuild/linux-x86_64-3.10/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cython/init.cmake:53 (find_package)
      legate_core_python.cmake:62 (rapids_cython_init)
      CMakeLists.txt:101 (include)
    This warning is for project developers.  Use -Wno-dev to suppress it.

    -- Found PythonLibs: /home/gmarkall/miniforge3/envs/legatedev/lib/libpython3.10.so (found version "3.10.14")
    -- Found Cython: /home/gmarkall/miniforge3/envs/legatedev/bin/cython
    -- Performing Test Weak Link MODULE -> SHARED (gnu_ld_ignore) - Success
    _modinit_prefix:PyInit_
    _modinit_prefix:PyInit_
    -- legate_core_python: cython_lib_dir='/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/legate-core-cpp/lib'
    -- Configuring done (15.6s)
    -- Generating done (0.0s)
    -- Build files have been written to: /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build
    [1/217] Building CUDA object _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o
    FAILED: _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o
    /home/gmarkall/miniforge3/envs/legatedev/bin/nvcc -forward-unknown-to-host-compiler -DCUDA_FATBIN_COMPILATION -I/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/legion-src/runtime -I/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/legion-build/runtime -O2 -std=c++17 "--generate-code=arch=compute_ada,code=[compute_ada,sm_ada]" -Xcudafe=--diag_suppress=boolean_controlling_expr_is_constant --fatbin -Wno-deprecated-gpu-targets -MD -MT _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o -MF _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o.d -x cu -c /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build/_deps/legion-src/runtime/realm/cuda/cuda_memcpy.cu -o _deps/legion-build/runtime/CMakeFiles/realm_cuda_fatbin.dir/realm/cuda/cuda_memcpy.cu.o
    nvcc fatal   : Unsupported gpu architecture 'compute_ada'
    [2/217] Generate install_info.py
    [3/217] Building CXX object _deps/legion-build/runtime/CMakeFiles/realm_cuhook.dir/realm/cuda/cuda_hook.cc.o
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
        cmkr.make(make_args, install_target=cmake_install_target, env=env)
      File "/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/cmaker.py", line 697, in make
        self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
      File "/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/site-packages/skbuild/cmaker.py", line 742, in make_impl
        raise SKBuildError(msg)

    An error occurred while building with CMake.
      Command:
        /home/gmarkall/miniforge3/envs/legatedev/bin/cmake --build . --target install --config Release -- -j20
      Install target:
        install
      Source directory:
        /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core
      Working directory:
        /home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/_skbuild/linux-x86_64-3.10/cmake-build
    Please check the install target is valid and see CMake's output for more information.

    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/./install.py", line 895, in <module>
    driver()
  File "/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/./install.py", line 891, in driver
    install(unknown=unknown, is_conda=is_conda, **vars(args))
  File "/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/./install.py", line 569, in install
    execute_command(pip_install_cmd, verbose, cwd=legate_core_dir, env=cmd_env)
  File "/home/gmarkall/numbadev/issues/debug-cuda-gdb/legatedev/legate.core/./install.py", line 77, in execute_command
    subprocess.check_call(args, **kwargs)
  File "/home/gmarkall/miniforge3/envs/legatedev/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/gmarkall/miniforge3/envs/legatedev/bin/python3', '-m', 'pip', 'install', '--root', '/', '--prefix', '/home/gmarkall/miniforge3/envs/legatedev', '--no-deps', '--no-build-isolation', '--editable', '.']' returned non-zero exit status 1.

Example code or instructions

I'm trying to build legate.core to work on cuNumeric on my laptop, which has an RTX 3500 Ada Generation GPU. Taking cues from https://nv-legate.github.io/legate.core/BUILD.html, I tried to build with CUDA for the Ada architecture with:

$ ./install.py --cuda --arch ada --editable

However, the message produced appears to suggest this is not supported, but perhaps I am doing something wrong. How should I proceed?



### Stack traceback or browser console output

_No response_
gmarkall commented 6 months ago

I see now it was missing from the mappings between names and compute capabilities in cuda_arch_helpers.cmake - I've created #944 to add it.