search

nv-legate / legate.core

The Foundation for All Legate Libraries
https://docs.nvidia.com/legate/24.06/
Apache License 2.0
186 stars 61 forks source link

Build error when trying to build legate/cunumeric with network support #948

Closed s769 closed 2 months ago

s769 commented 2 months ago

I am trying to install legate on my cluster (Lonestar6 at TACC). I followed the build instructions and developer blog and made a conda environment with ./scripts/generate-conda-envs.py --python 3.11 --ctk 12.5.0 --os linux --compilers --openmpi

and conda env create -n legate -f <env-file>.yaml

Then, I did ./install.py --network gasnetex --conduit mpi --cuda

and got the following error (showing end of output)

 [211/229] /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/x86_64-conda-linux-gnu-c++ -DLegionRuntime_EXPORTS -DUSE_CUDA -I/work/08435/srvenkat/ls6/legion/runtime -I/work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime -I/scratch/projects/compilers/nvhpc_24.5/Linux_x86_64/24.5/cuda/include -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /work/08435/srvenkat/ls6/miniconda3/envs/legate/include  -I/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/include  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib/stubs -march=haswell -O2 -std=gnu++17 -fPIC -MD -MT _deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/legion/region_tree_2.cc.o -MF _deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/legion/region_tree_2.cc.o.d -o _deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/legion/region_tree_2.cc.o -c /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime/legion/region_tree_2.cc
  In file included from /work/08435/srvenkat/ls6/legion/runtime/legion/region_tree_tmpl.cc:42,
                   from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime/legion/region_tree_2.cc:3:
  /work/08435/srvenkat/ls6/legion/runtime/legion/region_tree.inl: In member function 'void Legion::Internal::ColorSpaceLinearizationT<DIM, T>::MortonTile::delinearize(Legion::LegionColor, Legion::Point<DIM, T>&) const [with int DIM = 4; T = unsigned int]':
  /work/08435/srvenkat/ls6/legion/runtime/legion/region_tree.inl:6038:20: warning: writing 16 bytes into a region of size 4 [-Wstringop-overflow=]
   6038 |           point[i] = coords[i];
        |           ~~~~~~~~~^~~~~~~~~
  In file included from /work/08435/srvenkat/ls6/legion/runtime/realm/point.h:285,
                   from /work/08435/srvenkat/ls6/legion/runtime/realm/instance.h:28,
                   from /work/08435/srvenkat/ls6/legion/runtime/realm/profiling.h:29,
                   from /work/08435/srvenkat/ls6/legion/runtime/realm.h:43,
                   from /work/08435/srvenkat/ls6/legion/runtime/legion/legion_types.h:46,
                   from /work/08435/srvenkat/ls6/legion/runtime/legion.h:56,
                   from /work/08435/srvenkat/ls6/legion/runtime/legion/region_tree_tmpl.cc:18,
                   from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime/legion/region_tree_2.cc:3:
  /work/08435/srvenkat/ls6/legion/runtime/realm/point.inl:294:7: note: destination object 'Realm::Point<4, unsigned int>::x' of size 4
    294 |     T x, y, z, w;
        |       ^
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 672, in setup
      cmkr.make(make_args, install_target=cmake_install_target, env=env)
    File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/cmaker.py", line 696, in make
      self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
    File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/cmaker.py", line 741, in make_impl
      raise SKBuildError(msg)

  An error occurred while building with CMake.
    Command:
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/cmake --build . --target install --config Release -- -j96 --verbose
    Install target:
      install
    Source directory:
      /work/08435/srvenkat/ls6/legate.core
    Working directory:
      /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build
  Please check the install target is valid and see CMake's output for more information.

Is there anything I need to do differently here?

lightsighter commented 2 months ago

This actually looks like a GCC bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101854

Which version of GCC are you using?

s769 commented 2 months ago

I have GCC 11.4.0 

(legate) c318-001.ls6(1036)$ gcc --version
gcc (conda-forge gcc 11.4.0-13) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

From that link, it looks like I need a higher version of GCC? How should I install it?

s769 commented 2 months ago

It looks like the stringop-overflow is just a warning and is not what is causing the installation to break. The same warning shows up while compiling many of the previous files, but the installation doesn't break there.

I tried using a different makefile generator (Unix Makefiles) and got this output

       5934 |           point[i] = coords[i];
            |           ~~~~~~~~~^~~~~~~~~
      In file included from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/realm/point.h:280,
                       from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/realm/instance.h:28,
                       from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/realm/profiling.h:29,
                       from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/realm.h:43,
                       from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion/legion_types.h:45,
                       from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion.h:56,
                       from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion/region_tree_tmpl.cc:18,
                       from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime/legion/region_tree_3.cc:3:
      /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/realm/point.inl:280:7: note: destination object 'Realm::Point<4, unsigned int>::x' of size 4
        280 |     T x, y, z, w;
            |       ^
      [ 74%] Linking CXX shared library ../lib/liblegion.so
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/mapping_utilities.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      mapping_utilities.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/shim_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      shim_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/test_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      test_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/null_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      null_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/replay_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      replay_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/debug_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      debug_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/wrapper_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      wrapper_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/forwarding_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      forwarding_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/logging_wrapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      logging_wrapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/garbage_collection.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      garbage_collection.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/index_space_value.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      index_space_value.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_analysis.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_analysis.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_c.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_c.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_constraint.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_constraint.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_context.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_context.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_instances.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_instances.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_mapping.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_mapping.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_ops.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_ops.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_profiling.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_profiling.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_profiling_serializer.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_profiling_serializer.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_replication.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_replication.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_spy.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_spy.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_tasks.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_tasks.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_trace.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_trace.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_views.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_views.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_redop.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      legion_redop.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/mapper_manager.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      mapper_manager.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/runtime.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      runtime.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_1_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_1_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_1_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_1_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_2_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_2_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_2_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_2_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_3_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_3_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_3_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_3_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_4_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_4_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_4_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
      region_tree_4_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
      collect2: error: ld returned 1 exit status
      gmake[2]: *** [_deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/build.make:926: _deps/legion-build/lib/liblegion.so.1] Error 1
      gmake[1]: *** [CMakeFiles/Makefile2:493: _deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/all] Error 2
      gmake: *** [Makefile:156: all] Error 2
      Traceback (most recent call last):
        File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 672, in setup
          cmkr.make(make_args, install_target=cmake_install_target, env=env)
        File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/cmaker.py", line 696, in make
          self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
        File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/cmaker.py", line 741, in make_impl
          raise SKBuildError(msg)

      An error occurred while building with CMake.
        Command:
          /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/cmake --build . --target install --config Release -- -j96
        Install target:
          install
        Source directory:
          /work/08435/srvenkat/ls6/legate.core
        Working directory:
          /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build
      Please check the install target is valid and see CMake's output for more information.

I also tried using my system's GCC 13.2.0, which removed most of those warnings, but still has the error at the same place

 cd /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime && /scratch/tacc/apps/gcc/13.2.0/bin/g++ -DLegionRuntime_EXPORTS -DUSE_CUDA -I/work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime -I/work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /work/08435/srvenkat/ls6/miniconda3/envs/legate/include  -I/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/include  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib/stubs -march=haswell -O2 -std=gnu++17 -fPIC -MD -MT _deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/legion/region_tree_4_3.cc.o -MF CMakeFiles/LegionRuntime.dir/legion/region_tree_4_3.cc.o.d -o CMakeFiles/LegionRuntime.dir/legion/region_tree_4_3.cc.o -c /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime/legion/region_tree_4_3.cc
  In file included from /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/specfun.h:43,
                   from /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/cmath:3699,
                   from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion/legion_analysis.cc:16:
  In static member function ‘static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = Legion::Internal::CopyFillAggregator::CopyUpdate*; _Up = Legion::Internal::CopyFillAggregator::CopyUpdate*; bool _IsMove = false]’,
      inlined from ‘_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = Legion::Internal::CopyFillAggregator::CopyUpdate**; _OI = Legion::Internal::CopyFillAggregator::CopyUpdate**]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_algobase.h:506:30,
      inlined from ‘_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = Legion::Internal::CopyFillAggregator::CopyUpdate**; _OI = Legion::Internal::CopyFillAggregator::CopyUpdate**]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_algobase.h:533:42,
      inlined from ‘_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; _OI = Legion::Internal::CopyFillAggregator::CopyUpdate**]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_algobase.h:540:31,
      inlined from ‘_OI std::copy(_II, _II, _OI) [with _II = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; _OI = Legion::Internal::CopyFillAggregator::CopyUpdate**]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_algobase.h:633:7,
      inlined from ‘static _ForwardIterator std::__uninitialized_copy<true>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, std::vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; _ForwardIterator = Legion::Internal::CopyFillAggregator::CopyUpdate**]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_uninitialized.h:147:27,
      inlined from ‘_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; _ForwardIterator = Legion::Internal::CopyFillAggregator::CopyUpdate**]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_uninitialized.h:185:15,
      inlined from ‘_ForwardIterator std::__uninitialized_copy_a(_InputIterator, _InputIterator, _ForwardIterator, allocator<_Tp>&) [with _InputIterator = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; _ForwardIterator = Legion::Internal::CopyFillAggregator::CopyUpdate**; _Tp = Legion::Internal::CopyFillAggregator::CopyUpdate*]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_uninitialized.h:373:37,
      inlined from ‘void std::vector<_Tp, _Alloc>::_M_range_insert(iterator, _ForwardIterator, _ForwardIterator, std::forward_iterator_tag) [with _ForwardIterator = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, std::vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; _Tp = Legion::Internal::CopyFillAggregator::CopyUpdate*; _Alloc = std::allocator<Legion::Internal::CopyFillAggregator::CopyUpdate*>]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/vector.tcc:814:38,
      inlined from ‘std::vector<_Tp, _Alloc>::iterator std::vector<_Tp, _Alloc>::insert(const_iterator, _InputIterator, _InputIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, std::vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; <template-parameter-2-2> = void; _Tp = Legion::Internal::CopyFillAggregator::CopyUpdate*; _Alloc = std::allocator<Legion::Internal::CopyFillAggregator::CopyUpdate*>]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_vector.h:1483:19,
      inlined from ‘void Legion::Internal::CopyFillAggregator::issue_copies(Legion::Internal::InstanceView*, std::map<Legion::Internal::InstanceView*, std::vector<CopyUpdate*> >&, std::set<Legion::Internal::RtEvent>&, Legion::Internal::ApEvent, const Legion::Internal::FieldMask&, const Legion::Internal::PhysicalTraceInfo&, bool, bool, std::vector<Legion::Internal::ApEvent>*)’ at /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion/legion_analysis.cc:7295:28:
  /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_algobase.h:437:30: warning: ‘void* __builtin_memmove(void*, const void*, long unsigned int)’ writing between 9 and 9223372036854775800 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
    437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
        |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  In file included from /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/x86_64-pc-linux-gnu/bits/c++allocator.h:33,
                   from /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/allocator.h:46,
                   from /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_tree.h:64,
                   from /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/map:62,
                   from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion/legion_types.h:30,
                   from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion.h:56,
                   from /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion/legion_analysis.cc:17:
  In member function ‘_Tp* std::__new_allocator<_Tp>::allocate(size_type, const void*) [with _Tp = Legion::Internal::InstanceView*]’,
      inlined from ‘static _Tp* std::allocator_traits<std::allocator<_Tp1> >::allocate(allocator_type&, size_type) [with _Tp = Legion::Internal::InstanceView*]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/alloc_traits.h:482:28,
      inlined from ‘std::_Vector_base<_Tp, _Alloc>::pointer std::_Vector_base<_Tp, _Alloc>::_M_allocate(std::size_t) [with _Tp = Legion::Internal::InstanceView*; _Alloc = std::allocator<Legion::Internal::InstanceView*>]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_vector.h:378:33,
      inlined from ‘std::_Vector_base<_Tp, _Alloc>::pointer std::_Vector_base<_Tp, _Alloc>::_M_allocate(std::size_t) [with _Tp = Legion::Internal::CopyFillAggregator::CopyUpdate*; _Alloc = std::allocator<Legion::Internal::CopyFillAggregator::CopyUpdate*>]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_vector.h:375:7,
      inlined from ‘void std::vector<_Tp, _Alloc>::_M_range_insert(iterator, _ForwardIterator, _ForwardIterator, std::forward_iterator_tag) [with _ForwardIterator = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, std::vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; _Tp = Legion::Internal::CopyFillAggregator::CopyUpdate*; _Alloc = std::allocator<Legion::Internal::CopyFillAggregator::CopyUpdate*>]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/vector.tcc:805:40,
      inlined from ‘std::vector<_Tp, _Alloc>::iterator std::vector<_Tp, _Alloc>::insert(const_iterator, _InputIterator, _InputIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<Legion::Internal::CopyFillAggregator::CopyUpdate**, std::vector<Legion::Internal::CopyFillAggregator::CopyUpdate*> >; <template-parameter-2-2> = void; _Tp = Legion::Internal::CopyFillAggregator::CopyUpdate*; _Alloc = std::allocator<Legion::Internal::CopyFillAggregator::CopyUpdate*>]’ at /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/stl_vector.h:1483:19,
      inlined from ‘void Legion::Internal::CopyFillAggregator::issue_copies(Legion::Internal::InstanceView*, std::map<Legion::Internal::InstanceView*, std::vector<CopyUpdate*> >&, std::set<Legion::Internal::RtEvent>&, Legion::Internal::ApEvent, const Legion::Internal::FieldMask&, const Legion::Internal::PhysicalTraceInfo&, bool, bool, std::vector<Legion::Internal::ApEvent>*)’ at /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-src/runtime/legion/legion_analysis.cc:7295:28:
  /scratch/tacc/apps/gcc/13.2.0/include/c++/13.2.0/bits/new_allocator.h:147:55: note: at offset [-9223372036854775808, -1] into destination object of size [8, 9223372036854775800] allocated by ‘operator new’
    147 |         return static_cast<_Tp*>(_GLIBCXX_OPERATOR_NEW(__n * sizeof(_Tp)));
        |                                                       ^
  [ 74%] Linking CXX shared library ../lib/liblegion.so
  cd /work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/runtime && /work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/cmake -E cmake_link_script CMakeFiles/LegionRuntime.dir/link.txt --verbose=1
  /scratch/tacc/apps/gcc/13.2.0/bin/g++ -fPIC -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /work/08435/srvenkat/ls6/miniconda3/envs/legate/include  -I/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/include  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib/stubs -march=haswell -O2 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib -Wl,-rpath-link,/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib/stubs -shared -Wl,-soname,liblegion.so.1 -o ../lib/liblegion.so.1 CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/mapping_utilities.cc.o CMakeFiles/LegionRuntime.dir/mappers/shim_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/test_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/null_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/replay_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/debug_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/wrapper_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/forwarding_mapper.cc.o CMakeFiles/LegionRuntime.dir/mappers/logging_wrapper.cc.o CMakeFiles/LegionRuntime.dir/legion/garbage_collection.cc.o CMakeFiles/LegionRuntime.dir/legion/index_space_value.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_analysis.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_c.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_constraint.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_context.cc.o CMakeFiles/LegionRuntime.dir/legion/legion.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_instances.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_mapping.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_ops.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_profiling.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_profiling_serializer.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_replication.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_spy.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_tasks.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_trace.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_views.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_redop.cc.o CMakeFiles/LegionRuntime.dir/legion/mapper_manager.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree.cc.o CMakeFiles/LegionRuntime.dir/legion/runtime.cc.o CMakeFiles/LegionRuntime.dir/legion/legion_redop.cu.o CMakeFiles/LegionRuntime.dir/legion/region_tree_1.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_1_1.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_1_2.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_1_3.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_1_4.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_2.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_2_1.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_2_2.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_2_3.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_2_4.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_3.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_3_1.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_3_2.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_3_3.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_3_4.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_4.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_4_1.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_4_2.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_4_3.cc.o CMakeFiles/LegionRuntime.dir/legion/region_tree_4_4.cc.o   -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/sysroot/usr/lib  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/gcc/x86_64-conda-linux-gnu/11.4.0  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/gcc  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/lib  -L/work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/sysroot/lib  -Wl,-rpath,"\$ORIGIN:/work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/legion-build/lib:/work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/sysroot/usr/lib:" ../lib/librealm.so.1 /work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/libz.so ../embed-gasnet/install/lib/libgasnet-mpi-par.a ../embed-gasnet/install/lib/libgasnet-mpi-par.a ../embed-gasnet/install/lib/libammpi.a -lpthread /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/libgcc.a /work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/sysroot/usr/lib/libm.so /scratch/projects/compilers/intel24.1/oneapi/mpi/2021.12/lib/libmpi.so -lpthread /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/libgcc.a /work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/sysroot/usr/lib/libm.so /scratch/projects/compilers/intel24.1/oneapi/mpi/2021.12/lib/libmpi.so /work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/sysroot/usr/lib/libdl.so /work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/libcudart.so /work/08435/srvenkat/ls6/miniconda3/envs/legate/targets/x86_64-linux/lib/stubs/libcuda.so -lpthread -ldl /work/08435/srvenkat/ls6/miniconda3/envs/legate/x86_64-conda-linux-gnu/sysroot/usr/lib/librt.so -lcudadevrt -lcudart
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/mapping_utilities.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  mapping_utilities.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/shim_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  shim_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/test_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  test_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/null_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  null_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/replay_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  replay_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/debug_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  debug_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/wrapper_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  wrapper_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/forwarding_mapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  forwarding_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/mappers/logging_wrapper.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  logging_wrapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/garbage_collection.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  garbage_collection.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/index_space_value.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  index_space_value.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_analysis.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_analysis.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_c.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_c.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_constraint.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_constraint.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_context.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_context.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_instances.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_instances.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_mapping.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_mapping.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_ops.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_ops.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_profiling.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_profiling.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_profiling_serializer.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_profiling_serializer.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_replication.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_replication.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_spy.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_spy.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_tasks.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_tasks.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_trace.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_trace.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_views.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_views.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/legion_redop.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  legion_redop.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/mapper_manager.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  mapper_manager.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/runtime.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  runtime.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_1_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_1_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_1_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_1_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_1_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_2_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_2_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_2_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_2_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_2_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_3_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_3_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_3_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_3_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_3_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_1.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_4_1.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_2.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_4_2.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_3.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_4_3.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  /scratch/tacc/apps/gcc/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/LegionRuntime.dir/legion/region_tree_4_4.cc.o: in function `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)':
  region_tree_4_4.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): multiple definition of `cub::CUB_200400___CUDA_ARCH_LIST___NS::Debug(cudaError, char const*, int)'; CMakeFiles/LegionRuntime.dir/mappers/default_mapper.cc.o:default_mapper.cc:(.text._ZN3cub32CUB_200400___CUDA_ARCH_LIST___NS5DebugE9cudaErrorPKci+0x0): first defined here
  collect2: error: ld returned 1 exit status
  gmake[2]: *** [_deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/build.make:931: _deps/legion-build/lib/liblegion.so.1] Error 1
  gmake[2]: Leaving directory '/work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build'
  gmake[1]: *** [CMakeFiles/Makefile2:493: _deps/legion-build/runtime/CMakeFiles/LegionRuntime.dir/all] Error 2
  gmake[1]: Leaving directory '/work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build'
  gmake: *** [Makefile:156: all] Error 2
  Traceback (most recent call last):
    File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 672, in setup
      cmkr.make(make_args, install_target=cmake_install_target, env=env)
    File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/cmaker.py", line 696, in make
      self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
    File "/work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/python3.11/site-packages/skbuild/cmaker.py", line 741, in make_impl
      raise SKBuildError(msg)

  An error occurred while building with CMake.

so does that mean it's a linker error?

s769 commented 2 months ago

should I try using an older CUDA version? I am using 12.5.0 right now; maybe that has something to do with it?

manopapad commented 2 months ago

After some research, it looks like this was an issue introduced with CCCL 2.4 (which was included in CUDA 12.5), and fixed with https://github.com/NVIDIA/cccl/pull/1643, which should be included in CCCL 2.5 (which should be included in the next CUDA toolkit release).

For now maybe you could try downgrading to CUDA 12.4, which doesn't have the CCCL bug?

s769 commented 2 months ago

ok, downgrading to CUDA 12.4.1 worked.

When I try to run the cunumeric examples with

legate --nodes 1 --gpus 3 ./benchmark.py

I get this error:

/work/08435/srvenkat/ls6/miniconda3/envs/legate/bin/legion_python: symbol lookup error: /work/08435/srvenkat/ls6/miniconda3/envs/legate/lib/liblegion.so.1: undefined symbol: _ZN6Legion12SumReductionIN6thrust36THRUST_200302_500_600_700_800_900_NS7complexI6__halfEEE8identityE
s769 commented 2 months ago

I was able to fix the above issue by downgrading to CUDA 12.2.2. Actually, that makes sense since the output of nvidia-smi shows CUDA 12.2 as the version number.

I did install.py for legate.core and cunumeric using the --conduit mpi option. When I run an example such as

legate --nodes 1 --gpus 3 ./cholesky.py, I get this warning message

WARNING: There is at least one non-excluded one OpenFabrics device found,
but there are no active ports detected (or Open MPI was unable to use
them).  This is most certainly not what you wanted.  Check your
cables, subnet manager configuration, etc.  The openib BTL will be
ignored for this job.

  Local host: c315-014
--------------------------------------------------------------------------
*** WARNING: Using GASNet's mpi-conduit, which exists for portability convenience.
    WARNING: This system appears to contain recognized network hardware: InfiniBand IBV
    WARNING: which is supported by a GASNet native conduit, although
    WARNING: it was not detected at configure time (missing drivers?)
    WARNING: You should *really* use the high-performance native GASNet conduit
    WARNING: if communication performance is at all important in this program run.

or

WARNING: There was an error initializing an OpenFabrics device.

  Local host:   c315-014
  Local device: mlx5_0
--------------------------------------------------------------------------
*** WARNING: Using GASNet's mpi-conduit, which exists for portability convenience.
    WARNING: This system appears to contain recognized network hardware: InfiniBand IBV
    WARNING: which is supported by a GASNet native conduit, although
    WARNING: it was not detected at configure time (missing drivers?)
    WARNING: You should *really* use the high-performance native GASNet conduit
    WARNING: if communication performance is at all important in this program run.

Based on this, I tried to reinstall with the --conduit ibv option. However, then I get a build error again. If I check

/work/08435/srvenkat/ls6/legate.core/_skbuild/linux-x86_64-3.11/cmake-build/_deps/ legion-build/embed-gasnet/build.log, I find the error message

gasnet configure error: User requested --enable-ibv but I don't know how to build ibv programs for your system

I tried looking into this and found some similar issues online, but can't see how to fix it yet.

manopapad commented 2 months ago

There should be a configure log somewhere, where gasnet is printing out the commands it tried to run to detect the capabilities of the system, where it probably gives more details on why it "doesn't know how to build ibv programs for your system". The last failed command in that configuration file will show the compilation command it tried to run, and why it failed (e.g. it couldn't find a libibverbs.so).

If UCX is available for your system (e.g. pull it from conda-forge during execution of scripts/setup-conda-envs.py with --ucx), then you can try using that (with install.py --network ucx), and you don't have to debug the gasnet installation.

s769 commented 2 months ago

The configure log shows this

checking for ibv_devinfo... /usr/bin/ibv_devinfo
checking for IBV_HOME setting...           (default)  "/usr"
checking if /usr is the IB Verbs library install prefix... yes
checking for IBV_CFLAGS setting...         (default)  ""
checking for IBV_LIBS setting...           (default)  "-libverbs"
checking for IBV_LDFLAGS setting...        (default)  ""
checking for working IB Verbs configuration... no
checking for appropriate atomics support required by ibv-conduit... yes

I tried the UCX install like you said. For legate.core, the --network ucx option is supported. For cunumeric, I had to use --network gasnetex --conduit ucx.

With that, I still get this

--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   c318-001
  Local device: mlx5_0
--------------------------------------------------------------------------
manopapad commented 2 months ago

cuNumeric doesn't need to be fed the --network option if you're building legate.core first and passing it to the cuNumeric build. In this case cuNumeric will just use whatever --network that legate.core included (and ignores the --network and --conduit you passed to cuNumeric's install.py -- some upcoming changes will clean up this and other weirdness in the cuNumeric build).

This warning is (generally) benign, and only has to do with MPI itself (which we typically don't care about, since we don't go through MPI for most transfers, instead we go through UCX or GASNet, depending on the --network being used).

s769 commented 2 months ago

Thank you. In that case, I think my setup is working now. I have some other questions about how the GPU memory management works and the usage of the --fbmem and --eager-alloc-percentage options, but I will save those for a different post since it is getting too off-topic.