Closed yousecjoe closed 8 months ago
Hi @yousecjoe!
Thanks for submitting this issue - our team has been notified and we'll get back to you as soon as we can! In the mean time, feel free to add any relevant information to this issue.
@yousecjoe I was just able run through the entire example successfully. Could you please try the following steps?
branch-24.03
.workspace
directory:
rm -rf build-docker
rm -rf .cache
BUILD_DIR=build-docker ./scripts/compile.sh
If you're able to get past that, I noticed that you'll also have to manually install pydantic
:
mamba install pydantic
We already have an issue to pre-install it in the container.
I was using the latest from branch 24.03, and I tried again with the latest from branch 24.03 and got the same issue.
Looks like it is failing at a different step, but it is the same error: no instance of overloaded function "atomicCAS"
-- Generating done (0.2s)
-- Build files have been written to: /workspace/build-docker
+ echo 'Running CMake build...'
Running CMake build...
+ cmake --build build-docker -j
[0/2] Re-checking globbed directories...
[342/370] Building CUDA object morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
FAILED: morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
/workspace/build-docker/run_ccache_cuda.sh /opt/conda/envs/morpheus/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-c++ -DBOOST_ATOMIC_DYN_LINK -DBOOST_ATOMIC_NO_LIB -DBOOST_CONTEXT_DYN_LINK -DBOOST_CONTEXT_NO_LIB -DBOOST_FIBER_DYN_LINK -DBOOST_FIBER_NO_LIB -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_FILESYSTEM_NO_LIB -DFMT_HEADER_ONLY=1 -DGFLAGS_IS_A_DLL=0 -DGLOG_CUSTOM_PREFIX_SUPPORT -DINDEX_32_BIT -DLIBCUDACXX_ENABLE_EXPERIMENTAL_MEMORY_RESOURCE -DMATX_DISABLE_CUB_CACHE=1 -DMATX_ENABLE_CUTENSOR=0 -DMATX_ENABLE_CUTLASS=0 -DMATX_ENABLE_FILEIO=0 -DMATX_ENABLE_PYBIND11=0 -DRXCPP_USE_FIBERS -DSPDLOG_ACTIVE_LEVEL=SPDLOG_LEVEL_INFO -DSPDLOG_FMT_EXTERNAL -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_DISABLE_ABI_NAMESPACE -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTHRUST_IGNORE_ABI_NAMESPACE_ERROR -DTRITON_ENABLE_GPU=1 -D__STDC_FORMAT_MACROS -Dmorpheus_EXPORTS -I/workspace/morpheus/_lib/include -I/workspace/build-docker/autogenerated/include -I/workspace/build-docker/morpheus/_lib -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/kernels -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust/thrust/cmake/../.. -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/lib/cmake/libcudacxx/../../../include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub/cub/cmake/../.. -I/workspace/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src -I/workspace/.cache/cpm/tritonclient/f2dd9277615c6e3c56858568bcd1e470c4d30a1f/src/c++/library -isystem /opt/conda/envs/morpheus/targets/x86_64-linux/include -isystem /opt/conda/envs/morpheus/include -isystem /workspace/.cache/cpm/pybind11/7e2e3e2d8bdbe726c035292051ad01e0121f53b5/include -isystem /opt/conda/envs/morpheus/include/python3.10 -isystem /opt/conda/envs/morpheus/include/librdkafka --expt-relaxed-constexpr --extended-lambda -O3 -DNDEBUG -std=c++17 -arch=native -Xcompiler=-fPIC --expt-relaxed-constexpr -MD -MT morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o -MF morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o.d -x cu -c /workspace/morpheus/_lib/src/utilities/matx_util.cu -o morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(183): error: no instance of overloaded function "atomicCAS" matches the argument list
argument types are: (unsigned short *, unsigned short, short)
old.b = static_cast<float>(atomicCAS(address_as_other, assumed, tmpval.i));
^
../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(242): error: no instance of overloaded function "atomicCAS" matches the argument list
argument types are: (unsigned short *, unsigned short, short)
old.h = atomicCAS(address_as_other, assumed, tmpval.i);
^
2 errors detected in the compilation of "../morpheus/_lib/src/utilities/matx_util.cu".
[351/370] Building CXX object morpheus/_lib/CMakeFiles/morpheus._lib.messages.dir/messages/module.cpp.o
ninja: build stopped: subcommand failed.
I wiped the directory and pulled the latest data to try again. It stops consistently at
../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(183): error: no instance of overloaded function "atomicCAS" matches the argument list
argument types are: (unsigned short *, unsigned short, short)
old.b = static_cast<float>(atomicCAS(address_as_other, assumed, tmpval.i));
^
../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(242): error: no instance of overloaded function "atomicCAS" matches the argument list
argument types are: (unsigned short *, unsigned short, short)
old.h = atomicCAS(address_as_other, assumed, tmpval.i);
^
2 errors detected in the compilation of "../morpheus/_lib/src/utilities/matx_util.cu".
[351/370] Building CXX object morpheus/_lib/CMakeFiles/morpheus._lib.messages.dir/messages/module.cpp.o
ninja: build stopped: subcommand failed.
@yousecjoe We noticed that one of your GPUs is GeForce GTX 1080 Ti
. We just recently updated the 24.03 development branch to use RAPIDS 24.02 which no longer includes support for Pascal GPUs. Therefore only your GeForce RTX 3080
is supported for 24.03. You have two options:
examples/sid_visualization/docker-compose.yml
to only use the supported GPU. At the top of the file, update the x-with-gpus
section to look like this:
x-with-gpus: &with_gpus
deploy:
resources:
reservations:
devices:
- driver: "nvidia"
device_ids: ["0"]
capabilities: [gpu]
We'll also make sure the docs are updated to reflect the changes in the GPU requirements.
Thank you, that resolved the issue.
Version
24.03
Which installation method(s) does this occur on?
Docker
Describe the bug.
I am following SID Visualization Example And the step to build Morpheus fails.
Inside the container, compile Morpheus:
BUILD_DIR=build-docker ./scripts/compile.sh
The following occurs
I expected Morpheus to build so I could continue with the example.
Minimum reproducible example
No response
Relevant log output
Click here to see error details
Full env printout
Click here to see environment details
Other/Misc.
No response
Code of Conduct