nv-morpheus / Morpheus

Morpheus SDK
Apache License 2.0
364 stars 136 forks source link

[BUG]: Morpheus build fails in SID Visualization Example #1514

Closed yousecjoe closed 8 months ago

yousecjoe commented 8 months ago

Version

24.03

Which installation method(s) does this occur on?

Docker

Describe the bug.

I am following SID Visualization Example And the step to build Morpheus fails.

Inside the container, compile Morpheus: BUILD_DIR=build-docker ./scripts/compile.sh

The following occurs

[46/77] Building CUDA object morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
FAILED: morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o 
/workspace/build-docker/run_ccache_cuda.sh /opt/conda/envs/morpheus/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-c++ -DBOOST_ATOMIC_DYN_LINK -DBOOST_ATOMIC_NO_LIB -DBOOST_CONTEXT_DYN_LINK -DBOOST_CONTEXT_NO_LIB -DBOOST_FIBER_DYN_LINK -DBOOST_FIBER_NO_LIB -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_FILESYSTEM_NO_LIB -DFMT_HEADER_ONLY=1 -DGFLAGS_IS_A_DLL=0 -DGLOG_CUSTOM_PREFIX_SUPPORT -DINDEX_32_BIT -DLIBCUDACXX_ENABLE_EXPERIMENTAL_MEMORY_RESOURCE -DMATX_DISABLE_CUB_CACHE=1 -DMATX_ENABLE_CUTENSOR=0 -DMATX_ENABLE_CUTLASS=0 -DMATX_ENABLE_FILEIO=0 -DMATX_ENABLE_PYBIND11=0 -DRXCPP_USE_FIBERS -DSPDLOG_ACTIVE_LEVEL=SPDLOG_LEVEL_INFO -DSPDLOG_FMT_EXTERNAL -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_DISABLE_ABI_NAMESPACE -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTHRUST_IGNORE_ABI_NAMESPACE_ERROR -DTRITON_ENABLE_GPU=1 -D__STDC_FORMAT_MACROS -Dmorpheus_EXPORTS -I/workspace/morpheus/_lib/include -I/workspace/build-docker/autogenerated/include -I/workspace/build-docker/morpheus/_lib -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/kernels -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust/thrust/cmake/../.. -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/lib/cmake/libcudacxx/../../../include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub/cub/cmake/../.. -I/workspace/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src -I/workspace/.cache/cpm/tritonclient/f2dd9277615c6e3c56858568bcd1e470c4d30a1f/src/c++/library -isystem /opt/conda/envs/morpheus/targets/x86_64-linux/include -isystem /opt/conda/envs/morpheus/include -isystem /workspace/.cache/cpm/pybind11/7e2e3e2d8bdbe726c035292051ad01e0121f53b5/include -isystem /opt/conda/envs/morpheus/include/python3.10 -isystem /opt/conda/envs/morpheus/include/librdkafka --expt-relaxed-constexpr --extended-lambda -O3 -DNDEBUG -std=c++17 -arch=native -Xcompiler=-fPIC --expt-relaxed-constexpr -Xcompiler -pthread -MD -MT morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o -MF morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o.d -x cu -c /workspace/morpheus/_lib/src/utilities/matx_util.cu -o morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(183): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.b = static_cast<float>(atomicCAS(address_as_other, assumed, tmpval.i));
                                 ^

../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(242): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.h = atomicCAS(address_as_other, assumed, tmpval.i);
              ^

2 errors detected in the compilation of "../morpheus/_lib/src/utilities/matx_util.cu".
[55/77] Building CXX object morpheus/_lib/CMakeFiles/morpheus._lib.messages.dir/messages/module.cpp.o
ninja: build stopped: subcommand failed.

I expected Morpheus to build so I could continue with the example.

Minimum reproducible example

No response

Relevant log output

Click here to see error details

[46/77] Building CUDA object morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
FAILED: morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o 
/workspace/build-docker/run_ccache_cuda.sh /opt/conda/envs/morpheus/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-c++ -DBOOST_ATOMIC_DYN_LINK -DBOOST_ATOMIC_NO_LIB -DBOOST_CONTEXT_DYN_LINK -DBOOST_CONTEXT_NO_LIB -DBOOST_FIBER_DYN_LINK -DBOOST_FIBER_NO_LIB -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_FILESYSTEM_NO_LIB -DFMT_HEADER_ONLY=1 -DGFLAGS_IS_A_DLL=0 -DGLOG_CUSTOM_PREFIX_SUPPORT -DINDEX_32_BIT -DLIBCUDACXX_ENABLE_EXPERIMENTAL_MEMORY_RESOURCE -DMATX_DISABLE_CUB_CACHE=1 -DMATX_ENABLE_CUTENSOR=0 -DMATX_ENABLE_CUTLASS=0 -DMATX_ENABLE_FILEIO=0 -DMATX_ENABLE_PYBIND11=0 -DRXCPP_USE_FIBERS -DSPDLOG_ACTIVE_LEVEL=SPDLOG_LEVEL_INFO -DSPDLOG_FMT_EXTERNAL -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_DISABLE_ABI_NAMESPACE -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTHRUST_IGNORE_ABI_NAMESPACE_ERROR -DTRITON_ENABLE_GPU=1 -D__STDC_FORMAT_MACROS -Dmorpheus_EXPORTS -I/workspace/morpheus/_lib/include -I/workspace/build-docker/autogenerated/include -I/workspace/build-docker/morpheus/_lib -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/kernels -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust/thrust/cmake/../.. -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/lib/cmake/libcudacxx/../../../include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub/cub/cmake/../.. -I/workspace/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src -I/workspace/.cache/cpm/tritonclient/f2dd9277615c6e3c56858568bcd1e470c4d30a1f/src/c++/library -isystem /opt/conda/envs/morpheus/targets/x86_64-linux/include -isystem /opt/conda/envs/morpheus/include -isystem /workspace/.cache/cpm/pybind11/7e2e3e2d8bdbe726c035292051ad01e0121f53b5/include -isystem /opt/conda/envs/morpheus/include/python3.10 -isystem /opt/conda/envs/morpheus/include/librdkafka --expt-relaxed-constexpr --extended-lambda -O3 -DNDEBUG -std=c++17 -arch=native -Xcompiler=-fPIC --expt-relaxed-constexpr -Xcompiler -pthread -MD -MT morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o -MF morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o.d -x cu -c /workspace/morpheus/_lib/src/utilities/matx_util.cu -o morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(183): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.b = static_cast(atomicCAS(address_as_other, assumed, tmpval.i));
                                 ^

../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(242): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.h = atomicCAS(address_as_other, assumed, tmpval.i);
              ^

2 errors detected in the compilation of "../morpheus/_lib/src/utilities/matx_util.cu".
[55/77] Building CXX object morpheus/_lib/CMakeFiles/morpheus._lib.messages.dir/messages/module.cpp.o
ninja: build stopped: subcommand failed.

Full env printout

Click here to see environment details

     **git***
     commit ddd10d17060353bd019b96bbced6a1fd7f2c2fec (HEAD -> branch-24.03, origin/branch-24.03, origin/HEAD)
     Author: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
     Date:   Mon Feb 12 14:09:18 2024 -0800

     Fix or silence warnings emitted during tests (#1501)

     * Reduce the number of warnings emitted while running tests
     * Replace usage of deprecated methods/attribues where possible
     * When no other option is avail, silence known warnings.
     * Remove out-of-date warning filters

     ## By Submitting this PR I confirm:
     - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
     - When the PR is ready for review, new or existing tests cover these changes.
     - When the PR is ready for review, the documentation is up to date with these changes.

     Authors:
     - David Gardner (https://github.com/dagardner-nv)

     Approvers:
     - Michael Demoret (https://github.com/mdemoret-nv)

     URL: https://github.com/nv-morpheus/Morpheus/pull/1501
     **git submodules***
     f0a9633ec18c598263ad563e73546570051ff535 external/morpheus-visualizations (v24.03.00a-3-gf0a9633)
     6c9233a52a0f3e5dfa82a9d9397477b31a63b92e external/utilities (v24.03.00a-14-g6c9233a)

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=22.04
     DISTRIB_CODENAME=jammy
     DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
     PRETTY_NAME="Ubuntu 22.04.3 LTS"
     NAME="Ubuntu"
     VERSION_ID="22.04"
     VERSION="22.04.3 LTS (Jammy Jellyfish)"
     VERSION_CODENAME=jammy
     ID=ubuntu
     ID_LIKE=debian
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     UBUNTU_CODENAME=jammy
     Linux b50e3457cd4d 6.5.0-17-generic #17-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 11 14:01:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Tue Feb 13 05:09:55 2024
     +---------------------------------------------------------------------------------------+
     | NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
     |-----------------------------------------+----------------------+----------------------+
     | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
     |                                         |                      |               MIG M. |
     |=========================================+======================+======================|
     |   0  NVIDIA GeForce RTX 3080        Off | 00000000:01:00.0  On |                  N/A |
     | 44%   60C    P8              45W / 340W |   5963MiB / 10240MiB |     14%      Default |
     |                                         |                      |                  N/A |
     +-----------------------------------------+----------------------+----------------------+
     |   1  NVIDIA GeForce GTX 1080 Ti     Off | 00000000:02:00.0 Off |                  N/A |
     |  5%   51C    P8              12W / 250W |   1115MiB / 11264MiB |      0%      Default |
     |                                         |                      |                  N/A |
     +-----------------------------------------+----------------------+----------------------+

     +---------------------------------------------------------------------------------------+
     | Processes:                                                                            |
     |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
     |        ID   ID                                                             Usage      |
     |=======================================================================================|
     +---------------------------------------------------------------------------------------+

     ***CPU***
     Architecture:                       x86_64
     CPU op-mode(s):                     32-bit, 64-bit
     Address sizes:                      36 bits physical, 48 bits virtual
     Byte Order:                         Little Endian
     CPU(s):                             8
     On-line CPU(s) list:                0-7
     Vendor ID:                          GenuineIntel
     Model name:                         Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
     CPU family:                         6
     Model:                              42
     Thread(s) per core:                 2
     Core(s) per socket:                 4
     Socket(s):                          1
     Stepping:                           7
     CPU max MHz:                        3800.0000
     CPU min MHz:                        1600.0000
     BogoMIPS:                           6819.66
     Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
     Virtualization:                     VT-x
     L1d cache:                          128 KiB (4 instances)
     L1i cache:                          128 KiB (4 instances)
     L2 cache:                           1 MiB (4 instances)
     L3 cache:                           8 MiB (1 instance)
     NUMA node(s):                       1
     NUMA node0 CPU(s):                  0-7
     Vulnerability Gather data sampling: Not affected
     Vulnerability Itlb multihit:        KVM: Mitigation: VMX disabled
     Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
     Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Meltdown:             Mitigation; PTI
     Vulnerability Mmio stale data:      Unknown: No mitigations
     Vulnerability Retbleed:             Not affected
     Vulnerability Spec rstack overflow: Not affected
     Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
     Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
     Vulnerability Srbds:                Not affected
     Vulnerability Tsx async abort:      Not affected

     ***CMake***
     /opt/conda/envs/morpheus/bin/cmake
     cmake version 3.27.9

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /opt/conda/envs/morpheus/bin/g++
     g++ (conda-forge gcc 11.2.0-16) 11.2.0
     Copyright (C) 2021 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /opt/conda/envs/morpheus/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2023 NVIDIA Corporation
     Built on Mon_Apr__3_17:16:06_PDT_2023
     Cuda compilation tools, release 12.1, V12.1.105
     Build cuda_12.1.r12.1/compiler.32688072_0

     ***Python***
     /opt/conda/envs/morpheus/bin/python
     Python 3.10.13

     ***Environment Variables***
     PATH                            : /opt/conda/envs/morpheus/bin:/opt/conda/condabin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin
     LD_LIBRARY_PATH                 : /usr/local/nvidia/lib:/usr/local/nvidia/lib64
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /opt/conda/envs/morpheus
     PYTHON_PATH                     :

     ***conda packages***
     /opt/conda/condabin/conda
     # packages in environment at /opt/conda/envs/morpheus:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                       2_gnu    conda-forge
     _sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
     absl-py                   1.4.0              pyhd8ed1ab_0    conda-forge
     adagio                    0.2.4              pyhd8ed1ab_0    conda-forge
     alabaster                 0.7.16             pyhd8ed1ab_0    conda-forge
     alembic                   1.13.1             pyhd8ed1ab_1    conda-forge
     antlr-python-runtime      4.11.1             pyhd8ed1ab_0    conda-forge
     antlr4-python3-runtime    4.11.1             pyh1a96a4e_0    conda-forge
     appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
     argon2-cffi               23.1.0                   pypi_0    pypi
     argon2-cffi-bindings      21.2.0                   pypi_0    pypi
     astroid                   3.0.3           py310hff52083_0    conda-forge
     asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
     atk-1.0                   2.38.0               hd4edc92_1    conda-forge
     attrs                     23.2.0             pyh71513ae_0    conda-forge
     aws-c-auth                0.7.11               h0b4cabd_1    conda-forge
     aws-c-cal                 0.6.9                h14ec70c_3    conda-forge
     aws-c-common              0.9.12               hd590300_0    conda-forge
     aws-c-compression         0.2.17               h572eabf_8    conda-forge
     aws-c-event-stream        0.4.1                h97bb272_2    conda-forge
     aws-c-http                0.8.0                h9129f04_2    conda-forge
     aws-c-io                  0.14.0               hf8f278a_1    conda-forge
     aws-c-mqtt                0.10.1               h2b97f5f_0    conda-forge
     aws-c-s3                  0.4.9                hca09fc5_0    conda-forge
     aws-c-sdkutils            0.1.13               h572eabf_1    conda-forge
     aws-checksums             0.1.17               h572eabf_7    conda-forge
     aws-crt-cpp               0.26.0               h04327c0_8    conda-forge
     aws-sdk-cpp               1.11.210            hba3e011_10    conda-forge
     babel                     2.14.0             pyhd8ed1ab_0    conda-forge
     bcrypt                    4.1.2           py310hcb5633a_0    conda-forge
     beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
     benchmark                 1.8.3                h59595ed_0    conda-forge
     betterproto               1.2.5              pyhd3deb0d_1    conda-forge
     binutils                  2.36.1               hdd6e379_2    conda-forge
     binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
     binutils_linux-64         2.36                hf3e587d_10    conda-forge
     blas                      1.0                         mkl    conda-forge
     bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
     blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
     bokeh                     3.3.4              pyhd8ed1ab_0    conda-forge
     boost-cpp                 1.84.0               h44aadfe_0    conda-forge
     breathe                   4.35.0             pyhd8ed1ab_1    conda-forge
     brotli                    1.1.0                hd590300_1    conda-forge
     brotli-bin                1.1.0                hd590300_1    conda-forge
     brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
     bzip2                     1.0.8                hd590300_5    conda-forge
     c-ares                    1.26.0               hd590300_0    conda-forge
     c-compiler                1.5.2                h0b41bf4_0    conda-forge
     ca-certificates           2024.2.2             hbcca054_0    conda-forge
     cachetools                5.3.2              pyhd8ed1ab_0    conda-forge
     cairo                     1.18.0               h3faef2a_0    conda-forge
     cattrs                    23.2.3             pyhd8ed1ab_0    conda-forge
     ccache                    4.9.1                h1fcd64f_0    conda-forge
     certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
     cffi                      1.16.0          py310h2fee648_0    conda-forge
     cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
     charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
     clang                     16.0.6               hda56bd4_4    conda-forge
     clang-16                  16.0.6          default_hb11cfb5_4    conda-forge
     clang-format              16.0.6          default_hb11cfb5_4    conda-forge
     clang-format-16           16.0.6          default_hb11cfb5_4    conda-forge
     clang-tools               16.0.6          default_hb11cfb5_4    conda-forge
     clangdev                  16.0.6          default_hb11cfb5_4    conda-forge
     clangxx                   16.0.6          default_hb11cfb5_4    conda-forge
     click                     8.1.7           unix_pyh707e725_0    conda-forge
     cloudpickle               3.0.0              pyhd8ed1ab_0    conda-forge
     cmake                     3.27.9               hcfe8598_0    conda-forge
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     configparser              5.3.0              pyhd8ed1ab_0    conda-forge
     contourpy                 1.2.0           py310hd41b1e2_0    conda-forge
     coverage                  7.4.1           py310h2372a71_0    conda-forge
     cryptography              42.0.2          py310hb8475ec_0    conda-forge
     cuda-cccl                 12.1.109             ha770c72_0    conda-forge
     cuda-cccl-impl            2.0.1                ha770c72_1    conda-forge
     cuda-cccl_linux-64        12.1.109             ha770c72_0    conda-forge
     cuda-command-line-tools   12.1.1               ha770c72_0    conda-forge
     cuda-cudart               12.1.105             hd3aeb46_0    conda-forge
     cuda-cudart-dev           12.1.105             hd3aeb46_0    conda-forge
     cuda-cudart-dev_linux-64  12.1.105             h59595ed_0    conda-forge
     cuda-cudart-static        12.1.105             hd3aeb46_0    conda-forge
     cuda-cudart-static_linux-64 12.1.105             h59595ed_0    conda-forge
     cuda-cudart_linux-64      12.1.105             h59595ed_0    conda-forge
     cuda-cupti                12.1.105             h59595ed_0    conda-forge
     cuda-cupti-dev            12.1.105             h59595ed_0    conda-forge
     cuda-driver-dev           12.1.105             hd3aeb46_0    conda-forge
     cuda-driver-dev_linux-64  12.1.105             h59595ed_0    conda-forge
     cuda-gdb                  12.1.105             hd47b8d6_0    conda-forge
     cuda-libraries            12.1.0                        0    nvidia
     cuda-libraries-dev        12.1.0                        0    nvidia
     cuda-libraries-static     12.1.1               ha770c72_0    conda-forge
     cuda-nsight               12.1.105             ha770c72_0    conda-forge
     cuda-nsight-compute       12.3.2                        0    nvidia
     cuda-nvcc                 12.1.105             hcdd1206_1    conda-forge
     cuda-nvcc-dev_linux-64    12.1.105             ha770c72_0    conda-forge
     cuda-nvcc-impl            12.1.105             hd3aeb46_0    conda-forge
     cuda-nvcc-tools           12.1.105             hd3aeb46_0    conda-forge
     cuda-nvcc_linux-64        12.1.105             h8a487aa_1    conda-forge
     cuda-nvdisasm             12.1.105             h59595ed_0    conda-forge
     cuda-nvml-dev             12.1.105             h59595ed_0    conda-forge
     cuda-nvprof               12.1.105             h59595ed_0    conda-forge
     cuda-nvrtc                12.1.105             hd3aeb46_0    conda-forge
     cuda-nvrtc-dev            12.1.105             hd3aeb46_0    conda-forge
     cuda-nvrtc-static         12.1.105             hd3aeb46_0    conda-forge
     cuda-nvtx                 12.1.105             h59595ed_0    conda-forge
     cuda-nvtx-dev             12.1.105             ha770c72_0    conda-forge
     cuda-nvvp                 12.1.105             h59595ed_0    conda-forge
     cuda-opencl               12.1.105             h59595ed_0    conda-forge
     cuda-opencl-dev           12.1.105             h59595ed_0    conda-forge
     cuda-profiler-api         12.1.105             ha770c72_0    conda-forge
     cuda-python               12.3.0          py310h9f9f131_2    conda-forge
     cuda-runtime              12.1.0                        0    nvidia
     cuda-sanitizer-api        12.1.105             h59595ed_0    conda-forge
     cuda-tools                12.1.0                        0    nvidia
     cuda-version              12.1                 h1d6eff3_2    conda-forge
     cuda-visual-tools         12.1.0                        0    nvidia
     cudf                      24.02.00a244    cuda12_py310_240212_gc78033b7e7_244    rapidsai-nightly
     cupy                      13.0.0          py310h7aad9d2_3    conda-forge
     cupy-core                 13.0.0          py310had4011e_3    conda-forge
     cxx-compiler              1.5.2                hf52228f_0    conda-forge
     cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
     cyrus-sasl                2.1.27               h54b06d7_7    conda-forge
     cython                    3.0.8           py310hc6cd4ac_0    conda-forge
     cytoolz                   0.12.3          py310h2372a71_0    conda-forge
     dask                      2024.2.0           pyhd8ed1ab_0    conda-forge
     dask-core                 2024.2.0           pyhd8ed1ab_0    conda-forge
     dask-cuda                 24.02.00a28     py310_240212_g34e7404_28    rapidsai-nightly
     dask-cudf                 24.02.00a244    cuda12_py310_240212_gc78033b7e7_244    rapidsai-nightly
     databricks-cli            0.18.0             pyhd8ed1ab_0    conda-forge
     databricks-connect        14.3.0                   pypi_0    pypi
     databricks-sdk            0.19.0                   pypi_0    pypi
     dataclasses               0.8                pyhc8e2a94_3    conda-forge
     datacompy                 0.10.5             pyhd8ed1ab_0    conda-forge
     decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
     defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
     dill                      0.3.7              pyhd8ed1ab_0    conda-forge
     distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
     distributed               2024.2.0           pyhd8ed1ab_0    conda-forge
     distro                    1.9.0              pyhd8ed1ab_0    conda-forge
     dlpack                    0.5                  h9c3ff4c_0    conda-forge
     docker-py                 6.1.3              pyhd8ed1ab_0    conda-forge
     docutils                  0.17.1          py310hff52083_4    conda-forge
     doxygen                   1.9.2                hb166930_0    conda-forge
     elastic-transport         8.12.0             pyhd8ed1ab_0    conda-forge
     elasticsearch             8.9.0              pyhd8ed1ab_0    conda-forge
     entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     environs                  9.5.0                    pypi_0    pypi
     exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
     executing                 2.0.1              pyhd8ed1ab_0    conda-forge
     exhale                    0.3.6              pyhd8ed1ab_0    conda-forge
     expat                     2.5.0                hcb278e6_1    conda-forge
     fastrlock                 0.8.2           py310hc6cd4ac_2    conda-forge
     feedparser                6.0.10             pyhd8ed1ab_0    conda-forge
     filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
     flake8                    7.0.0              pyhd8ed1ab_0    conda-forge
     flask                     3.0.2              pyhd8ed1ab_0    conda-forge
     fmt                       10.2.1               h00ab1b0_0    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
     fontconfig                2.14.2               h14ed4e7_0    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     fonttools                 4.48.1          py310h2372a71_0    conda-forge
     freetype                  2.12.1               h267a509_2    conda-forge
     fribidi                   1.0.10               h36c2ea0_0    conda-forge
     fs                        2.4.16             pyhd8ed1ab_0    conda-forge
     fsspec                    2024.2.0           pyhca7485f_0    conda-forge
     fugue                     0.8.7              pyhd8ed1ab_0    conda-forge
     fugue-sql-antlr           0.2.0              pyhd8ed1ab_0    conda-forge
     gcc                       11.2.0              h702ea55_10    conda-forge
     gcc_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
     gcc_linux-64              11.2.0              h39a9532_10    conda-forge
     gdk-pixbuf                2.42.10              h829c605_4    conda-forge
     gds-tools                 1.6.1.9              hd3aeb46_0    conda-forge
     gettext                   0.21.1               h27087fc_0    conda-forge
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     giflib                    5.2.1                h0b41bf4_3    conda-forge
     git-lfs                   3.4.1                ha770c72_0    conda-forge
     gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
     gitpython                 3.1.41             pyhd8ed1ab_0    conda-forge
     glog                      0.6.0                h6f12383_0    conda-forge
     gmock                     1.14.0               ha770c72_1    conda-forge
     gmp                       6.3.0                h59595ed_0    conda-forge
     gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
     google-auth               2.27.0                   pypi_0    pypi
     googleapis-common-protos  1.62.0             pyhd8ed1ab_0    conda-forge
     graphite2                 1.3.13            h58526e2_1001    conda-forge
     graphviz                  9.0.0                h78e8752_1    conda-forge
     greenlet                  3.0.3           py310hc6cd4ac_0    conda-forge
     grpcio                    1.60.0                   pypi_0    pypi
     grpcio-status             1.60.0                   pypi_0    pypi
     grpclib                   0.4.7              pyhd8ed1ab_0    conda-forge
     gtest                     1.14.0               h00ab1b0_1    conda-forge
     gtk2                      2.24.33              h7f000aa_3    conda-forge
     gts                       0.7.6                h977cf35_4    conda-forge
     gunicorn                  21.2.0          py310hff52083_1    conda-forge
     gxx                       11.2.0              h702ea55_10    conda-forge
     gxx_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
     gxx_linux-64              11.2.0              hacbe6df_10    conda-forge
     h2                        4.1.0              pyhd8ed1ab_0    conda-forge
     harfbuzz                  8.3.0                h3d44ed6_0    conda-forge
     hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
     hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
     icu                       73.2                 h59595ed_0    conda-forge
     identify                  2.5.34             pyhd8ed1ab_0    conda-forge
     idna                      3.6                pyhd8ed1ab_0    conda-forge
     imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
     importlib-metadata        7.0.1              pyha770c72_0    conda-forge
     importlib_metadata        7.0.1                hd8ed1ab_0    conda-forge
     importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
     include-what-you-use      0.20                 h59595ed_0    conda-forge
     iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
     intel-openmp              2023.1.0         hdb19cb5_46306
     ipython                   8.21.0             pyh707e725_0    conda-forge
     isort                     5.13.2             pyhd8ed1ab_0    conda-forge
     itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
     jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
     joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
     jsonschema                4.21.1             pyhd8ed1ab_0    conda-forge
     jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
     jupyter_client            8.6.0              pyhd8ed1ab_0    conda-forge
     jupyter_core              5.7.1           py310hff52083_0    conda-forge
     jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
     kafka-python              2.0.2                    pypi_0    pypi
     kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     kiwisolver                1.4.5           py310hd41b1e2_1    conda-forge
     krb5                      1.21.2               h659d440_0    conda-forge
     lcms2                     2.16                 hb7c19ff_0    conda-forge
     ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
     lerc                      4.0.0                h27087fc_0    conda-forge
     libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
     libarrow                  14.0.2           h84dd17c_3_cpu    conda-forge
     libarrow-acero            14.0.2           h59595ed_3_cpu    conda-forge
     libarrow-dataset          14.0.2           h59595ed_3_cpu    conda-forge
     libarrow-flight           14.0.2           h120cb0d_3_cpu    conda-forge
     libarrow-flight-sql       14.0.2           h61ff412_3_cpu    conda-forge
     libarrow-gandiva          14.0.2           hacb8726_3_cpu    conda-forge
     libarrow-substrait        14.0.2           h61ff412_3_cpu    conda-forge
     libblas                   3.9.0           1_h86c2bf4_netlib    conda-forge
     libboost                  1.84.0               h6fcfa73_0    conda-forge
     libboost-devel            1.84.0               h00ab1b0_0    conda-forge
     libboost-headers          1.84.0               ha770c72_0    conda-forge
     libbrotlicommon           1.1.0                hd590300_1    conda-forge
     libbrotlidec              1.1.0                hd590300_1    conda-forge
     libbrotlienc              1.1.0                hd590300_1    conda-forge
     libcblas                  3.9.0           5_h92ddd45_netlib    conda-forge
     libclang                  16.0.6          default_hb11cfb5_4    conda-forge
     libclang-cpp              16.0.6          default_hb11cfb5_4    conda-forge
     libclang-cpp16            16.0.6          default_hb11cfb5_4    conda-forge
     libclang13                16.0.6          default_ha2b6cf4_4    conda-forge
     libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
     libcublas                 12.1.0.26                     0    nvidia
     libcublas-dev             12.1.0.26                     0    nvidia
     libcublas-static          12.1.3.1             hd3aeb46_0    conda-forge
     libcudf                   24.02.00a244    cuda12_240212_gc78033b7e7_244    rapidsai-nightly
     libcufft                  11.0.2.4                      0    nvidia
     libcufft-dev              11.0.2.4                      0    nvidia
     libcufft-static           11.0.2.54            hd3aeb46_0    conda-forge
     libcufile                 1.6.1.9              hd3aeb46_0    conda-forge
     libcufile-dev             1.6.1.9              hd3aeb46_0    conda-forge
     libcufile-static          1.6.1.9              hd3aeb46_0    conda-forge
     libcurand                 10.3.2.106           hd3aeb46_0    conda-forge
     libcurand-dev             10.3.2.106           hd3aeb46_0    conda-forge
     libcurand-static          10.3.2.106           hd3aeb46_0    conda-forge
     libcurl                   8.5.0                hca28451_0    conda-forge
     libcusolver               11.4.4.55                     0    nvidia
     libcusolver-dev           11.4.4.55                     0    nvidia
     libcusolver-static        11.4.5.107           hd3aeb46_0    conda-forge
     libcusparse               12.0.2.55                     0    nvidia
     libcusparse-dev           12.0.2.55                     0    nvidia
     libcusparse-static        12.1.0.106           hd3aeb46_0    conda-forge
     libdeflate                1.19                 hd590300_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 hd590300_2    conda-forge
     libevent                  2.1.12               hf998b51_1    conda-forge
     libexpat                  2.5.0                hcb278e6_1    conda-forge
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-devel_linux-64     11.2.0              h0952999_16    conda-forge
     libgcc-ng                 13.2.0               h807b86a_5    conda-forge
     libgd                     2.3.3                h119a65a_9    conda-forge
     libgfortran-ng            13.2.0               h69a702a_5    conda-forge
     libgfortran5              13.2.0               ha4646dd_5    conda-forge
     libglib                   2.78.3               h783c2da_0    conda-forge
     libgomp                   13.2.0               h807b86a_5    conda-forge
     libgoogle-cloud           2.12.0               h5206363_4    conda-forge
     libgrpc                   1.59.3               hd6c4280_0    conda-forge
     libhiredis                1.0.2                h2cc385e_0    conda-forge
     libhwloc                  2.9.2           default_h554bfaf_1009    conda-forge
     libiconv                  1.17                 hd590300_2    conda-forge
     libjpeg-turbo             3.0.0                hd590300_1    conda-forge
     libkvikio                 24.02.00a       cuda12_240212_g6b7d9f0_18    rapidsai-nightly
     liblapack                 3.9.0           5_h92ddd45_netlib    conda-forge
     libllvm14                 14.0.6               hcd5def8_4    conda-forge
     libllvm15                 15.0.7               hb3ce162_4    conda-forge
     libllvm16                 16.0.6               hb3ce162_3    conda-forge
     libmrc                    24.03.00a       cuda_12.1_h0dae25b_13    nvidia/label/dev
     libnghttp2                1.58.0               h47da74e_1    conda-forge
     libnl                     3.9.0                hd590300_0    conda-forge
     libnpp                    12.0.2.50                     0    nvidia
     libnpp-dev                12.0.2.50                     0    nvidia
     libnpp-static             12.1.0.40            hd3aeb46_0    conda-forge
     libnsl                    2.0.1                hd590300_0    conda-forge
     libntlm                   1.4               h7f98852_1002    conda-forge
     libnuma                   2.0.16               h0b41bf4_1    conda-forge
     libnvjitlink              12.1.105             hd3aeb46_0    conda-forge
     libnvjitlink-dev          12.1.105             hd3aeb46_0    conda-forge
     libnvjitlink-static       12.1.105             hd3aeb46_0    conda-forge
     libnvjpeg                 12.1.1.14                     0    nvidia
     libnvjpeg-dev             12.1.1.14                     0    nvidia
     libnvjpeg-static          12.2.0.2             ha770c72_0    conda-forge
     libparquet                14.0.2           h352af49_3_cpu    conda-forge
     libpng                    1.6.42               h2797004_0    conda-forge
     libprotobuf               4.24.4               hf27288f_0    conda-forge
     librdkafka                1.9.2                ha5a0de0_2    conda-forge
     libre2-11                 2023.06.02           h7a70373_0    conda-forge
     librmm                    24.02.00a37     cuda12_240212_gf32d35b4_37    rapidsai-nightly
     librsvg                   2.56.3               he3f83f7_1    conda-forge
     libsanitizer              11.2.0              he4da1e4_16    conda-forge
     libsodium                 1.0.18               h36c2ea0_1    conda-forge
     libsqlite                 3.45.1               h2797004_0    conda-forge
     libssh2                   1.11.0               h0841786_0    conda-forge
     libstdcxx-devel_linux-64  11.2.0              h0952999_16    conda-forge
     libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
     libthrift                 0.19.0               hb90f79a_1    conda-forge
     libtiff                   4.6.0                ha9c0a0a_2    conda-forge
     libutf8proc               2.8.0                h166bdaf_0    conda-forge
     libuuid                   2.38.1               h0b41bf4_0    conda-forge
     libuv                     1.46.0               hd590300_0    conda-forge
     libwebp                   1.3.2                h658648e_1    conda-forge
     libwebp-base              1.3.2                hd590300_0    conda-forge
     libxcb                    1.15                 h0b41bf4_0    conda-forge
     libxcrypt                 4.4.36               hd590300_1    conda-forge
     libxml2                   2.12.5               h232c23b_0    conda-forge
     libxslt                   1.1.39               h76b75d6_0    conda-forge
     libzlib                   1.2.13               hd590300_5    conda-forge
     llvm-openmp               15.0.7               h0cdce71_0    conda-forge
     llvm-tools                16.0.6               hb3ce162_3    conda-forge
     llvmdev                   16.0.6               hb3ce162_3    conda-forge
     llvmlite                  0.40.1          py310h1b8f574_0    conda-forge
     locket                    1.0.0              pyhd8ed1ab_0    conda-forge
     lxml                      5.1.0           py310hcfd0673_0    conda-forge
     lz4                       4.3.3           py310h350c4a5_0    conda-forge
     lz4-c                     1.9.4                hcb278e6_0    conda-forge
     mako                      1.3.2              pyhd8ed1ab_0    conda-forge
     markdown                  3.5.2              pyhd8ed1ab_0    conda-forge
     markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.5           py310h2372a71_0    conda-forge
     marshmallow               3.20.2                   pypi_0    pypi
     matplotlib-base           3.8.2           py310h62c0568_0    conda-forge
     matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
     mccabe                    0.7.0              pyhd8ed1ab_0    conda-forge
     mdit-py-plugins           0.4.0              pyhd8ed1ab_0    conda-forge
     mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
     merlin-core               23.08.00                   py_0    nvidia
     merlin-dataloader         23.08.00                   py_0    nvidia
     milvus                    2.3.5                    pypi_0    pypi
     minio                     7.2.4                    pypi_0    pypi
     mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
     mkl                       2023.1.0         h213fc3f_46344
     mlflow                    2.9.2           py310ha13cd29_0    conda-forge
     mpc                       1.3.1                hfe3b2da_0    conda-forge
     mpfr                      4.2.1                h9458935_0    conda-forge
     mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
     mrc                       24.03.00a       cuda_12.1_py310_h572eed8_13    nvidia/label/dev
     msgpack-python            1.0.7           py310hd41b1e2_0    conda-forge
     multidict                 6.0.5           py310h2372a71_0    conda-forge
     munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
     myst-parser               0.18.1             pyhd8ed1ab_0    conda-forge
     nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
     nbconvert                 7.16.0             pyhd8ed1ab_0    conda-forge
     nbconvert-core            7.16.0             pyhd8ed1ab_0    conda-forge
     nbconvert-pandoc          7.16.0             pyhd8ed1ab_0    conda-forge
     nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
     nbsphinx                  0.9.3              pyhd8ed1ab_0    conda-forge
     ncurses                   6.4                  h59595ed_2    conda-forge
     networkx                  2.8.8              pyhd8ed1ab_0    conda-forge
     ninja                     1.11.1               h924138e_0    conda-forge
     nlohmann_json             3.9.1                h9c3ff4c_1    conda-forge
     nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
     nodejs                    18.19.0              hb753e55_0    conda-forge
     npy-append-array          0.9.16             pyhd8ed1ab_0    conda-forge
     nsight-compute            2023.3.1.1                    0    nvidia
     numba                     0.57.1          py310h0f6aa51_0    conda-forge
     numpy                     1.24.4          py310ha4c1d20_0    conda-forge
     numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge
     nvcomp                    3.0.5                h10b603f_0    conda-forge
     nvtabular                 23.08.00                py310_0    nvidia
     nvtx                      0.2.8           py310h2372a71_1    conda-forge
     oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
     ocl-icd                   2.3.1                h7f98852_0    conda-forge
     openjpeg                  2.5.0                h488ebb8_3    conda-forge
     openssl                   3.2.1                hd590300_0    conda-forge
     orc                       1.9.2                h4b38347_0    conda-forge
     ordered-set               4.1.0              pyhd8ed1ab_0    conda-forge
     packaging                 23.2               pyhd8ed1ab_0    conda-forge
     pandas                    1.5.3           py310h9b08913_1    conda-forge
     pandoc                    3.1.11.1             ha770c72_0    conda-forge
     pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
     pango                     1.50.14              ha41ecd1_2    conda-forge
     paramiko                  3.4.0              pyhd8ed1ab_0    conda-forge
     parso                     0.8.3              pyhd8ed1ab_0    conda-forge
     partd                     1.4.1              pyhd8ed1ab_0    conda-forge
     pcre2                     10.42                hcad00b1_0    conda-forge
     pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
     pickleshare               0.7.5                   py_1003    conda-forge
     pillow                    10.2.0          py310h01dd4db_0    conda-forge
     pip                       24.0               pyhd8ed1ab_0    conda-forge
     pixman                    0.43.2               h59595ed_0    conda-forge
     pkg-config                0.29.2            h36c2ea0_1008    conda-forge
     pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
     platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
     pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
     port-for                  0.7.2                    pypi_0    pypi
     pre-commit                3.6.1              pyha770c72_0    conda-forge
     prometheus_client         0.19.0             pyhd8ed1ab_0    conda-forge
     prometheus_flask_exporter 0.23.0             pyhd8ed1ab_0    conda-forge
     prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
     protobuf                  4.24.4          py310h620c231_0    conda-forge
     psutil                    5.9.8           py310h2372a71_0    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
     pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
     py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
     py4j                      0.10.9.7                 pypi_0    pypi
     pyarrow                   14.0.2          py310hf9e7431_3_cpu    conda-forge
     pyarrow-hotfix            0.6                pyhd8ed1ab_0    conda-forge
     pyasn1                    0.5.1                    pypi_0    pypi
     pyasn1-modules            0.3.0                    pypi_0    pypi
     pybind11-stubgen          0.10.5             pyhd8ed1ab_0    conda-forge
     pycodestyle               2.11.1             pyhd8ed1ab_0    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pycryptodome              3.20.0                   pypi_0    pypi
     pyflakes                  3.2.0              pyhd8ed1ab_0    conda-forge
     pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
     pyjwt                     2.8.0              pyhd8ed1ab_1    conda-forge
     pylint                    3.0.3              pyhd8ed1ab_0    conda-forge
     pymilvus                  2.3.6                    pypi_0    pypi
     pymupdf                   1.23.21                  pypi_0    pypi
     pymupdfb                  1.23.9                   pypi_0    pypi
     pynacl                    1.5.0           py310h2372a71_3    conda-forge
     pynvjitlink               0.1.11          py310hbc2f0e0_0    rapidsai-nightly
     pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
     pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
     pysocks                   1.7.1              pyha2e5f31_6    conda-forge
     pytest                    7.4.4              pyhd8ed1ab_0    conda-forge
     pytest-asyncio            0.23.5             pyhd8ed1ab_0    conda-forge
     pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
     pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
     pytest-kafka              0.6.0                    pypi_0    pypi
     python                    3.10.13         hd12c33a_1_cpython    conda-forge
     python-confluent-kafka    1.9.2           py310h5764c6d_2    conda-forge
     python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
     python-docx               1.1.0              pyhd8ed1ab_0    conda-forge
     python-dotenv             1.0.1                    pypi_0    pypi
     python-fastjsonschema     2.19.1             pyhd8ed1ab_0    conda-forge
     python-graphviz           0.20.1             pyh22cad53_0    conda-forge
     python-rapidjson          1.14            py310hc6cd4ac_0    conda-forge
     python_abi                3.10                    4_cp310    conda-forge
     pytorch                   2.2.0           py3.10_cuda12.1_cudnn8.9.2_0    pytorch
     pytorch-cuda              12.1                 ha16c6d3_5    pytorch
     pytorch-mutex             1.0                        cuda    pytorch
     pytz                      2023.4             pyhd8ed1ab_0    conda-forge
     pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
     pyyaml                    6.0.1           py310h2372a71_1    conda-forge
     pyzmq                     25.1.2          py310h795f18f_0    conda-forge
     qpd                       0.4.4              pyhd8ed1ab_1    conda-forge
     querystring_parser        1.2.4                      py_0    conda-forge
     rapidjson                 1.1.0             he1b5a44_1002    conda-forge
     rapids-dask-dependency    24.02.00a11                   0    rapidsai-nightly
     rdma-core                 50.0                 hd3aeb46_0    conda-forge
     re2                       2023.06.02           h2873b5e_0    conda-forge
     readline                  8.2                  h8228510_1    conda-forge
     referencing               0.33.0             pyhd8ed1ab_0    conda-forge
     requests                  2.31.0             pyhd8ed1ab_0    conda-forge
     requests-cache            1.1.1              pyhd8ed1ab_0    conda-forge
     rhash                     1.4.4                hd590300_0    conda-forge
     rich                      13.7.0             pyhd8ed1ab_0    conda-forge
     rmm                       24.02.00a37     cuda12_py310_240212_gf32d35b4_37    rapidsai-nightly
     rpds-py                   0.17.1          py310hcb5633a_0    conda-forge
     rsa                       4.9                      pypi_0    pypi
     s2n                       1.4.1                h06160fa_0    conda-forge
     scikit-build              0.17.6             pyh4af843d_0    conda-forge
     scikit-learn              1.3.2           py310h1fdf081_2    conda-forge
     scipy                     1.12.0          py310hb13e2d6_2    conda-forge
     setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
     sgmllib3k                 1.0.0              pyh9f0ad1d_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
     snappy                    1.1.10               h9fff704_0    conda-forge
     snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
     spdlog                    1.12.0               hd2e6256_2    conda-forge
     sphinx                    4.5.0              pyh6c4a22f_0    conda-forge
     sphinx_rtd_theme          1.3.0              pyha770c72_0    conda-forge
     sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
     sphinxcontrib-htmlhelp    2.0.1              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jquery      4.1                pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
     sqlalchemy                2.0.26          py310h2372a71_0    conda-forge
     sqlglot                   21.0.2             pyhd8ed1ab_0    conda-forge
     sqlparse                  0.4.4              pyhd8ed1ab_0    conda-forge
     stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
     stringcase                1.2.0                      py_0    conda-forge
     sympy                     1.12            pypyh9d50eac_103    conda-forge
     sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
     tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
     tbb                       2021.10.0            h00ab1b0_0    conda-forge
     tblib                     3.0.0              pyhd8ed1ab_0    conda-forge
     tensorflow-metadata       1.13.1             pyhd8ed1ab_0    conda-forge
     threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
     tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
     tk                        8.6.13          noxft_h4845f30_101    conda-forge
     toml                      0.10.2             pyhd8ed1ab_0    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     tomlkit                   0.12.3             pyha770c72_0    conda-forge
     toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
     torchtriton               2.2.0                     py310    pytorch
     tornado                   6.3.3           py310h2372a71_1    conda-forge
     tqdm                      4.66.2             pyhd8ed1ab_0    conda-forge
     traitlets                 5.14.1             pyhd8ed1ab_0    conda-forge
     triad                     0.9.5              pyhd8ed1ab_0    conda-forge
     tritonclient              2.34.0             pyhff2d567_0    conda-forge
     typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
     typing_extensions         4.9.0              pyha770c72_0    conda-forge
     typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
     tzdata                    2024a                h0c530f3_0    conda-forge
     ucx                       1.15.0               h6d2d1ec_3    conda-forge
     ujson                     5.9.0           py310hc6cd4ac_0    conda-forge
     ukkonen                   1.0.1           py310hd41b1e2_4    conda-forge
     unicodedata2              15.1.0          py310h2372a71_0    conda-forge
     url-normalize             1.4.3              pyhd8ed1ab_0    conda-forge
     urllib3                   2.2.0              pyhd8ed1ab_0    conda-forge
     versioneer                0.29               pyhd8ed1ab_0    conda-forge
     virtualenv                20.25.0            pyhd8ed1ab_0    conda-forge
     watchdog                  3.0.0           py310hff52083_1    conda-forge
     wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
     webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
     websocket-client          1.7.0              pyhd8ed1ab_0    conda-forge
     websockets                12.0            py310h2372a71_0    conda-forge
     werkzeug                  3.0.1              pyhd8ed1ab_0    conda-forge
     wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
     xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
     xorg-libice               1.1.1                hd590300_0    conda-forge
     xorg-libsm                1.2.4                h7391055_0    conda-forge
     xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
     xorg-libxau               1.0.11               hd590300_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
     xorg-libxrender           0.9.11               hd590300_0    conda-forge
     xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
     xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
     xorg-xproto               7.0.31            h7f98852_1007    conda-forge
     xyzservices               2023.10.1          pyhd8ed1ab_0    conda-forge
     xz                        5.2.6                h166bdaf_0    conda-forge
     yaml                      0.2.5                h7f98852_2    conda-forge
     yapf                      0.40.1             pyhd8ed1ab_0    conda-forge
     zeromq                    4.3.5                h59595ed_0    conda-forge
     zict                      3.0.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.2.13               hd590300_5    conda-forge
     zstd                      1.5.5                hfc55251_0    conda-forge

Other/Misc.

No response

Code of Conduct

jarmak-nv commented 8 months ago

Hi @yousecjoe!

Thanks for submitting this issue - our team has been notified and we'll get back to you as soon as we can! In the mean time, feel free to add any relevant information to this issue.

efajardo-nv commented 8 months ago

@yousecjoe I was just able run through the entire example successfully. Could you please try the following steps?

  1. Make sure you have the very latest from branch-24.03.
  2. Delete the build and cache files from your container. In your container workspace directory:
    rm -rf build-docker
    rm -rf .cache
  3. Then compile again:
    BUILD_DIR=build-docker ./scripts/compile.sh

    If you're able to get past that, I noticed that you'll also have to manually install pydantic:

    mamba install pydantic

    We already have an issue to pre-install it in the container.

yousecjoe commented 8 months ago

I was using the latest from branch 24.03, and I tried again with the latest from branch 24.03 and got the same issue.

Looks like it is failing at a different step, but it is the same error: no instance of overloaded function "atomicCAS"

-- Generating done (0.2s)
-- Build files have been written to: /workspace/build-docker
+ echo 'Running CMake build...'
Running CMake build...
+ cmake --build build-docker -j
[0/2] Re-checking globbed directories...
[342/370] Building CUDA object morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
FAILED: morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o 
/workspace/build-docker/run_ccache_cuda.sh /opt/conda/envs/morpheus/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-c++ -DBOOST_ATOMIC_DYN_LINK -DBOOST_ATOMIC_NO_LIB -DBOOST_CONTEXT_DYN_LINK -DBOOST_CONTEXT_NO_LIB -DBOOST_FIBER_DYN_LINK -DBOOST_FIBER_NO_LIB -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_FILESYSTEM_NO_LIB -DFMT_HEADER_ONLY=1 -DGFLAGS_IS_A_DLL=0 -DGLOG_CUSTOM_PREFIX_SUPPORT -DINDEX_32_BIT -DLIBCUDACXX_ENABLE_EXPERIMENTAL_MEMORY_RESOURCE -DMATX_DISABLE_CUB_CACHE=1 -DMATX_ENABLE_CUTENSOR=0 -DMATX_ENABLE_CUTLASS=0 -DMATX_ENABLE_FILEIO=0 -DMATX_ENABLE_PYBIND11=0 -DRXCPP_USE_FIBERS -DSPDLOG_ACTIVE_LEVEL=SPDLOG_LEVEL_INFO -DSPDLOG_FMT_EXTERNAL -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_DISABLE_ABI_NAMESPACE -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTHRUST_IGNORE_ABI_NAMESPACE_ERROR -DTRITON_ENABLE_GPU=1 -D__STDC_FORMAT_MACROS -Dmorpheus_EXPORTS -I/workspace/morpheus/_lib/include -I/workspace/build-docker/autogenerated/include -I/workspace/build-docker/morpheus/_lib -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include -I/workspace/.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/kernels -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/thrust/thrust/cmake/../.. -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/libcudacxx/lib/cmake/libcudacxx/../../../include -I/workspace/.cache/cpm/cccl/4eb28e32f7fa15e673d2dd5a08a503dffab263dd/cub/cub/cmake/../.. -I/workspace/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src -I/workspace/.cache/cpm/tritonclient/f2dd9277615c6e3c56858568bcd1e470c4d30a1f/src/c++/library -isystem /opt/conda/envs/morpheus/targets/x86_64-linux/include -isystem /opt/conda/envs/morpheus/include -isystem /workspace/.cache/cpm/pybind11/7e2e3e2d8bdbe726c035292051ad01e0121f53b5/include -isystem /opt/conda/envs/morpheus/include/python3.10 -isystem /opt/conda/envs/morpheus/include/librdkafka --expt-relaxed-constexpr --extended-lambda -O3 -DNDEBUG -std=c++17 -arch=native -Xcompiler=-fPIC --expt-relaxed-constexpr -MD -MT morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o -MF morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o.d -x cu -c /workspace/morpheus/_lib/src/utilities/matx_util.cu -o morpheus/_lib/CMakeFiles/morpheus.dir/src/utilities/matx_util.cu.o
../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(183): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.b = static_cast<float>(atomicCAS(address_as_other, assumed, tmpval.i));
                                 ^

../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(242): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.h = atomicCAS(address_as_other, assumed, tmpval.i);
              ^

2 errors detected in the compilation of "../morpheus/_lib/src/utilities/matx_util.cu".
[351/370] Building CXX object morpheus/_lib/CMakeFiles/morpheus._lib.messages.dir/messages/module.cpp.o
ninja: build stopped: subcommand failed.

I wiped the directory and pulled the latest data to try again. It stops consistently at

../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(183): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.b = static_cast<float>(atomicCAS(address_as_other, assumed, tmpval.i));
                                 ^

../.cache/cpm/matx/99d7b1a13efae6c3bcfb3585d8b74c0d9e17300f/include/matx/transforms/reduce.h(242): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, short)
      old.h = atomicCAS(address_as_other, assumed, tmpval.i);
              ^

2 errors detected in the compilation of "../morpheus/_lib/src/utilities/matx_util.cu".
[351/370] Building CXX object morpheus/_lib/CMakeFiles/morpheus._lib.messages.dir/messages/module.cpp.o
ninja: build stopped: subcommand failed.
efajardo-nv commented 8 months ago

@yousecjoe We noticed that one of your GPUs is GeForce GTX 1080 Ti. We just recently updated the 24.03 development branch to use RAPIDS 24.02 which no longer includes support for Pascal GPUs. Therefore only your GeForce RTX 3080 is supported for 24.03. You have two options:

  1. Switch to the Morpheus 23.11 release.
  2. If you'd prefer to stay with 24.03, you'll have to update your examples/sid_visualization/docker-compose.yml to only use the supported GPU. At the top of the file, update the x-with-gpus section to look like this:
    x-with-gpus: &with_gpus
    deploy:
    resources:
      reservations:
        devices:
          - driver: "nvidia"
            device_ids: ["0"]
            capabilities: [gpu]

    We'll also make sure the docs are updated to reflect the changes in the GPU requirements.

yousecjoe commented 8 months ago

Thank you, that resolved the issue.