Installation on AMD GPU with ROCm / HIP: error: use of undeclared identifier 'CUDART_VERSION'

[ehratjon]

I checked the issues in this Repo and searched the error on search engines and forums but found nothing helpful.

Hi there, thanks for releasing this amazing work!

I am trying to run MMLab's Open Tracking. Part of this is the installation of GroundingDino, where I run into difficulties.

When not setting CUDA_HOME of course the typical NameError: name '_C' is not defined error occurs when running the model.

Since my PC has an AMD GPU I am not sure if I set the environment variable correctly.

Details about my environment

 

Running mmdetections ``mmdet/utils/collect_env.py``

``` bash /bin/sh: 1: /opt/rocm-6.0.2/bin/nvcc: not found sys.platform: linux Python: 3.9.19 (main, Mar 21 2024, 17:11:28) [GCC 11.2.0] CUDA available: True MUSA available: False numpy_random_seed: 2147483648 GPU 0: AMD Radeon RX 6800 XT CUDA_HOME: /opt/rocm-6.0.2 NVCC: Not Available GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 PyTorch: 2.4.0.dev20240501+rocm6.0 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201703 - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX512 - HIP Runtime 6.0.32830 - MIOpen 3.0.0 - Magma 2.7.2 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, TorchVision: 0.19.0.dev20240501+rocm6.0 OpenCV: 4.9.0 MMEngine: 0.10.4 MMDetection: 3.3.0+16e0ccd ```

Running pytorch's ``torch.utils.collect_env``

``` bash /home/zupan/miniconda3/envs/venv_pythia/lib/python3.9/runpy.py:127: RuntimeWarning: 'torch.utils.collect_env' found in sys.modules after import of package 'torch.utils', but prior to execution of 'torch.utils.collect_env'; this may result in unpredictable behaviour warn(RuntimeWarning(msg)) Collecting environment information... PyTorch version: 2.4.0.dev20240501+rocm6.0 Is debug build: False CUDA used to build PyTorch: N/A ROCM used to build PyTorch: 6.0.32830-d62f6a171 OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.35 Python version: 3.9.19 (main, Mar 21 2024, 17:11:28) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: Could not collect CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: AMD Radeon RX 6800 XT (gfx1030) Nvidia driver version: Could not collect cuDNN version: Could not collect HIP runtime version: 6.0.32830 MIOpen runtime version: 3.0.0 Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: GenuineIntel Model name: 11th Gen Intel(R) Core(TM) i9-11900K @ 3.50GHz CPU family: 6 Model: 167 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 1 Stepping: 1 CPU max MHz: 5300.0000 CPU min MHz: 800.0000 BogoMIPS: 7008.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap avx512ifma clflushopt intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm md_clear flush_l1d arch_capabilities L1d cache: 384 KiB (8 instances) L1i cache: 256 KiB (8 instances) L2 cache: 4 MiB (8 instances) L3 cache: 16 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] pytorch-triton-rocm==3.0.0+bbe6246e37 [pip3] torch==2.4.0.dev20240501+rocm6.0 [pip3] torch-tb-profiler==0.4.3 [pip3] torchaudio==2.2.0.dev20240501+rocm6.0 [pip3] torchdata==0.7.1+8e84488 [pip3] torchvision==0.19.0.dev20240501+rocm6.0 [conda] libopenvino-pytorch-frontend 2023.2.0 h59595ed_0 conda-forge [conda] libtorch 2.1.2 cpu_mkl_hcefb67d_103 conda-forge [conda] mkl 2023.2.0 h84fe81f_50496 conda-forge [conda] numpy 1.26.4 py39heeff2f4_0 [conda] numpy-base 1.26.4 py39h8a23956_0 [conda] pytorch-triton-rocm 3.0.0+bbe6246e37 pypi_0 pypi [conda] torch 2.4.0.dev20240501+rocm6.0 pypi_0 pypi [conda] torch-tb-profiler 0.4.3 pypi_0 pypi [conda] torchaudio 2.2.0.dev20240501+rocm6.0 pypi_0 pypi [conda] torchdata 0.7.1 py39h50156ae_3 conda-forge [conda] torchvision 0.19.0.dev20240501+rocm6.0 pypi_0 pypi ```

I export CUDA_HOME=/opt/rocm-6.0.2/ and tried to install GroundingDINO

git clone git+https://github.com/IDEA-Research/GroundingDINO.git
cd GroundingDINO
pip install -v -e .

Unfortunately I get the following warning and errors during the installation:

...
  76 warnings generated when compiling for gfx1030.

...
    groundingdino/models/GroundingDINO/csrc/hip_version.hip:6:10: error: use of undeclared identifier 'CUDART_VERSION'
      return CUDART_VERSION;
             ^
    1 error generated when compiling for gfx1030.
    error: command '/opt/rocm-6.0.2/bin/hipcc' failed with exit code 1

Content of ``hip_version.hip``

``` cpp // !!! This is a file automatically generated by hipify!!! #include namespace groundingdino { int get_cudart_version() { return CUDART_VERSION; } } // namespace groundingdino ```

I tried export CUDART_VERSION=60032831 as an environment variable using python:

from cuda import cudart
cudart.cudaRuntimeGetVersion() 
# (<hipError_t.hipSuccess: 0>, 60032831)

to no avail.

Do I use the environment variables correctly or are there other requirements when using AMD/ROCm/HIP? If more information is required, please let me know what you need (and how to get the information). Any help or pointers in the right direction would be appreciated.

[alisvkr]

Hi, I have same issue, anyone found any workaround ?

[ehratjon]

No, unfortunately I haven't found anything useful. For now I have given up on investigating further, but if you come across something, please let me know. Good luck!

[alisvkr]

I made it run with defining CUDART_VERSION explicitly in path "Grounded-Segment-Anything/GroundingDINO/groundingdino/models/GroundingDINO/csrc/cuda_version.cu". As below; namespace groundingdino { int get_cudart_version() { //return CUDART_VERSION; return 6032831; //your cuda version } } // namespace groundingdino

[ehratjon]

It worked for me as well. Makes sense that replacing the code part in the .hip file didn't work as it gets generated and overwritten. Thank you!