Closed blue-cat-whale closed 6 months ago
Thanks for the question. I am not sure what the root cause is for this. Can you try running the following on the RHEL 8 machine?
import rmm
print(rmm.__file__)
print(rmm.__version__)
print(dir(rmm))
I will try to set up a reproducer on Rocky 8 later today (the closest OS to RHEL 8.9).
Also, I see your path has names like condabin. Have you tried conda instead of pip?
It turns out that there was a version conflict in the underlying CUDA tookit. The problem is gone after reinstallation.
[Resolved] Cleaning and Reinstalling CUDA toolkit solves the problem.
In short,
cudf
runs well in one environment, but throws an exception in another environment. The two environments have identical cuda-related package versions. I have two machines, one is Ubuntu 22.04 while the other is RHEL8.9. Both machines have CUDA 12 installed. I runpython3 -m cudf.pandas my_code.py
smoothly on the Ubuntu machine, but got an error on RHEL8.9. The Ubuntu is defined as follow:As we can see from pic1 and pic2, CUDA related packages in the two environments have identical version numbers. But when I run the same command on the RHEL8.9 machine, it returns this error:
Apparently I've installed
rmm-cuda12
(see pic2 and pic3 below) and the two environments have identical version. I've tried removingrmm
andcudf
and re-installing them, but the problem is still there.PATH=/usr/bin:/home/<user_name>/.local/bin:/home/<user_name>/bin:/usr/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
andPYTHONPATH=/usr/local/lib64/python3.9/site-packages