Old issue( std::bad_alloc: CUDA error at: /workspace/.conda-bld/work/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorNotSupported operation not supported)

[Shafi2016]

Hi,

I'm getting the error given below and using the WSL2 Ubuntu 20.04 instance on Windows 11 Preview.

RuntimeError: CUDA error encountered at: /workspace/.conda-bld/work/cpp/src/bitmask/null_mask.cu:93: 801 cudaErrorNotSupported operation not supported.

I saw old posts but could not find the solution. The following chunk of codes works fine

import cudf; print('cuDF Version:', cudf.__version__)
import numpy as np; print('NumPy Version:', np.__version__)

output

cuDF Version: 22.06.01 NumPy Version: 1.22.4

s = cudf.Series([1,2,3,None,4])
s

[bdice]

@Shafi2016 What kind of GPU do you have? I see GeForce ... in the output of nvidia-smi. It sounds like a previous issue concluded that Pascal GPUs (which would include the GeForce 10xx series) are not supported on WSL2. You may need newer hardware to run cuDF on WSL2. See: https://github.com/rapidsai/cudf/issues/9427#issuecomment-975902493

[Shafi2016]

Pascal GPUs

Thank you @bdice. I have GeForce GTX 1050 Ti. So used driver. By the way how to know which hardware supports the CuDF. I followed this tutorial for RAPIDS.

[davidwendt]

The WSL2 requirements are listed here: https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-apps The RAPIDS requirements are documented here: https://rapids.ai/start.html#requirements From https://github.com/rapidsai/cudf/issues/9427#issuecomment-975902493 it looks like we may need to update the requirements page specifically for WSL2 and RAPIDS. @JohnZed @beckernick @taureandyernv

[Shafi2016]

The WSL2 requirements are listed here: https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-apps The RAPIDS requirements are documented here: https://rapids.ai/start.html#requirements From #9427 (comment) it looks like we may need to update the requirements page specifically for WSL2 and RAPIDS. @JohnZed @beckernick @taureandyernv

Thank you for the updated info.

[taureandyernv]

@shafi2016, the GTX1050TI is a Pascal card. I regret to inform you that the last RAPIDS version to work with WSL2 and Pascal cards was 21.08, as @davidwendt shared via my issue. We still have conda and docker packages for hat release. After that release, things stopped working and we haven't yet figured out a workaround or solution for newer versions of RAPIDS. We hope that it could resolve itself through driver, CUDA, and WSL2 updates, like how running dask-cudf on WSL2 did. Please remember that WSL2 has Experimental support and we have to prioritize with our limited bandwidth. I say that as someone who was also affected by this issue and also terribly disappointed at first, but knowing what I know, I'm really understanding about the situation.

While this is not a supported method, you can refer to https://docs.rapids.ai/notices/rgn0024/ for documentation on installing WSL2 on your system and then trying the older versions with the newer driver (I had been using an older driver). If it works out, please let us know so that we can continue to best advise others in the community who are trying WSL2 and RAPIDS on a Pascal card. We'll also clarify this on the website, as @davidwendt suggested.

[Shafi2016]

Thank you, @taureandyernv, for the detailed note. It is indeed encouraging. Let me do the experiments.

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[mattharrison]

I'm running into the same error with 22.08 and 22.10 on my WSL2 machine. I have a Quadro P1000 card.

Is there a known older version that works? (FWIW Cupy seems to work just fine with my card).

[taureandyernv]

@mattharrison , 21.08 had success on the P1000 (I have one too), but ymmv, as I did the old install method as well. Let us know if you are successful

[mattharrison]

I used the conda command generated from the "Install Rapids" gui/thing https://rapids.ai/start.html#get-rapids.

@taureandyernv what is the "old install method"?

[taureandyernv]

wsl2 had a few changes to the installation method. I don't know if there are breaking changes that occurred for 21.08's compatibility with the P series cards. Try both conda and docker and it would be awesome to get an update on if it still works.

[mattharrison]

For those following along at home, I was able to get version 21.08 installed with this command and create dataframes succesfully!

conda create -n rapidsai -c rapidsai -c nvidia -c conda-forge cudf=21.08 python=3.8 cudatoolkit=11.4

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[davidwendt]

Closing this as resolved. Feel free to reopen if there is new information.