WSL2-Ubuntu Nvidia GPU not recognize either from CUDA or OpenCL

diegoxcn commented 2 years ago

Checklist

Guide I followed to make JtR:

Attach details about your OS and about john, including:

The output of ./john --list=build-info.

╰─λ /opt/john/run/john --list=build-info                                             
Version: 1.9.0-jumbo-1+bleeding-ff16ecb93 2021-12-08 20:03:10 +0100
Build: linux-gnu 64-bit x86_64 AVX512BW AC OMP
SIMD: AVX512BW, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
CPU tests: AVX512BW
$JOHN is /opt/john/run/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 9.3.0
GNU libc version: 2.31 (loaded: 2.31)
OpenCL headers version: 1.2
Crypto library: OpenSSL
OpenSSL library version: 01010106f
OpenSSL 1.1.1f  31 Mar 2020
GMP library version: 6.2.0
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
times(2) sysconf(_SC_CLK_TCK) is 100
Using times(2) for timers, resolution 10 ms
HR timer: clock_gettime(), latency 40 ns
Total physical host memory: 31970 MiB
Available physical host memory: 28778 MiB
Terminal locale string: C.UTF-8
Parsed terminal locale: UTF-8

The command line you are using.


╰─λ nvidia-smi                                                                               
Fri Dec 10 10:45:46 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.00       Driver Version: 510.06       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A200...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A   45C    P8     6W /  N/A |    391MiB /  4096MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

╰─λ uname -m && cat /etc/*release x86_64 DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS" NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

╰─λ wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin 130 (50.128s) ╰─λ sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600 ╰─λ wget https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda-repo-wsl-ubuntu-11-5-local_11.5.1-1_amd64.deb ╰─λ sudo dpkg -i cuda-repo-wsl-ubuntu-11-5-local_11.5.1-1_amd64.deb ╰─λ sudo apt-key add /var/cuda-repo-wsl-ubuntu-11-5-local/7fa2af80.pub
OK ╰─λ sudo apt-get update ╰─λ sudo apt-get -y install cuda ╰─λ sudo apt-get -y install git build-essential libssl-dev zlib1g-dev ╰─λ sudo apt-get -y install yasm pkg-config libgmp-dev libpcap-dev libbz2-dev ╰─λ sudo apt-get -y install nvidia-opencl-dev ╰─λ sudo apt-get -y install ocl-icd-opencl-dev opencl-headers pocl-opencl-icd ╰─λ sudo git clone https://github.com/openwall/john -b bleeding-jumbo john ╰─λ sudo ./configure Configured for building John the Ripper jumbo:

Target CPU ......................................... x86_64 AVX512BW, 64-bit LE AES-NI support ..................................... run-time detection Target OS .......................................... linux-gnu Cross compiling .................................... no Legacy arch header ................................. x86-64.h

Optional libraries/features found: Memory map (share/page large files) ................ yes Fork support ....................................... yes OpenMP support ..................................... yes (not for fast formats) OpenCL support ..................................... yes Generic crypt(3) format ............................ yes OpenSSL (many additional formats) .................. yes libgmp (PRINCE mode and faster SRP formats) ........ yes 128-bit integer (faster PRINCE mode) ............... yes libz (7z, pkzip and some other formats) ............ yes libbz2 (7z and gpg2john bz2 support) ............... yes libpcap (vncpcap2john and SIPdump) ................. yes Non-free unrar code (complete RAR support) ......... yes librexgen (regex mode, see doc/README.librexgen) ... no OpenMPI support (default disabled) ................. no Experimental code (default disabled) ............... no ZTEX USB-FPGA module 1.15y support ................. no

Install missing libraries to get any needed features that were omitted.

╰─λ sudo make -s clean && sudo make -sj4 ╰─λ ../run/john --list=opencl-devices Platform #0 name: Intel(R) OpenCL HD Graphics, version: OpenCL 3.0 Device #0 (1) name: Intel(R) Graphics [0x9a60] Device vendor: Intel(R) Corporation Device type: GPU (LE) Device version: OpenCL 3.0 NEO Driver version: 21.37.20939 Native vector widths: char 16, short 8, int 4, long 1 Preferred vector width: char 16, short 8, int 4, long 1 Global Memory: 26103 MiB Global Memory Cache: 512 KiB Local Memory: 64 KiB (Local) Constant Buffer size: 1 GiB Max memory alloc. size: 1 GiB Max clock (MHz): 1450 Profiling timer res.: 52 ns Max Work Group Size: 256 Parallel compute cores: 32 Stream processors: 256 (32 x 8) Speed index: 371200

Platform #1 name: Portable Computing Language, version: OpenCL 1.2 pocl 1.4, None+Asserts, LLVM 9.0.1, RELOC, SLEEF, DISTRO, POCL_DEBUG Device #0 (2) name: pthread-11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz Device vendor: GenuineIntel Device type: CPU (LE) Device version: OpenCL 1.2 pocl HSTR: pthread-x86_64-pc-linux-gnu-icelake-client Driver version: 1.4 Native vector widths: char 16, short 16, int 16, long 8 Preferred vector width: char 16, short 16, int 16, long 8 Global Memory: 29922 MiB Global Memory Cache: 24 MiB Local Memory: 16 MiB (Global) Constant Buffer size: 16 MiB Max memory alloc. size: 8 GiB Max clock (MHz): 2496 Profiling timer res.: 1 ns Max Work Group Size: 4096 Parallel compute cores: 16 Speed index: 319488

╰─λ ls /dev/dxg /dev/dxg



Well, this procedure works totally fine on Ubuntu Bare-metal installation. Not exactly the CUDA part, but at least JtR could get the same nvidia RTX A2000 GPU from opencl driver. However, it's not working on WSL2 Ubuntu at all. I tried several times with Windows Nvidia driver installation (guide says WSL2 would not actually need to install nvidia driver like Bare-metal installation), or with actually both Windows Nvidia driver installation along with WSL2 Ubuntu Nvidia-driver-470 installation. Everytime john would only recognize the Intel GPU instead of Nvidia.

May I ask if there's any possibility to do this on WSL2 Ubuntu at all? If yes, which part of me doing wrong please? 

Thank you very much. If there's any other information you required, I'm more than happy to keep updating this post.

Cheers,
Channing.

solardiz commented 2 years ago

@diegoxcn Thank you for providing so much detail.

What files do you have under /etc/OpenCL? Do you have something like this? -

$ cat /etc/OpenCL/vendors/nvidia.icd
libnvidia-opencl.so.1

diegoxcn commented 2 years ago

Hi @solardiz , thanks for quick reply, and yes, I do have it:

╰─λ ls /etc/OpenCL/vendors                                                                                 
intel.icd  nvidia.icd  pocl.icd

╰─λ cat /etc/OpenCL/vendors/nvidia.icd                                                                      
libnvidia-opencl.so.1

solardiz commented 2 years ago

@diegoxcn Does clinfo see the NVIDIA GPU?

diegoxcn commented 2 years ago

@solardiz Nope, it only shows Intel OpenCL HD Graphics and Portable Computing Language, no nvidia at all.

claudioandre-br commented 2 years ago

Last I read about it, WSL doesn't support GPU Acc/OpenCL.

It is unclear whether this is still the case at https://docs.nvidia.com/cuda/wsl-user-guide/index.html#features-not-yet-supported. However, if clinfo fails, the problem is not related to the JtR binary or the build process.

I would run ldd libnvidia-opencl.so.1 just to keep trying to figure out if I can debug more. And if it's more than an experiment, please use the native binary.

diegoxcn commented 2 years ago

Hi @claudioandre-br Thanks for the infor, I tried ldd libnvidia-opencl.so.1 which shows no such file, seems it's not installed by nvidia-opencl-dev package, will check if I can get it somehow. And update this post later. Thank you very much.

╰─λ find /usr/lib -name libnvidia-opencl.so.1 2>/dev/null                                                  
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1

Well, apparently this file does exist, just somehow it's not registered correctly by ldd?

solardiz commented 2 years ago

@diegoxcn I think @claudioandre-br assumed you'd run this command when in the right directory, so after cd /usr/lib/x86_64-linux-gnu in your case. ldd isn't supposed to find it automatically, so no problem there.

You could want to run ldconfig -v as root and see if its output lists that library or not. Also, this action might help on its own, so retest clinfo after it.

diegoxcn commented 2 years ago

@solardiz Thanks for the instruction.

╰─λ ldd /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1                                                                                 0 (0.000s)
        linux-vdso.so.1 (0x00007ffcaafcb000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6575f07000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6575db8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6575db2000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6575d8f000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f6575d84000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f65773bf000)

Then it works fine. :)
And ldconfig -v does show this so.1 file:

╰─λ sudo ldconfig -v
....omit output...
libnvidia-opencl.so.1 -> libnvidia-opencl.so.460.106.00
...omit output...

But unfortunately the clinfo still have very same result. Well, not sure if it's the limitation of the WSL2.

Thanks.

solardiz commented 2 years ago

@diegoxcn I'm afraid we don't currently have further advice for you.

Guide I followed to make JtR:
2. https://openwall.info/wiki/john/GPU (NVIDIA part)

Ouch. There was nothing supposed to be currently useful in there. I had previously added a note that the content in there was merely historical, but apparently that wasn't enough, so I've just fully dropped the content of that wiki page. If anyone wants to maintain an up-to-date replacement, please feel free to do so, but for now it's better for this page not to exist than to keep it up with content dating so many years back. (Since it's a wiki, the old content can still be seen in page edits history. Hopefully, that's obscure enough that people wouldn't be wasting time on it.)

diegoxcn commented 2 years ago

@solardiz No Worries, I really appreciate all your help. Just wonder if there's any possibility for JtR could run against nvidia opencl GPU on WSL2.

Hope that your team could do some testing on WSL2 would be perfect.

Thanks again.

openwall / john

WSL2-Ubuntu Nvidia GPU not recognize either from CUDA or OpenCL #4924

Checklist