search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.12k stars 2.08k forks source link

JtR on Windows: OpenCL DLL usage #3191

Closed dwendt closed 8 months ago

dwendt commented 6 years ago

This bug initially started with clGetPlatformIDs returning -1001 which is CL_PLATFORM_NOT_FOUND_KHR and john dies which is wrong/bad.

hashcat on the same system detects the platform fine using literally the same function. The only conclusion I can come to is that hashcat is doing something correctly in their build, and john is not.

Steps to reproduce

  1. grab the first build of bleeding-jumbo you can find on google http://openwall.info/wiki/_media/john/johntheripper-v1.8.0.12-jumbo-1-bleeding-e6214ceab-2018-02-07-win-x64.7z
  2. install a graphics card whose platform is opencl 1.2
  3. run john.exe --list=opencl-devices with a card that's restricted to OCL 1.2 platform (this should work fine with 2.1/2.2 headers...)

System configuration

windows enterprise 10.0.16299.19, 64-bit

H:\john180j1w\bleeding>.\run\john.exe --verbosity=5 --list=opencl-devices
Warning: '/dev/shm' does not exists or is not a directory.

POSIX shared memory objects require the existance of this directory.
Create the directory '/dev/shm' and set the permissions to 01777.
For instance on the command line: mkdir -m 01777 /dev/shm
initUnicode(UNICODE, UTF-8/ISO-8859-1)
UTF-8 -> UTF-8 -> UTF-8
Error: No OpenCL-capable platforms were detected by the installed OpenCL driver.
Throw clError: clGetPlatformIDs() = -1001
Error: No OpenCL-capable devices were detected by the installed OpenCL driver.

H:\john180j1w\bleeding>.\run\john.exe --list=build-info
Warning: '/dev/shm' does not exists or is not a directory.

POSIX shared memory objects require the existance of this directory.
Create the directory '/dev/shm' and set the permissions to 01777.
For instance on the command line: mkdir -m 01777 /dev/shm
Version: 1.8.0.12-jumbo-1-bleeding-e6214ceab 2018-02-07 17:56:37 +0100
Build: cygwin 64-bit x86_64 SSE4.2 AC MPI + OMP
SIMD: SSE4.1, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
CPU tests: SSE4.2
$JOHN is ./run/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 6.4.0
OpenCL headers version: 2.2
Crypto library: OpenSSL
OpenSSL library version: 0100020ef
OpenSSL 1.0.2n  7 Dec 2017
GMP library version: 6.1.2
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
H:\hashcat-4.0.1>hashcat64.exe -I
hashcat (v4.0.1) starting...

OpenCL Info:

Platform ID #1
  Vendor  : NVIDIA Corporation
  Name    : NVIDIA CUDA
  Version : OpenCL 1.2 CUDA 9.1.84

  Device ID #1
    Type           : GPU
    Vendor ID      : 32
    Vendor         : NVIDIA Corporation
    Name           : GeForce GTX 970
    Version        : OpenCL 1.2 CUDA
    Processor(s)   : 13
    Clock          : 1253
    Memory         : 1024/4096 MB allocatable
    OpenCL Version : OpenCL C 1.2
    Driver Version : 391.01

root cause

almost working after replacing the junk cygopencl DLL

POSIX shared memory objects require the existance of this directory. Create the directory '/dev/shm' and set the permissions to 01777. For instance on the command line: mkdir -m 01777 /dev/shm initUnicode(UNICODE, UTF-8/ISO-8859-1) UTF-8 -> UTF-8 -> UTF-8 Device 0@konata: GeForce GTX 970 Using default input encoding: UTF-8 Loaded 1 password hash (dmg-opencl, Apple DMG [PBKDF2-SHA1 OpenCL 3DES/AES]) Cost 1 (iteration count) is 270270 for all loaded hashes Will run 8 OpenMP threads Loaded 7 hashes with 7 different salts to test db from test vectors Options used: -I /run/kernels -cl-mad-enable -DSM_MAJOR=5 -DSM_MINOR=2 -cl-nv-verbose -DGPU -DDEVICE_INFO=262162 -DSIZEOF_SIZE_T=8 -DDEV_VER_MAJOR=391 -DDEV_VER_MINOR=1 -D_OPENCL_COMPILER -DHASH_LOOPS=753 -DOUTLEN=32 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 $JOHN/kernels/pbkdf2_hmac_sha1_kernel.cl Build log: :46:10: fatal error: 'opencl_device_info.h' file not found

include "opencl_device_info.h"

     ^

Error -11 building kernel $JOHN/kernels/pbkdf2_hmac_sha1_kernel.cl. DEVICE_INFO=262162 OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in common-opencl.c:1111 - clBuildProgram failed.

magnumripper commented 4 years ago

I recently (do not hold this against me, I will forever deny it) helped someone to install JtR on some Windoes gear. I tried using your (@claudioandre-br) latest binary and did just that - copied a known-good nvidia OpenCL.dll right over cygOpenCL-1.dll. Worked like a champ (but not until that). Good lord knows I was relieved.

solardiz commented 1 year ago

Given all of the above and the recent reminder in #5262, I wonder if we can workaround the issue by somehow linking against OpenCL.dll instead of cygOpenCL.dll (or is it cygOpenCL-1.dll). As I understand. right now we link against -lOpenCL, so the usage of cygOpenCL.dll (or cygOpenCL-1.dll) is added implicitly by Cygwin. Can we bypass that by usage of different options to the linker? If not, or not easily, then maybe we should add a step where we use e.g. sed -i to patch the library name in the already-linked binary? Luckily, cygOpenCL.dll is longer than OpenCL.dll, so we should be able to replace one string by the other plus several NUL bytes. @claudioandre-br or anyone else who's on Windows (maybe @boulevard-h from #5262 as someone who ran into this issue most recently?), can you please give this a try? Thank you!

solardiz commented 1 year ago

This seems to do what's needed, but I did not actually test on Windows (just checked that the file is changed the way I want, on two Linux systems with different versions of sed and different locales):

LC_CTYPE=C sed -rbi 's/cygOpenCL-1\.dll(.)/OpenCL.dll\1\1\1\1\1\1/' john.exe

Please test and let us all know. If this works as desired, then let's get it into our build procedure for Cygwin (perhaps into the configure-generated Makefile and into Makefile.legacy).

claudioandre-br commented 1 year ago

I'm afraid this is not a good solution as we also need to package the cygWin library (otherwise error in non-OpenCL environments).

The example below will show where the devil lives:

C:\Temp\debug> run\john --test --format=nt-opencl
Device 1: gfx902 [AMD Radeon(TM) Vega 8 Graphics]
Benchmarking: NT-opencl [MD4 OpenCL/mask accel]... LWS=64 GWS=2048 (32 blocks) x2470 DONE
Raw:    1170M c/s real, 6911M c/s virtual

C:\Temp\debug> del run\cygOpenCL-1.dll

C:\Temp\debug> dir run\cyg*.dll
    [...]  
   8 file(s)      7.180.198 bytes

# => john useless
C:\Temp\debug> run\john --test --format=nt
C:/Temp/debug/run/john-best.exe: error while loading shared libraries: cygOpenCL-1.dll: cannot open shared object file: No such file or directory

C:\Temp\debug> run\john --help
C:/Temp/debug/run/john-best.exe: error while loading shared libraries: cygOpenCL-1.dll: cannot open shared object file: No such file or directory

C:\Temp\debug> copy c:\Windows\System32\OpenCL.dll c:\Windows\System32\cygOpenCL-1.dll
# Admin is required.
# => Now we have a valid and functional OpenCL library, please don't worry about the name.

C:\Temp\debug> run\john --test --format=nt-opencl
Device 1: gfx902 [AMD Radeon(TM) Vega 8 Graphics]
Benchmarking: NT-opencl [MD4 OpenCL/mask accel]... LWS=64 GWS=16384 (256 blocks) x2470 DONE
Raw:    1489M c/s real, 29880M c/s virtual

What if we have some cygwin OpenCL DLL in run folder:

#=> Copy the CygWin original cygOpenCL-1.dll to the run folder.

C:\Temp\debug> dir run\cyg*.dll
[...]
01/01/2018  09:55           110.099 cygOpenCL-1.dll
[...]               9 files(s)      7.290.297 bytes

C:\Temp\debug> run\john --test --format=nt-opencl
No OpenCL devices found
solardiz commented 1 year ago

I'm afraid this is not a good solution as we also need to package the cygWin library (otherwise error in non-OpenCL environments).

I think I get what you mean, even though your examples don't really show it - this would matter for Windows systems that don't have c:\Windows\System32\OpenCL.dll, and it would prevent usage of our CPU formats, which otherwise should work even on such systems. (That nt-opencl fail, like in your examples, is of less importance - on such systems, it'd have failed anyway, just differently. However, if you ran e.g. run\john --test --format=nt, it'd also bump into the error while loading shared libraries, which would actually make john useless.)

Is c:\Windows\System32\OpenCL.dll in fact optional on modern Windows, or do modern versions of Windows provide it even when there are no OpenCL device drivers installed? Would it be sane for us to assume that any 64-bit Windows has this DLL?

claudioandre-br commented 1 year ago

e.g. run\john --test --format=nt, it'd also bump into the error while loading shared libraries, which would actually make john useless.)

100% useless, not even a run\john --help.

do modern versions of Windows provide it even when there are no OpenCL device drivers installed?

Well, I don't know, but I can uninstall stuff, so I would say that's a dangerous assumption. But we should call people to comment on the matter.

[edited]

In fact, we can use AWS Amazon EC2 T2 (Free Tier) to verify this. I'll do that when I have my security keys in hand.

solardiz commented 1 year ago

Skimming https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order I see that "Known DLLs" are of higher priority than those in "The folder from which the application loaded." So if OpenCL.dll is listed in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs, then we can get around the issue just brought up by @claudioandre-br above by shipping a copy of Cygwin's cygOpenCL-1.dll renamed to OpenCL.dll in the current directory, for it to be used as fallback when the system does not have its own copy of OpenCL.dll. OTOH, I don't know if it's commonly or ever the case that OpenCL.dll is listed among known ones even when it's missing?

If OpenCL.dll isn't "known", then unfortunately our program directory's version of it will take precedence over the system's, which is not what we want here.

claudioandre-br commented 1 year ago

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs

Starting with 'O' only 2 files: ole*.dll.

Just to document it here, another way to fix it:

02/04/2023 18:25

. 02/04/2023 18:25 .. 02/04/2023 18:30 104 amd.icd 31/03/2023 13:20 38 intel.icd 31/03/2023 13:20 33 nvidia.icd 3 arquivo(s) 175 bytes 2 pasta(s) 321.126.096.896 bytes disponíveis


- Edit the `.icd` file and point it to the correct vendor DLL file. Just put the DLL full name inside the file.

This is not valid Windows syntax, In fact, I used Notepad

echo 'C:\WINDOWS\System32\DriverStore\FileRepository\u0357645.inf_amd64_3c66802dbd4d45a0\B357435\amdocl64.dll' > etc\OpenCL\vendors\amd.icd



[1] A very good and easy to remember place to save a library.
      On Linux, we use something awful like `/lib` and a symbolic link.
solardiz commented 1 year ago
  • Edit the .icd file and point it to the correct vendor DLL file. Just put the DLL full name inside the file.

Apparently, we could automate that by reading the registry and writing to Cygwin's .icd file (we could do this in john or in a separate program, and would need to inform the user of us making such changes outside of our program tree). See: https://github.com/openwall/john/issues/3191#issuecomment-508086364

alainesp commented 10 months ago

What is the problem with:

In the configure script. If we are under Cygwin and OpenCL is supported:

solardiz commented 10 months ago

In the configure script. If we are under Cygwin and OpenCL is supported:

* copy the file C:\Windows\System32\OpenCL.dll to JtR's run directory, renaming it to cygOpenCL-1.dll

* If no file exists we print a warning/error

This only works for those building from source on the very machine where they intend to run. Most Windows users will probably run our pre-built binary distribution instead.

I actually thought of doing something like this in john.exe, so it'd work right starting from the second invocation of it - but I'd prefer us finding/implementing a cleaner solution.

alainesp commented 10 months ago

This only works for those building from source on the very machine where they intend to run. Most Windows users will probably run our pre-built binary distribution instead.

I understand. Question. How does JtR manage the distribution of OpenCL-supported JtR on Linux on systems without libOpenCl.so?

The solution that I guess gives the most compatibility is to not link to OpenCL statically, create a *.c file with all the OpenCL functions, and implement them as a hub to the pointer of functions dynamically loaded. If we are lucky the compiler will inline most of them and we probably only need to check if the pointers are good on the couple of OpenCL functions that you can call at the beginning.

solardiz commented 10 months ago

How does JtR manage the distribution of OpenCL-supported JtR on Linux on systems without libOpenCl.so?

It does not, and there's not as much need for that. Packages for Linux typically target a specific distro/version and have dependencies on other packages specified in them. As an alternative for that, @claudioandre-br provides Flatpak and Snap packages, which bundle their dependencies. Anyway, we recommend building for Linux from source, and the releases on JtR homepage are currently only source tarballs except for Windows, where we do provide binary builds.

The solution that I guess gives the most compatibility is to not link to OpenCL statically, create a *.c file with all the OpenCL functions, and implement them as a hub to the pointer of functions dynamically loaded. If we are lucky the compiler will inline most of them and we probably only need to check if the pointers are good on the couple of OpenCL functions that you can call at the beginning.

Right, I think (but am not sure) this is similar to what hashcat does (please take a look). I was thinking we'd use an .h file with macros wrapping around dlsym, but I don't mind wrapper functions as well (can also be static inline ones in an .h file). I'm not worried about performance since OpenCL calls shouldn't be frequent (if they are, we have a problem anyway).

So if you're willing to proceed to try and implement this, please go ahead. Thank you very much!

solardiz commented 10 months ago

I think (but am not sure) this is similar to what hashcat does (please take a look).

@alainesp See https://github.com/openwall/john/issues/3191#issuecomment-508119262

solardiz commented 10 months ago

I'm not worried about performance since OpenCL calls shouldn't be frequent (if they are, we have a problem anyway).

That said, we should cache dlsym results. The overhead of a function call doesn't worry me much, but the overhead of redoing a symbol lookup by name could be too great.

alainesp commented 10 months ago

How does JtR manage the distribution of OpenCL-supported JtR on Linux on systems without libOpenCl.so?

It does not, and there's not as much need for that. Packages for Linux typically target a specific distro/version and have dependencies on other packages specified in them. As an alternative for that, @claudioandre-br provides Flatpak and Snap packages, which bundle their dependencies.

Could you do that? Reference a supposed driver library from a Linux package manager? I was under the impression that the OpenCL support was installed by the GPU driver, at least on Windows.

solardiz commented 10 months ago

Could you do that? Reference a supposed driver library from a Linux package manager?

I think the auto-generated dependency would be on libOpenCL.so.1 or such, and e.g. on Debian/Ubuntu that can be provided by https://packages.debian.org/sid/ocl-icd-libopencl1 - which is just the loader - or by the actual GPU driver package. You could also add various dependencies manually (such as to require a specific OpenCL backend installed via the package manager), but that's inconvenient and not normally done.

Anyway, I think we should go for the dlopen approach for Windows, and when we have it then also use it on Linux (which would also provide more debugging/testing of it).

solardiz commented 9 months ago

I think we should go for the dlopen approach for Windows, and when we have it then also use it on Linux

We now have initial implementation of this in #5386 by @alainesp - thank you, Alain!

mainmind83 commented 8 months ago
  • Locate the vendor's OpenCL dll file. In my case it is [1]: C:\WINDOWS\System32\DriverStore\FileRepository\u0357645.inf_amd64_3c66802dbd4d45a0\B357435\amdocl64.dll

If there are several directories with the DLL library, use the one with the most recent date on C:\WINDOWS\System32\DriverStore\FileRepository

It might also be possible to use the ObjectListView component of Driver Store Explore where it is possible to access the directory with the dll of AMD-OpenCL User Mode Driver:

image

solardiz commented 8 months ago

Thank you @mainmind83, but none of this should be needed now that we've merged @alainesp's #5386 and there's a new Windows build at https://github.com/openwall/john-packages/releases/tag/jumbo-dev also linked from the "Download Windows Build" badge in README.md in this repo. So I'll close this issue as completed. We already had some people report that those changes have helped, but it'd be great if you @mainmind83 also test and let us know in here (it's OK to add comments to a closed issue).

mainmind83 commented 8 months ago

It was still open 🥲 It's working without edit .icd file but appear warning:

john.exe --test --format=nt-opencl
Device 1: gfx1031 [AMD Radeon RX 6700 XT]
Benchmarking: NT-opencl [MD4 OpenCL/mask accel]... Build log: warning: argument unused during compilation: '-I opencl' [-Wunused-command-line-argument]
1 warning generated.

Build log: warning: argument unused during compilation: '-I opencl' [-Wunused-command-line-argument]
1 warning generated.

LWS=128 GWS=327680 (2560 blocks) x9500 DONE
Raw:    36402M c/s real, 6031G c/s virtual
solardiz commented 8 months ago

Thank you for testing this @mainmind83!

The Build log: warning: argument unused during compilation: '-I opencl' is a separate (non-)issue related to our fix of #4335, which was #4373. I'm surprised by this warning, though, since opencl/nt_kernel.cl does include a few other files and we're only chdir'ing to the run directory, not inside opencl. Maybe that's an extra reason for us to chdir to run/opencl instead, like we discussed in https://github.com/openwall/john/pull/4938#issuecomment-992621520

Edit: Upon a second thought, I am not surprised by the warning. Perhaps the newer/better OpenCL runtimes are looking for the #include "..." files in the same directory with the file that includes them, like it would be in C. However, we historically had problems with this not happening, which is why we added the -I (with a different path back then).